Jul 27 17:25:28.966875 ip-10-0-140-0 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 27 17:25:28.966888 ip-10-0-140-0 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 27 17:25:28.966897 ip-10-0-140-0 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 27 17:25:28.967223 ip-10-0-140-0 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 27 17:25:39.099723 ip-10-0-140-0 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 27 17:25:39.099742 ip-10-0-140-0 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot ffb0b60d6de5408f9f4bb5e8c23871ea --
Jul 27 17:27:36.584751 ip-10-0-140-0 systemd[1]: Starting Kubernetes Kubelet...
Jul 27 17:27:36.992530 ip-10-0-140-0 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 17:27:36.992530 ip-10-0-140-0 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 27 17:27:36.992530 ip-10-0-140-0 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 17:27:36.992530 ip-10-0-140-0 kubenswrapper[2573]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 27 17:27:36.992530 ip-10-0-140-0 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 17:27:36.994623 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:36.994528    2573 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 27 17:27:36.998946 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.998930    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 17:27:36.998946 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.998945    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 17:27:36.999015 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.998949    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 17:27:36.999015 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.998954    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 17:27:36.999015 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.998962    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 17:27:36.999015 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.998964    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 17:27:36.999015 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.998967    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 17:27:36.999015 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.998971    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 17:27:36.999015 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.998974    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 17:27:36.999015 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.998976    2573 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 17:27:36.999015 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.998979    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 17:27:36.999015 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.998982    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 17:27:36.999015 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.998984    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 17:27:36.999015 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.998987    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 17:27:36.999015 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.998990    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 17:27:36.999015 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.998992    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 17:27:36.999015 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.998995    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 17:27:36.999015 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.998998    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 17:27:36.999015 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999001    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 17:27:36.999015 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999003    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 17:27:36.999015 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999006    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 17:27:36.999015 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999009    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 17:27:36.999490 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999012    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 17:27:36.999490 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999014    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 17:27:36.999490 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999017    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 17:27:36.999490 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999020    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 17:27:36.999490 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999022    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 17:27:36.999490 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999026    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 17:27:36.999490 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999028    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 17:27:36.999490 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999031    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 17:27:36.999490 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999033    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 17:27:36.999490 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999036    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 17:27:36.999490 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999039    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 17:27:36.999490 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999042    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 17:27:36.999490 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999045    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 17:27:36.999490 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999048    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 17:27:36.999490 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999050    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 17:27:36.999490 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999053    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 17:27:36.999490 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999058    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 17:27:36.999490 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999062    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 17:27:36.999490 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999065    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 17:27:36.999490 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999068    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 17:27:37.000247 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999071    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 17:27:37.000247 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999075    2573 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 17:27:37.000247 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999077    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 17:27:37.000247 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999081    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 17:27:37.000247 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999085    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 17:27:37.000247 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999088    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 17:27:37.000247 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999092    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 17:27:37.000247 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999094    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 17:27:37.000247 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999097    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 17:27:37.000247 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999100    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 17:27:37.000247 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999103    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 17:27:37.000247 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999106    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 17:27:37.000247 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999111    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 17:27:37.000247 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999115    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 17:27:37.000247 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999118    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 17:27:37.000247 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999121    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 17:27:37.000247 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999124    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 17:27:37.000247 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999127    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 17:27:37.000247 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999130    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 17:27:37.001018 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999133    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 17:27:37.001018 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999136    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 17:27:37.001018 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999139    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 17:27:37.001018 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999141    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 17:27:37.001018 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999144    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 17:27:37.001018 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999147    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 17:27:37.001018 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999149    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 17:27:37.001018 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999153    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 17:27:37.001018 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999155    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 17:27:37.001018 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999158    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 17:27:37.001018 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999160    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 17:27:37.001018 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999163    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 17:27:37.001018 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999165    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 17:27:37.001018 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999168    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 17:27:37.001018 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999171    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 17:27:37.001018 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999173    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 17:27:37.001018 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999175    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 17:27:37.001018 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999178    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 17:27:37.001018 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999180    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 17:27:37.001018 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999183    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 17:27:37.001509 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999185    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 17:27:37.001509 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999188    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 17:27:37.001509 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999190    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 17:27:37.001509 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999192    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 17:27:37.001509 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999195    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 17:27:37.001509 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999198    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 17:27:37.001509 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999201    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 17:27:37.001509 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999203    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 17:27:37.001509 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999206    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 17:27:37.001509 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999209    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 17:27:37.001509 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999211    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 17:27:37.001509 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999214    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 17:27:37.001509 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999216    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 17:27:37.001509 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999219    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 17:27:37.001509 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999221    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 17:27:37.001509 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999224    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 17:27:37.001509 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999227    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 17:27:37.001509 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:36.999229    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 17:27:37.002799 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002781    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 17:27:37.002799 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002797    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 17:27:37.002799 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002801    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 17:27:37.002916 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002805    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 17:27:37.002916 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002809    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 17:27:37.002916 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002812    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 17:27:37.002916 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002815    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 17:27:37.002916 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002818    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 17:27:37.002916 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002821    2573 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 17:27:37.002916 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002824    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 17:27:37.002916 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002827    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 17:27:37.002916 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002829    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 17:27:37.002916 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002834    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 17:27:37.002916 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002839    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 17:27:37.002916 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002841    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 17:27:37.002916 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002844    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 17:27:37.002916 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002847    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 17:27:37.002916 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002850    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 17:27:37.002916 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002852    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 17:27:37.002916 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002855    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 17:27:37.002916 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002857    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 17:27:37.002916 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002860    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 17:27:37.002916 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002863    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 17:27:37.003416 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002865    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 17:27:37.003416 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002868    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 17:27:37.003416 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002871    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 17:27:37.003416 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002873    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 17:27:37.003416 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002876    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 17:27:37.003416 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002880    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 17:27:37.003416 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002882    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 17:27:37.003416 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002886    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 17:27:37.003416 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002888    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 17:27:37.003416 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002897    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 17:27:37.003416 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002900    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 17:27:37.003416 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002902    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 17:27:37.003416 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002905    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 17:27:37.003416 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002908    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 17:27:37.003416 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002912    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 17:27:37.003416 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002915    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 17:27:37.003416 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002918    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 17:27:37.003416 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002921    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 17:27:37.003416 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002924    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 17:27:37.003942 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002927    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 17:27:37.003942 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002930    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 17:27:37.003942 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002933    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 17:27:37.003942 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002935    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 17:27:37.003942 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002938    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 17:27:37.003942 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002940    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 17:27:37.003942 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002943    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 17:27:37.003942 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002945    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 17:27:37.003942 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002948    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 17:27:37.003942 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002950    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 17:27:37.003942 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002953    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 17:27:37.003942 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002956    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 17:27:37.003942 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002959    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 17:27:37.003942 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002964    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 17:27:37.003942 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002967    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 17:27:37.003942 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002970    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 17:27:37.003942 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002973    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 17:27:37.003942 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002976    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 17:27:37.003942 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002979    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 17:27:37.004403 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002982    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 17:27:37.004403 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002984    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 17:27:37.004403 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002987    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 17:27:37.004403 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002989    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 17:27:37.004403 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002992    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 17:27:37.004403 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002994    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 17:27:37.004403 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.002997    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 17:27:37.004403 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003001    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 17:27:37.004403 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003004    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 17:27:37.004403 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003006    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 17:27:37.004403 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003009    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 17:27:37.004403 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003012    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 17:27:37.004403 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003015    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 17:27:37.004403 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003018    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 17:27:37.004403 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003021    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 17:27:37.004403 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003023    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 17:27:37.004403 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003025    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 17:27:37.004403 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003028    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 17:27:37.004403 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003030    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 17:27:37.004403 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003033    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 17:27:37.004900 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003035    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 17:27:37.004900 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003038    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 17:27:37.004900 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003041    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 17:27:37.004900 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003044    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 17:27:37.004900 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003047    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 17:27:37.004900 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003050    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 17:27:37.004900 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003052    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 17:27:37.004900 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003055    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 17:27:37.004900 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003057    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 17:27:37.004900 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003060    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 17:27:37.004900 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003062    2573 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 17:27:37.004900 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003065    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 17:27:37.004900 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003067    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 17:27:37.004900 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003070    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 17:27:37.004900 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003072    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 17:27:37.004900 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003075    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 17:27:37.004900 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003077    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 17:27:37.004900 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003080    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 17:27:37.004900 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003471    2573 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 17:27:37.004900 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003476    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 17:27:37.005400 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003479    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 17:27:37.005400 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003482    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 17:27:37.005400 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003485    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 17:27:37.005400 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003487    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 17:27:37.005400 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003490    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 17:27:37.005400 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003493    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 17:27:37.005400 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003496    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 17:27:37.005400 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003498    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 17:27:37.005400 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003501    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 17:27:37.005400 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003503    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 17:27:37.005400 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003506    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 17:27:37.005400 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003508    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 17:27:37.005400 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003511    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 17:27:37.005400 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003514    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 17:27:37.005400 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003517    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 17:27:37.005400 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003520    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 17:27:37.005400 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003523    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 17:27:37.005400 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003525    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 17:27:37.005400 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003528    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 17:27:37.005875 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003531    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 17:27:37.005875 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003534    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 17:27:37.005875 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003536    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 17:27:37.005875 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003539    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 17:27:37.005875 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003541    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 17:27:37.005875 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003544    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 17:27:37.005875 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003546    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 17:27:37.005875 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003549    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 17:27:37.005875 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003551    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 17:27:37.005875 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003554    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 17:27:37.005875 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003556    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 17:27:37.005875 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003559    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 17:27:37.005875 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003561    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 17:27:37.005875 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003564    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 17:27:37.005875 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003566    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 17:27:37.005875 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003569    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 17:27:37.005875 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003571    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 17:27:37.005875 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003574    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 17:27:37.005875 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003576    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 17:27:37.005875 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003579    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 17:27:37.006372 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003581    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 17:27:37.006372 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003584    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 17:27:37.006372 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003586    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 17:27:37.006372 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003589    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 17:27:37.006372 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003591    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 17:27:37.006372 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003593    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 17:27:37.006372 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003596    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 17:27:37.006372 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003599    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 17:27:37.006372 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003608    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 17:27:37.006372 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003611    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 17:27:37.006372 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003614    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 17:27:37.006372 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003616    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 17:27:37.006372 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003619    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 17:27:37.006372 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003622    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 17:27:37.006372 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003624    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 17:27:37.006372 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003628    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 17:27:37.006372 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003632    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 17:27:37.006372 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003635    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 17:27:37.006372 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003637    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 17:27:37.006911 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003640    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 17:27:37.006911 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003642    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 17:27:37.006911 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003645    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 17:27:37.006911 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003648    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 17:27:37.006911 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003650    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 17:27:37.006911 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003653    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 17:27:37.006911 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003656    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 17:27:37.006911 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003659    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 17:27:37.006911 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003662    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 17:27:37.006911 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003664    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 17:27:37.006911 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003666    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 17:27:37.006911 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003687    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 17:27:37.006911 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003691    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 17:27:37.006911 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003695    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 17:27:37.006911 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003699    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 17:27:37.006911 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003702    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 17:27:37.006911 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003704    2573 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 17:27:37.006911 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003709    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 17:27:37.006911 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003712    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 17:27:37.006911 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003715    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 17:27:37.007406 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003718    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 17:27:37.007406 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003721    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 17:27:37.007406 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003725    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 17:27:37.007406 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003728    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 17:27:37.007406 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003731    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 17:27:37.007406 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003753    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 17:27:37.007406 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003757    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 17:27:37.007406 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003761    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 17:27:37.007406 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003764    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 17:27:37.007406 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003766    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 17:27:37.007406 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003769    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 17:27:37.007406 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003772    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 17:27:37.007406 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003776    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 17:27:37.007406 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003779    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 17:27:37.007406 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003782    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 17:27:37.007406 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003785    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 17:27:37.007406 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003787    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 17:27:37.007406 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003790    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 17:27:37.007406 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.003793    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 17:27:37.007406 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.003861    2573 flags.go:64] FLAG: --address="0.0.0.0"
Jul 27 17:27:37.007899 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.003869    2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 27 17:27:37.007899 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.003876    2573 flags.go:64] FLAG: --anonymous-auth="true"
Jul 27 17:27:37.007899 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.003880    2573 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 27 17:27:37.007899 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.003885    2573 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 27 17:27:37.007899 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.003888    2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 27 17:27:37.007899 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.003892    2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 27 17:27:37.007899 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.003897    2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 27 17:27:37.007899 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.003901    2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 27 17:27:37.007899 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.003904    2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 27 17:27:37.007899 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.003907    2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 27 17:27:37.007899 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.003911    2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 27 17:27:37.007899 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.003914    2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 27 17:27:37.007899 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.003917    2573 flags.go:64] FLAG: --cgroup-root=""
Jul 27 17:27:37.007899 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.003920    2573 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 27 17:27:37.007899 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.003923    2573 flags.go:64] FLAG: --client-ca-file=""
Jul 27 17:27:37.007899 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.003927    2573 flags.go:64] FLAG: --cloud-provider="external"
Jul 27 17:27:37.007899 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.003930    2573 flags.go:64] FLAG: --cluster-dns="[]"
Jul 27 17:27:37.007899 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.003935    2573 flags.go:64] FLAG: --cluster-domain=""
Jul 27 17:27:37.007899 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.003938    2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 27 17:27:37.007899 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.003941    2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 27 17:27:37.007899 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.003944    2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 27 17:27:37.007899 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.003948    2573 flags.go:64] FLAG: --container-log-max-files="5"
Jul 27 17:27:37.007899 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.003951    2573 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 27 17:27:37.007899 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.003955    2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 27 17:27:37.008508 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.003958    2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 27 17:27:37.008508 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.003961    2573 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 27 17:27:37.008508 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.003965    2573 flags.go:64] FLAG: --contention-profiling="false"
Jul 27 17:27:37.008508 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.003968    2573 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 27 17:27:37.008508 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.003971    2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 27 17:27:37.008508 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.003974    2573 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 27 17:27:37.008508 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.003977    2573 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 27 17:27:37.008508 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.003981    2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 27 17:27:37.008508 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.003984    2573 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 27 17:27:37.008508 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.003987    2573 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 27 17:27:37.008508 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.003989    2573 flags.go:64] FLAG: --enable-load-reader="false"
Jul 27 17:27:37.008508 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.003992    2573 flags.go:64] FLAG: --enable-server="true"
Jul 27 17:27:37.008508 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.003995    2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 27 17:27:37.008508 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004000    2573 flags.go:64] FLAG: --event-burst="100"
Jul 27 17:27:37.008508 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004003    2573 flags.go:64] FLAG: --event-qps="50"
Jul 27 17:27:37.008508 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004006    2573 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 27 17:27:37.008508 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004009    2573 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 27 17:27:37.008508 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004012    2573 flags.go:64] FLAG: --eviction-hard=""
Jul 27 17:27:37.008508 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004016    2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 27 17:27:37.008508 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004019    2573 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 27 17:27:37.008508 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004021    2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 27 17:27:37.008508 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004025    2573 flags.go:64] FLAG: --eviction-soft=""
Jul 27 17:27:37.008508 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004028    2573 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 27 17:27:37.008508 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004030    2573 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 27 17:27:37.008508 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004034    2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 27 17:27:37.009202 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004037    2573 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 27 17:27:37.009202 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004040    2573 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 27 17:27:37.009202 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004043    2573 flags.go:64] FLAG: --fail-swap-on="true"
Jul 27 17:27:37.009202 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004046    2573 flags.go:64] FLAG: --feature-gates=""
Jul 27 17:27:37.009202 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004049    2573 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 27 17:27:37.009202 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004052    2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 27 17:27:37.009202 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004055    2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 27 17:27:37.009202 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004058    2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 27 17:27:37.009202 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004061    2573 flags.go:64] FLAG: --healthz-port="10248"
Jul 27 17:27:37.009202 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004064    2573 flags.go:64] FLAG: --help="false"
Jul 27 17:27:37.009202 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004067    2573 flags.go:64] FLAG: --hostname-override="ip-10-0-140-0.ec2.internal"
Jul 27 17:27:37.009202 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004070    2573 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 27 17:27:37.009202 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004073    2573 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 27 17:27:37.009202 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004076    2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 27 17:27:37.009202 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004079    2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 27 17:27:37.009202 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004083    2573 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 27 17:27:37.009202 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004086    2573 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 27 17:27:37.009202 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004088    2573 flags.go:64] FLAG: --image-service-endpoint=""
Jul 27 17:27:37.009202 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004091    2573 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 27 17:27:37.009202 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004094    2573 flags.go:64] FLAG: --kube-api-burst="100"
Jul 27 17:27:37.009202 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004097    2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 27 17:27:37.009202 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004100    2573 flags.go:64] FLAG: --kube-api-qps="50"
Jul 27 17:27:37.009202 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004102    2573 flags.go:64] FLAG: --kube-reserved=""
Jul 27 17:27:37.009202 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004105    2573 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 27 17:27:37.009830 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004108    2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 27 17:27:37.009830 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004111    2573 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 27 17:27:37.009830 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004114    2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 27 17:27:37.009830 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004116    2573 flags.go:64] FLAG: --lock-file=""
Jul 27 17:27:37.009830 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004122    2573 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 27 17:27:37.009830 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004125    2573 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 27 17:27:37.009830 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004128    2573 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 27 17:27:37.009830 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004133    2573 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 27 17:27:37.009830 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004137    2573 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 27 17:27:37.009830 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004140    2573 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 27 17:27:37.009830 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004142    2573 flags.go:64] FLAG: --logging-format="text"
Jul 27 17:27:37.009830 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004145    2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 27 17:27:37.009830 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004149    2573 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 27 17:27:37.009830 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004152    2573 flags.go:64] FLAG: --manifest-url=""
Jul 27 17:27:37.009830 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004155    2573 flags.go:64] FLAG: --manifest-url-header=""
Jul 27 17:27:37.009830 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004159    2573 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 27 17:27:37.009830 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004162    2573 flags.go:64] FLAG: --max-open-files="1000000"
Jul 27 17:27:37.009830 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004166    2573 flags.go:64] FLAG: --max-pods="110"
Jul 27 17:27:37.009830 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004169    2573 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 27 17:27:37.009830 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004172    2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 27 17:27:37.009830 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004174    2573 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 27 17:27:37.009830 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004177    2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 27 17:27:37.009830 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004180    2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 27 17:27:37.009830 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004183    2573 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 27 17:27:37.009830 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004186    2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 27 17:27:37.010479 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004193    2573 flags.go:64] FLAG: --node-status-max-images="50"
Jul 27 17:27:37.010479 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004196    2573 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 27 17:27:37.010479 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004199    2573 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 27 17:27:37.010479 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004202    2573 flags.go:64] FLAG: --pod-cidr=""
Jul 27 17:27:37.010479 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004205    2573 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 27 17:27:37.010479 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004210    2573 flags.go:64] FLAG: --pod-manifest-path=""
Jul 27 17:27:37.010479 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004213    2573 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 27 17:27:37.010479 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004216    2573 flags.go:64] FLAG: --pods-per-core="0"
Jul 27 17:27:37.010479 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004219    2573 flags.go:64] FLAG: --port="10250"
Jul 27 17:27:37.010479 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004222    2573 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 27 17:27:37.010479 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004226    2573 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-07d76add4174e68f5"
Jul 27 17:27:37.010479 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004233    2573 flags.go:64] FLAG: --qos-reserved=""
Jul 27 17:27:37.010479 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004236    2573 flags.go:64] FLAG: --read-only-port="10255"
Jul 27 17:27:37.010479 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004239    2573 flags.go:64] FLAG: --register-node="true"
Jul 27 17:27:37.010479 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004242    2573 flags.go:64] FLAG: --register-with-taints=""
Jul 27 17:27:37.010479 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004246    2573 flags.go:64] FLAG: --registry-burst="10"
Jul 27 17:27:37.010479 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004249    2573 flags.go:64] FLAG: --registry-qps="5"
Jul 27 17:27:37.010479 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004252    2573 flags.go:64] FLAG: --reserved-cpus=""
Jul 27 17:27:37.010479 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004255    2573 flags.go:64] FLAG: --reserved-memory=""
Jul 27 17:27:37.010479 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004259    2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 27 17:27:37.010479 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004262    2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 27 17:27:37.010479 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004265    2573 flags.go:64] FLAG: --rotate-certificates="false"
Jul 27 17:27:37.010479 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004268    2573 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 27 17:27:37.010479 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004271    2573 flags.go:64] FLAG: --runonce="false"
Jul 27 17:27:37.010479 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004274    2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 27 17:27:37.011143 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004277    2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 27 17:27:37.011143 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004280    2573 flags.go:64] FLAG: --seccomp-default="false"
Jul 27 17:27:37.011143 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004282    2573 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 27 17:27:37.011143 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004285    2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 27 17:27:37.011143 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004288    2573 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 27 17:27:37.011143 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004291    2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 27 17:27:37.011143 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004294    2573 flags.go:64] FLAG: --storage-driver-password="root"
Jul 27 17:27:37.011143 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004297    2573 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 27 17:27:37.011143 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004300    2573 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 27 17:27:37.011143 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004303    2573 flags.go:64] FLAG: --storage-driver-user="root"
Jul 27 17:27:37.011143 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004305    2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 27 17:27:37.011143 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004308    2573 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 27 17:27:37.011143 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004311    2573 flags.go:64] FLAG: --system-cgroups=""
Jul 27 17:27:37.011143 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004314    2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 27 17:27:37.011143 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004319    2573 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 27 17:27:37.011143 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004322    2573 flags.go:64] FLAG: --tls-cert-file=""
Jul 27 17:27:37.011143 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004329    2573 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 27 17:27:37.011143 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004333    2573 flags.go:64] FLAG: --tls-min-version=""
Jul 27 17:27:37.011143 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004338    2573 flags.go:64] FLAG: --tls-private-key-file=""
Jul 27 17:27:37.011143 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004341    2573 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 27 17:27:37.011143 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004344    2573 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 27 17:27:37.011143 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004347    2573 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 27 17:27:37.011143 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004350    2573 flags.go:64] FLAG: --v="2"
Jul 27 17:27:37.011143 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004354    2573 flags.go:64] FLAG: --version="false"
Jul 27 17:27:37.011143 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004359    2573 flags.go:64] FLAG: --vmodule=""
Jul 27 17:27:37.011774 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004363    2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 27 17:27:37.011774 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.004367    2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 27 17:27:37.011774 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004484    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 17:27:37.011774 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004487    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 17:27:37.011774 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004491    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 17:27:37.011774 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004494    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 17:27:37.011774 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004496    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 17:27:37.011774 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004499    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 17:27:37.011774 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004502    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 17:27:37.011774 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004504    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 17:27:37.011774 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004507    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 17:27:37.011774 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004509    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 17:27:37.011774 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004512    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 17:27:37.011774 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004514    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 17:27:37.011774 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004517    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 17:27:37.011774 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004519    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 17:27:37.011774 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004522    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 17:27:37.011774 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004524    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 17:27:37.011774 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004527    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 17:27:37.011774 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004529    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 17:27:37.012276 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004532    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 17:27:37.012276 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004534    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 17:27:37.012276 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004537    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 17:27:37.012276 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004541    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 17:27:37.012276 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004544    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 17:27:37.012276 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004548    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 17:27:37.012276 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004551    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 17:27:37.012276 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004553    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 17:27:37.012276 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004556    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 17:27:37.012276 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004558    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 17:27:37.012276 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004561    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 17:27:37.012276 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004564    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 17:27:37.012276 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004567    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 17:27:37.012276 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004569    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 17:27:37.012276 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004572    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 17:27:37.012276 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004576    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 17:27:37.012276 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004579    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 17:27:37.012276 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004583    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 17:27:37.012276 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004586    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 17:27:37.012757 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004588    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 17:27:37.012757 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004591    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 17:27:37.012757 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004594    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 17:27:37.012757 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004596    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 17:27:37.012757 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004599    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 17:27:37.012757 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004602    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 17:27:37.012757 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004604    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 17:27:37.012757 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004607    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 17:27:37.012757 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004609    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 17:27:37.012757 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004612    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 17:27:37.012757 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004615    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 17:27:37.012757 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004617    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 17:27:37.012757 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004620    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 17:27:37.012757 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004622    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 17:27:37.012757 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004625    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 17:27:37.012757 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004627    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 17:27:37.012757 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004631    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 17:27:37.012757 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004634    2573 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 17:27:37.012757 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004638    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 17:27:37.012757 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004641    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 17:27:37.013277 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004643    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 17:27:37.013277 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004646    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 17:27:37.013277 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004648    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 17:27:37.013277 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004651    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 17:27:37.013277 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004653    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 17:27:37.013277 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004656    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 17:27:37.013277 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004659    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 17:27:37.013277 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004662    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 17:27:37.013277 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004665    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 17:27:37.013277 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004680    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 17:27:37.013277 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004683    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 17:27:37.013277 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004686    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 17:27:37.013277 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004688    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 17:27:37.013277 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004691    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 17:27:37.013277 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004694    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 17:27:37.013277 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004698    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 17:27:37.013277 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004702    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 17:27:37.013277 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004705    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 17:27:37.013277 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004708    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 17:27:37.013767 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004711    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 17:27:37.013767 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004713    2573 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 17:27:37.013767 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004716    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 17:27:37.013767 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004718    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 17:27:37.013767 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004721    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 17:27:37.013767 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004724    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 17:27:37.013767 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004728    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 17:27:37.013767 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004731    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 17:27:37.013767 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004735    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 17:27:37.013767 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004739    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 17:27:37.013767 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004742    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 17:27:37.013767 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004746    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 17:27:37.013767 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004749    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 17:27:37.013767 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004752    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 17:27:37.013767 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004755    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 17:27:37.013767 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004757    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 17:27:37.013767 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004760    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 17:27:37.013767 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004763    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 17:27:37.013767 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004766    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 17:27:37.013767 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004768    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 17:27:37.014289 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004772    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 17:27:37.014289 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004774    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 17:27:37.014289 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.004777    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 17:27:37.014289 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.005346    2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 17:27:37.014289 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.012141    2573 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 27 17:27:37.014289 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.012158    2573 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 27 17:27:37.014289 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.013174    2573 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 27 17:27:37.014289 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.013183    2573 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 27 17:27:37.014289 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013244    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 17:27:37.014289 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013249    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 17:27:37.014289 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013252    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 17:27:37.014289 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013256    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 17:27:37.014289 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013259    2573 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 17:27:37.014289 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013263    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 17:27:37.014289 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013266    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 17:27:37.014737 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013269    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 17:27:37.014737 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013272    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 17:27:37.014737 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013274    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 17:27:37.014737 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013277    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 17:27:37.014737 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013281    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 17:27:37.014737 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013285    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 17:27:37.014737 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013288    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 17:27:37.014737 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013291    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 17:27:37.014737 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013294    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 17:27:37.014737 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013298    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 17:27:37.014737 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013301    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 17:27:37.014737 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013303    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 17:27:37.014737 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013306    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 17:27:37.014737 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013308    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 17:27:37.014737 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013311    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 17:27:37.014737 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013314    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 17:27:37.014737 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013317    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 17:27:37.014737 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013319    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 17:27:37.014737 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013322    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 17:27:37.014737 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013325    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 17:27:37.015233 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013328    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 17:27:37.015233 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013331    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 17:27:37.015233 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013335    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 17:27:37.015233 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013338    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 17:27:37.015233 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013341    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 17:27:37.015233 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013344    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 17:27:37.015233 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013347    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 17:27:37.015233 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013349    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 17:27:37.015233 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013352    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 17:27:37.015233 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013354    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 17:27:37.015233 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013357    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 17:27:37.015233 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013360    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 17:27:37.015233 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013363    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 17:27:37.015233 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013365    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 17:27:37.015233 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013369    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 17:27:37.015233 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013371    2573 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 17:27:37.015233 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013374    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 17:27:37.015233 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013377    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 17:27:37.015233 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013380    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 17:27:37.015233 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013383    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 17:27:37.015742 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013385    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 17:27:37.015742 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013389    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 17:27:37.015742 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013393    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 17:27:37.015742 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013396    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 17:27:37.015742 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013398    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 17:27:37.015742 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013401    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 17:27:37.015742 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013403    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 17:27:37.015742 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013406    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 17:27:37.015742 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013409    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 17:27:37.015742 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013411    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 17:27:37.015742 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013414    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 17:27:37.015742 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013417    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 17:27:37.015742 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013419    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 17:27:37.015742 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013422    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 17:27:37.015742 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013425    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 17:27:37.015742 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013428    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 17:27:37.015742 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013431    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 17:27:37.015742 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013434    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 17:27:37.015742 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013436    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 17:27:37.016207 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013439    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 17:27:37.016207 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013442    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 17:27:37.016207 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013444    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 17:27:37.016207 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013447    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 17:27:37.016207 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013449    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 17:27:37.016207 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013451    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 17:27:37.016207 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013454    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 17:27:37.016207 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013457    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 17:27:37.016207 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013459    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 17:27:37.016207 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013462    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 17:27:37.016207 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013464    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 17:27:37.016207 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013467    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 17:27:37.016207 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013470    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 17:27:37.016207 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013474    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 17:27:37.016207 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013477    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 17:27:37.016207 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013479    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 17:27:37.016207 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013482    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 17:27:37.016207 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013485    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 17:27:37.016207 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013488    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 17:27:37.016207 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013491    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 17:27:37.016714 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013493    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 17:27:37.016714 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013496    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 17:27:37.016714 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013498    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 17:27:37.016714 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013501    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 17:27:37.016714 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013503    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 17:27:37.016714 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013506    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 17:27:37.016714 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013508    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 17:27:37.016714 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013512    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 17:27:37.016714 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013514    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 17:27:37.016714 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013518    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 17:27:37.016714 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013521    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 17:27:37.016714 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013523    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 17:27:37.016714 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013526    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 17:27:37.016714 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.013531    2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 17:27:37.016714 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013651    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 17:27:37.017091 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013657    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 17:27:37.017091 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013660    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 17:27:37.017091 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013662    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 17:27:37.017091 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013666    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 17:27:37.017091 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013682    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 17:27:37.017091 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013686    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 17:27:37.017091 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013689    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 17:27:37.017091 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013692    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 17:27:37.017091 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013695    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 17:27:37.017091 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013697    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 17:27:37.017091 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013700    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 17:27:37.017091 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013703    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 17:27:37.017091 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013705    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 17:27:37.017091 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013708    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 17:27:37.017091 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013710    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 17:27:37.017091 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013713    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 17:27:37.017091 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013715    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 17:27:37.017091 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013718    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 17:27:37.017091 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013720    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 17:27:37.017091 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013723    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 17:27:37.017584 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013725    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 17:27:37.017584 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013728    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 17:27:37.017584 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013730    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 17:27:37.017584 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013733    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 17:27:37.017584 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013736    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 17:27:37.017584 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013739    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 17:27:37.017584 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013742    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 17:27:37.017584 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013745    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 17:27:37.017584 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013748    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 17:27:37.017584 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013751    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 17:27:37.017584 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013754    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 17:27:37.017584 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013756    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 17:27:37.017584 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013759    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 17:27:37.017584 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013761    2573 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 17:27:37.017584 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013764    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 17:27:37.017584 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013767    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 17:27:37.017584 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013769    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 17:27:37.017584 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013772    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 17:27:37.017584 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013775    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 17:27:37.017584 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013778    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 17:27:37.018100 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013780    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 17:27:37.018100 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013783    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 17:27:37.018100 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013786    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 17:27:37.018100 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013788    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 17:27:37.018100 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013791    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 17:27:37.018100 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013793    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 17:27:37.018100 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013796    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 17:27:37.018100 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013798    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 17:27:37.018100 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013801    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 17:27:37.018100 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013803    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 17:27:37.018100 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013806    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 17:27:37.018100 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013808    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 17:27:37.018100 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013811    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 17:27:37.018100 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013813    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 17:27:37.018100 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013816    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 17:27:37.018100 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013818    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 17:27:37.018100 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013821    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 17:27:37.018100 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013824    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 17:27:37.018100 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013826    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 17:27:37.018573 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013829    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 17:27:37.018573 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013832    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 17:27:37.018573 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013835    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 17:27:37.018573 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013837    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 17:27:37.018573 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013840    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 17:27:37.018573 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013844    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 17:27:37.018573 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013847    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 17:27:37.018573 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013850    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 17:27:37.018573 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013853    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 17:27:37.018573 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013856    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 17:27:37.018573 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013859    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 17:27:37.018573 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013861    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 17:27:37.018573 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013864    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 17:27:37.018573 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013867    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 17:27:37.018573 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013869    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 17:27:37.018573 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013872    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 17:27:37.018573 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013875    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 17:27:37.018573 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013877    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 17:27:37.018573 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013880    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 17:27:37.019172 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013882    2573 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 17:27:37.019172 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013885    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 17:27:37.019172 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013887    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 17:27:37.019172 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013890    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 17:27:37.019172 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013893    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 17:27:37.019172 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013895    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 17:27:37.019172 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013898    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 17:27:37.019172 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013901    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 17:27:37.019172 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013905    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 17:27:37.019172 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013908    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 17:27:37.019172 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013910    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 17:27:37.019172 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013913    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 17:27:37.019172 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013915    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 17:27:37.019172 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013918    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 17:27:37.019172 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013920    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 17:27:37.019172 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013923    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 17:27:37.019172 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013926    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 17:27:37.019172 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013928    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 17:27:37.019172 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013931    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 17:27:37.019719 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:37.013934    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 17:27:37.019719 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.013939    2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 17:27:37.019719 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.014069    2573 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 27 17:27:37.019719 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.015992    2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 27 17:27:37.019719 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.017386    2573 server.go:1019] "Starting client certificate rotation"
Jul 27 17:27:37.019719 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.017493    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 27 17:27:37.019719 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.017552    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 27 17:27:37.019904 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.019784    2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 27 17:27:37.023357 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.023340    2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 27 17:27:37.032428 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.032406    2573 log.go:25] "Validated CRI v1 runtime API"
Jul 27 17:27:37.035267 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.035246    2573 log.go:25] "Validated CRI v1 image API"
Jul 27 17:27:37.036065 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.036044    2573 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 27 17:27:37.038598 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.038578    2573 fs.go:135] Filesystem UUIDs: map[51eb07c7-debf-4f25-b8ae-3d624e58ba78:/dev/nvme0n1p4 5dda343d-6184-4597-ab46-771d5b70f719:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Jul 27 17:27:37.038660 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.038598    2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 27 17:27:37.044140 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.044004    2573 manager.go:216] Machine: {Timestamp:2026-07-27 17:27:37.042719625 +0000 UTC m=+0.339698573 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100115 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23fbf5ca62ce5c0bdc51b87a1ad4ef SystemUUID:ec23fbf5-ca62-ce5c-0bdc-51b87a1ad4ef BootID:ffb0b60d-6de5-408f-9f4b-b5e8c23871ea Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:a4:ff:6d:8c:89 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:a4:ff:6d:8c:89 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f2:b0:4c:1f:ed:12 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 27 17:27:37.044140 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.044128    2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 27 17:27:37.044282 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:37.044272    2573 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 27 17:27:38.201077 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.201059    2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 27 17:27:38.201138 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.201071    2573 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 27 17:27:38.201316 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.201125    2573 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-0.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 27 17:27:38.201987 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.201975    2573 topology_manager.go:138] "Creating topology manager with none policy"
Jul 27 17:27:38.202042 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.201997    2573 container_manager_linux.go:309] "Creating device plugin manager"
Jul 27 17:27:38.202042 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.202014    2573 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 27 17:27:38.202987 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.202973    2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 27 17:27:38.203792 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.203780    2573 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 27 17:27:38.205724 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.205711    2573 state_mem.go:36] "Initialized new in-memory state store"
Jul 27 17:27:38.205867 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.205855    2573 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 27 17:27:38.205923 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.205893    2573 kubelet.go:486] "Attempting to sync node with API server"
Jul 27 17:27:38.205923 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.205909    2573 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 27 17:27:38.206010 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.205929    2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 27 17:27:38.206010 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.205942    2573 kubelet.go:398] "Adding apiserver pod source"
Jul 27 17:27:38.206010 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.205954    2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 27 17:27:38.206382 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.206369    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 27 17:27:38.206464 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.206393    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 27 17:27:38.208825 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.208811    2573 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 27 17:27:38.210732 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.210719    2573 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 27 17:27:38.210778 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.210749    2573 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 27 17:27:38.212705 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.212690    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 27 17:27:38.212705 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.212710    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 27 17:27:38.212705 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.212716    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 27 17:27:38.212904 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.212722    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 27 17:27:38.212904 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.212731    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 27 17:27:38.212904 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.212738    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 27 17:27:38.212904 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.212746    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 27 17:27:38.212904 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.212751    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 27 17:27:38.212904 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.212759    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 27 17:27:38.212904 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.212765    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 27 17:27:38.212904 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.212784    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 27 17:27:38.212904 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.212790    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 27 17:27:38.214541 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.214531    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 27 17:27:38.214541 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.214540    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 27 17:27:38.215704 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:38.215683    2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 27 17:27:38.215704 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:38.215686    2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-0.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 27 17:27:38.216168 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.216151    2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-0.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 27 17:27:38.217828 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.217815    2573 server.go:1268] "Started kubelet"
Jul 27 17:27:38.217982 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.217920    2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 27 17:27:38.218084 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.218006    2573 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 27 17:27:38.218084 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.217956    2573 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 27 17:27:38.220746 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.220720    2573 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 27 17:27:38.220762 ip-10-0-140-0 systemd[1]: Started Kubernetes Kubelet.
Jul 27 17:27:38.221748 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.221729    2573 server.go:310] "Adding debug handlers to kubelet server"
Jul 27 17:27:38.225113 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:38.225086    2573 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 27 17:27:38.227166 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.227147    2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xszw9"
Jul 27 17:27:38.229347 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.229327    2573 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 27 17:27:38.229421 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.229331    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 27 17:27:38.229543 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:38.229512    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-0.ec2.internal\" not found"
Jul 27 17:27:38.229933 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.229901    2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 27 17:27:38.230027 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.230009    2573 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 27 17:27:38.230082 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.230032    2573 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 27 17:27:38.230165 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.230150    2573 reconstruct.go:97] "Volume reconstruction finished"
Jul 27 17:27:38.230214 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.230165    2573 reconciler.go:29] "Reconciler: start to sync state"
Jul 27 17:27:38.231610 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.231591    2573 factory.go:55] Registering systemd factory
Jul 27 17:27:38.231704 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.231614    2573 factory.go:223] Registration of the systemd container factory successfully
Jul 27 17:27:38.232007 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.231980    2573 factory.go:153] Registering CRI-O factory
Jul 27 17:27:38.232007 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.232000    2573 factory.go:223] Registration of the crio container factory successfully
Jul 27 17:27:38.232153 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.232076    2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 27 17:27:38.232153 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.232100    2573 factory.go:103] Registering Raw factory
Jul 27 17:27:38.232153 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.232116    2573 manager.go:1196] Started watching for new ooms in manager
Jul 27 17:27:38.232548 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.232534    2573 manager.go:349] Starting recovery of all containers
Jul 27 17:27:38.235022 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:38.234998    2573 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-0.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 27 17:27:38.235022 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:38.234996    2573 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 27 17:27:38.236177 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:38.235095    2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-0.ec2.internal.18c6359ebe990d4d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-0.ec2.internal,UID:ip-10-0-140-0.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-0.ec2.internal,},FirstTimestamp:2026-07-27 17:27:38.217786701 +0000 UTC m=+0.361188743,LastTimestamp:2026-07-27 17:27:38.217786701 +0000 UTC m=+0.361188743,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-0.ec2.internal,}"
Jul 27 17:27:38.238513 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.238483    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xszw9"
Jul 27 17:27:38.241866 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.241845    2573 manager.go:354] Recovery completed
Jul 27 17:27:38.247109 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.247095    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 17:27:38.248946 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.248928    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-0.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 17:27:38.249021 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.248965    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-0.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 17:27:38.249021 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.248979    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-0.ec2.internal" event="NodeHasSufficientPID"
Jul 27 17:27:38.249648 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.249636    2573 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 27 17:27:38.249648 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.249646    2573 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 27 17:27:38.249648 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.249647    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 17:27:38.249818 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.249661    2573 state_mem.go:36] "Initialized new in-memory state store"
Jul 27 17:27:38.250412 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.250396    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-0.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 17:27:38.250502 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.250422    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-0.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 17:27:38.250502 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.250437    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-0.ec2.internal" event="NodeHasSufficientPID"
Jul 27 17:27:38.250576 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:38.250446    2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-0.ec2.internal.18c6359ec0748393  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-0.ec2.internal,UID:ip-10-0-140-0.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-0.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-0.ec2.internal,},FirstTimestamp:2026-07-27 17:27:38.248946579 +0000 UTC m=+0.392348626,LastTimestamp:2026-07-27 17:27:38.248946579 +0000 UTC m=+0.392348626,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-0.ec2.internal,}"
Jul 27 17:27:38.251160 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.251147    2573 policy_none.go:49] "None policy: Start"
Jul 27 17:27:38.251210 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.251163    2573 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 27 17:27:38.251210 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.251174    2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 27 17:27:38.252463 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.252451    2573 policy_none.go:47] "Start"
Jul 27 17:27:38.291957 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.291796    2573 manager.go:337] "Starting Device Plugin manager"
Jul 27 17:27:38.292084 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:38.291975    2573 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 27 17:27:38.292084 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.291989    2573 server.go:87] "Starting device plugin registration server"
Jul 27 17:27:38.292303 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.292289    2573 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 27 17:27:38.292370 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.292305    2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 27 17:27:38.292424 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.292411    2573 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 27 17:27:38.292522 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.292503    2573 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 27 17:27:38.292522 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.292519    2573 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 27 17:27:38.293041 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:38.293017    2573 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 27 17:27:38.293118 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:38.293076    2573 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-0.ec2.internal\" not found"
Jul 27 17:27:38.392491 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.392455    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 17:27:38.393995 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.393519    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-0.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 17:27:38.393995 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.393551    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-0.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 17:27:38.393995 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.393565    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-0.ec2.internal" event="NodeHasSufficientPID"
Jul 27 17:27:38.393995 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.393598    2573 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-0.ec2.internal"
Jul 27 17:27:38.403774 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.403719    2573 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-0.ec2.internal"
Jul 27 17:27:38.403774 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:38.403744    2573 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-0.ec2.internal\": node \"ip-10-0-140-0.ec2.internal\" not found"
Jul 27 17:27:38.418511 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:38.418486    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-0.ec2.internal\" not found"
Jul 27 17:27:38.461688 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.461647    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 27 17:27:38.462977 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.462950    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 27 17:27:38.463118 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.463002    2573 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 27 17:27:38.463118 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.463030    2573 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 27 17:27:38.463118 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:38.463075    2573 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 27 17:27:38.465856 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.465835    2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 17:27:38.518891 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:38.518861    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-0.ec2.internal\" not found"
Jul 27 17:27:38.563243 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.563185    2573 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-140-0.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-0.ec2.internal"]
Jul 27 17:27:38.563357 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.563307    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 17:27:38.565637 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.565618    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-0.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 17:27:38.565750 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.565656    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-0.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 17:27:38.565750 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.565668    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-0.ec2.internal" event="NodeHasSufficientPID"
Jul 27 17:27:38.566893 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.566876    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 17:27:38.567012 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.566994    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-0.ec2.internal"
Jul 27 17:27:38.567060 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.567045    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 17:27:38.567764 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.567744    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-0.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 17:27:38.567882 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.567779    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-0.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 17:27:38.567882 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.567794    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-0.ec2.internal" event="NodeHasSufficientPID"
Jul 27 17:27:38.567882 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.567748    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-0.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 17:27:38.567882 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.567842    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-0.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 17:27:38.567882 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.567853    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-0.ec2.internal" event="NodeHasSufficientPID"
Jul 27 17:27:38.568955 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.568942    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-0.ec2.internal"
Jul 27 17:27:38.569006 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.568968    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 17:27:38.569670 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.569654    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-0.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 17:27:38.569758 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.569676    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-0.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 17:27:38.569758 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.569685    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-0.ec2.internal" event="NodeHasSufficientPID"
Jul 27 17:27:38.593064 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:38.593037    2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-0.ec2.internal\" not found" node="ip-10-0-140-0.ec2.internal"
Jul 27 17:27:38.596495 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:38.596477    2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-0.ec2.internal\" not found" node="ip-10-0-140-0.ec2.internal"
Jul 27 17:27:38.619849 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:38.619822    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-0.ec2.internal\" not found"
Jul 27 17:27:38.631200 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.631178    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/99d4260fbe93baf5233f885e6c10aaab-config\") pod \"kube-apiserver-proxy-ip-10-0-140-0.ec2.internal\" (UID: \"99d4260fbe93baf5233f885e6c10aaab\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-0.ec2.internal"
Jul 27 17:27:38.631285 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.631205    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f3bba782fd46ee2e0fd21ec3413b04ad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-0.ec2.internal\" (UID: \"f3bba782fd46ee2e0fd21ec3413b04ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-0.ec2.internal"
Jul 27 17:27:38.631285 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.631223    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3bba782fd46ee2e0fd21ec3413b04ad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-0.ec2.internal\" (UID: \"f3bba782fd46ee2e0fd21ec3413b04ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-0.ec2.internal"
Jul 27 17:27:38.720280 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:38.720172    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-0.ec2.internal\" not found"
Jul 27 17:27:38.731508 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.731490    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f3bba782fd46ee2e0fd21ec3413b04ad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-0.ec2.internal\" (UID: \"f3bba782fd46ee2e0fd21ec3413b04ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-0.ec2.internal"
Jul 27 17:27:38.731561 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.731517    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f3bba782fd46ee2e0fd21ec3413b04ad-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-0.ec2.internal\" (UID: \"f3bba782fd46ee2e0fd21ec3413b04ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-0.ec2.internal"
Jul 27 17:27:38.731595 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.731579    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3bba782fd46ee2e0fd21ec3413b04ad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-0.ec2.internal\" (UID: \"f3bba782fd46ee2e0fd21ec3413b04ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-0.ec2.internal"
Jul 27 17:27:38.731627 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.731598    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/99d4260fbe93baf5233f885e6c10aaab-config\") pod \"kube-apiserver-proxy-ip-10-0-140-0.ec2.internal\" (UID: \"99d4260fbe93baf5233f885e6c10aaab\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-0.ec2.internal"
Jul 27 17:27:38.731661 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.731633    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/99d4260fbe93baf5233f885e6c10aaab-config\") pod \"kube-apiserver-proxy-ip-10-0-140-0.ec2.internal\" (UID: \"99d4260fbe93baf5233f885e6c10aaab\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-0.ec2.internal"
Jul 27 17:27:38.731695 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.731668    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3bba782fd46ee2e0fd21ec3413b04ad-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-0.ec2.internal\" (UID: \"f3bba782fd46ee2e0fd21ec3413b04ad\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-0.ec2.internal"
Jul 27 17:27:38.820662 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:38.820618    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-0.ec2.internal\" not found"
Jul 27 17:27:38.896174 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.896145    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-0.ec2.internal"
Jul 27 17:27:38.899138 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:38.899115    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-0.ec2.internal"
Jul 27 17:27:38.921389 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:38.921357    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-0.ec2.internal\" not found"
Jul 27 17:27:39.021984 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:39.021895    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-0.ec2.internal\" not found"
Jul 27 17:27:39.122437 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:39.122404    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-0.ec2.internal\" not found"
Jul 27 17:27:39.170871 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:39.170840    2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 27 17:27:39.171347 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:39.170986    2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 27 17:27:39.223226 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:39.223199    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-0.ec2.internal\" not found"
Jul 27 17:27:39.244665 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:39.244632    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-26 17:22:38 +0000 UTC" deadline="2028-01-17 19:22:43.397847183 +0000 UTC"
Jul 27 17:27:39.244665 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:39.244661    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12937h55m4.153189864s"
Jul 27 17:27:39.300134 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:39.300110    2573 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 17:27:39.323797 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:39.323761    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-0.ec2.internal\" not found"
Jul 27 17:27:39.403265 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:39.403224    2573 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 17:27:39.414265 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:39.414217    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3bba782fd46ee2e0fd21ec3413b04ad.slice/crio-4a292bcc38ab6ceb41ddc60ab18e7f993a1d98dab06870adcae14a19362c76cb WatchSource:0}: Error finding container 4a292bcc38ab6ceb41ddc60ab18e7f993a1d98dab06870adcae14a19362c76cb: Status 404 returned error can't find the container with id 4a292bcc38ab6ceb41ddc60ab18e7f993a1d98dab06870adcae14a19362c76cb
Jul 27 17:27:39.414628 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:39.414607    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99d4260fbe93baf5233f885e6c10aaab.slice/crio-269ac4afdab5c48d0a6eeda1a57ccfecc57f6332b0de27411faf25f05dc38c83 WatchSource:0}: Error finding container 269ac4afdab5c48d0a6eeda1a57ccfecc57f6332b0de27411faf25f05dc38c83: Status 404 returned error can't find the container with id 269ac4afdab5c48d0a6eeda1a57ccfecc57f6332b0de27411faf25f05dc38c83
Jul 27 17:27:39.420887 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:39.420871    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 17:27:39.424603 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:39.424587    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-0.ec2.internal\" not found"
Jul 27 17:27:39.466528 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:39.466473    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-0.ec2.internal" event={"ID":"99d4260fbe93baf5233f885e6c10aaab","Type":"ContainerStarted","Data":"269ac4afdab5c48d0a6eeda1a57ccfecc57f6332b0de27411faf25f05dc38c83"}
Jul 27 17:27:39.467461 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:39.467438    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-0.ec2.internal" event={"ID":"f3bba782fd46ee2e0fd21ec3413b04ad","Type":"ContainerStarted","Data":"4a292bcc38ab6ceb41ddc60ab18e7f993a1d98dab06870adcae14a19362c76cb"}
Jul 27 17:27:39.525649 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:39.525620    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-0.ec2.internal\" not found"
Jul 27 17:27:39.566361 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:39.566294    2573 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 17:27:39.629925 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:39.629903    2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-0.ec2.internal"
Jul 27 17:27:39.638704 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:39.638688    2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 27 17:27:39.639841 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:39.639828    2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-0.ec2.internal"
Jul 27 17:27:39.651191 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:39.651177    2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 27 17:27:40.206894 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.206856    2573 apiserver.go:52] "Watching apiserver"
Jul 27 17:27:40.215859 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.215833    2573 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 27 17:27:40.216213 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.216188    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-sqfbc","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l2h7k","openshift-cluster-node-tuning-operator/tuned-rmfml","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-0.ec2.internal","openshift-multus/network-metrics-daemon-zxdxx","openshift-network-operator/iptables-alerter-qgh42","openshift-ovn-kubernetes/ovnkube-node-vc2tz","kube-system/global-pull-secret-syncer-zr64w","kube-system/konnectivity-agent-67cbg","kube-system/kube-apiserver-proxy-ip-10-0-140-0.ec2.internal","openshift-dns/node-resolver-95qjs","openshift-image-registry/node-ca-zb877","openshift-multus/multus-additional-cni-plugins-sgsbl","openshift-multus/multus-p97fd"]
Jul 27 17:27:40.218752 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.218723    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sgsbl"
Jul 27 17:27:40.221357 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.221212    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l2h7k"
Jul 27 17:27:40.222074 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.221722    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 27 17:27:40.222074 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.221830    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wxqc9\""
Jul 27 17:27:40.222074 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.221853    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 27 17:27:40.222074 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.221940    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 27 17:27:40.222074 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.221953    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 27 17:27:40.222074 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.222053    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 27 17:27:40.223515 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.223496    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rmfml"
Jul 27 17:27:40.223775 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.223713    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 27 17:27:40.224041 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.224021    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 27 17:27:40.225575 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.224124    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-cjq9r\""
Jul 27 17:27:40.225575 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.224027    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 27 17:27:40.226263 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.226161    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 27 17:27:40.226358 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.226317    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sqfbc"
Jul 27 17:27:40.226427 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:40.226390    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sqfbc" podUID="1f0e6c35-0c8a-427b-9647-00055e7fa320"
Jul 27 17:27:40.226920 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.226671    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-whdr9\""
Jul 27 17:27:40.226920 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.226844    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 27 17:27:40.228585 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.228124    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxdxx"
Jul 27 17:27:40.228585 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:40.228184    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxdxx" podUID="2fedc794-294b-44d7-ba67-55dbad330693"
Jul 27 17:27:40.229703 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.229679    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 27 17:27:40.232018 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.231995    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qgh42"
Jul 27 17:27:40.234541 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.234516    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.235949 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.235926    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-2mm7r\""
Jul 27 17:27:40.236045 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.236030    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 27 17:27:40.236269 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.236237    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 27 17:27:40.236421 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.236400    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 27 17:27:40.237333 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.237239    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 27 17:27:40.240533 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.240448    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zr64w"
Jul 27 17:27:40.240615 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:40.240537    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zr64w" podUID="f3460441-b43e-430b-b5ec-21ea720fbd7f"
Jul 27 17:27:40.241036 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.241019    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-d9lxx\""
Jul 27 17:27:40.242196 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.242176    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 27 17:27:40.242830 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.242637    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 27 17:27:40.242942 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.242907    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 27 17:27:40.243114 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.243096    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 27 17:27:40.243772 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.243748    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvb8n\" (UniqueName: \"kubernetes.io/projected/ff92db47-ab64-4850-a9a7-51267958262f-kube-api-access-jvb8n\") pod \"aws-ebs-csi-driver-node-l2h7k\" (UID: \"ff92db47-ab64-4850-a9a7-51267958262f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l2h7k"
Jul 27 17:27:40.243849 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.243816    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwhzr\" (UniqueName: \"kubernetes.io/projected/1869a1a5-58a8-4839-a479-679491d1f97c-kube-api-access-mwhzr\") pod \"multus-additional-cni-plugins-sgsbl\" (UID: \"1869a1a5-58a8-4839-a479-679491d1f97c\") " pod="openshift-multus/multus-additional-cni-plugins-sgsbl"
Jul 27 17:27:40.243915 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.243850    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5682e309-a980-4018-8f8d-c942f245b0f1-etc-kubernetes\") pod \"tuned-rmfml\" (UID: \"5682e309-a980-4018-8f8d-c942f245b0f1\") " pod="openshift-cluster-node-tuning-operator/tuned-rmfml"
Jul 27 17:27:40.243915 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.243878    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5682e309-a980-4018-8f8d-c942f245b0f1-lib-modules\") pod \"tuned-rmfml\" (UID: \"5682e309-a980-4018-8f8d-c942f245b0f1\") " pod="openshift-cluster-node-tuning-operator/tuned-rmfml"
Jul 27 17:27:40.243915 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.243909    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5682e309-a980-4018-8f8d-c942f245b0f1-var-lib-kubelet\") pod \"tuned-rmfml\" (UID: \"5682e309-a980-4018-8f8d-c942f245b0f1\") " pod="openshift-cluster-node-tuning-operator/tuned-rmfml"
Jul 27 17:27:40.244115 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.243936    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmn25\" (UniqueName: \"kubernetes.io/projected/5682e309-a980-4018-8f8d-c942f245b0f1-kube-api-access-fmn25\") pod \"tuned-rmfml\" (UID: \"5682e309-a980-4018-8f8d-c942f245b0f1\") " pod="openshift-cluster-node-tuning-operator/tuned-rmfml"
Jul 27 17:27:40.244115 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.243960    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ff92db47-ab64-4850-a9a7-51267958262f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l2h7k\" (UID: \"ff92db47-ab64-4850-a9a7-51267958262f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l2h7k"
Jul 27 17:27:40.244115 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.243987    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5682e309-a980-4018-8f8d-c942f245b0f1-etc-sysconfig\") pod \"tuned-rmfml\" (UID: \"5682e309-a980-4018-8f8d-c942f245b0f1\") " pod="openshift-cluster-node-tuning-operator/tuned-rmfml"
Jul 27 17:27:40.244115 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.244016    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5682e309-a980-4018-8f8d-c942f245b0f1-etc-sysctl-conf\") pod \"tuned-rmfml\" (UID: \"5682e309-a980-4018-8f8d-c942f245b0f1\") " pod="openshift-cluster-node-tuning-operator/tuned-rmfml"
Jul 27 17:27:40.244115 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.244043    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5682e309-a980-4018-8f8d-c942f245b0f1-etc-systemd\") pod \"tuned-rmfml\" (UID: \"5682e309-a980-4018-8f8d-c942f245b0f1\") " pod="openshift-cluster-node-tuning-operator/tuned-rmfml"
Jul 27 17:27:40.244115 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.244069    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5682e309-a980-4018-8f8d-c942f245b0f1-host\") pod \"tuned-rmfml\" (UID: \"5682e309-a980-4018-8f8d-c942f245b0f1\") " pod="openshift-cluster-node-tuning-operator/tuned-rmfml"
Jul 27 17:27:40.244115 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.244095    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h76x8\" (UniqueName: \"kubernetes.io/projected/1f0e6c35-0c8a-427b-9647-00055e7fa320-kube-api-access-h76x8\") pod \"network-check-target-sqfbc\" (UID: \"1f0e6c35-0c8a-427b-9647-00055e7fa320\") " pod="openshift-network-diagnostics/network-check-target-sqfbc"
Jul 27 17:27:40.244559 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.244122    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9b682b7-4f6d-4f8e-ae7a-d6a6034859d2-host-slash\") pod \"iptables-alerter-qgh42\" (UID: \"d9b682b7-4f6d-4f8e-ae7a-d6a6034859d2\") " pod="openshift-network-operator/iptables-alerter-qgh42"
Jul 27 17:27:40.244559 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.244149    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ff92db47-ab64-4850-a9a7-51267958262f-etc-selinux\") pod \"aws-ebs-csi-driver-node-l2h7k\" (UID: \"ff92db47-ab64-4850-a9a7-51267958262f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l2h7k"
Jul 27 17:27:40.244559 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.244177    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5682e309-a980-4018-8f8d-c942f245b0f1-sys\") pod \"tuned-rmfml\" (UID: \"5682e309-a980-4018-8f8d-c942f245b0f1\") " pod="openshift-cluster-node-tuning-operator/tuned-rmfml"
Jul 27 17:27:40.244559 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.244198    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5682e309-a980-4018-8f8d-c942f245b0f1-tmp\") pod \"tuned-rmfml\" (UID: \"5682e309-a980-4018-8f8d-c942f245b0f1\") " pod="openshift-cluster-node-tuning-operator/tuned-rmfml"
Jul 27 17:27:40.244559 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.244227    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzptj\" (UniqueName: \"kubernetes.io/projected/2fedc794-294b-44d7-ba67-55dbad330693-kube-api-access-zzptj\") pod \"network-metrics-daemon-zxdxx\" (UID: \"2fedc794-294b-44d7-ba67-55dbad330693\") " pod="openshift-multus/network-metrics-daemon-zxdxx"
Jul 27 17:27:40.244559 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.244375    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1869a1a5-58a8-4839-a479-679491d1f97c-system-cni-dir\") pod \"multus-additional-cni-plugins-sgsbl\" (UID: \"1869a1a5-58a8-4839-a479-679491d1f97c\") " pod="openshift-multus/multus-additional-cni-plugins-sgsbl"
Jul 27 17:27:40.244559 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.244414    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1869a1a5-58a8-4839-a479-679491d1f97c-cnibin\") pod \"multus-additional-cni-plugins-sgsbl\" (UID: \"1869a1a5-58a8-4839-a479-679491d1f97c\") " pod="openshift-multus/multus-additional-cni-plugins-sgsbl"
Jul 27 17:27:40.244559 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.244446    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1869a1a5-58a8-4839-a479-679491d1f97c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sgsbl\" (UID: \"1869a1a5-58a8-4839-a479-679491d1f97c\") " pod="openshift-multus/multus-additional-cni-plugins-sgsbl"
Jul 27 17:27:40.244559 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.244491    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5682e309-a980-4018-8f8d-c942f245b0f1-etc-modprobe-d\") pod \"tuned-rmfml\" (UID: \"5682e309-a980-4018-8f8d-c942f245b0f1\") " pod="openshift-cluster-node-tuning-operator/tuned-rmfml"
Jul 27 17:27:40.244559 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.244517    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1869a1a5-58a8-4839-a479-679491d1f97c-cni-binary-copy\") pod \"multus-additional-cni-plugins-sgsbl\" (UID: \"1869a1a5-58a8-4839-a479-679491d1f97c\") " pod="openshift-multus/multus-additional-cni-plugins-sgsbl"
Jul 27 17:27:40.244559 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.244546    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1869a1a5-58a8-4839-a479-679491d1f97c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sgsbl\" (UID: \"1869a1a5-58a8-4839-a479-679491d1f97c\") " pod="openshift-multus/multus-additional-cni-plugins-sgsbl"
Jul 27 17:27:40.244973 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.244576    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d9b682b7-4f6d-4f8e-ae7a-d6a6034859d2-iptables-alerter-script\") pod \"iptables-alerter-qgh42\" (UID: \"d9b682b7-4f6d-4f8e-ae7a-d6a6034859d2\") " pod="openshift-network-operator/iptables-alerter-qgh42"
Jul 27 17:27:40.244973 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.244606    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1869a1a5-58a8-4839-a479-679491d1f97c-os-release\") pod \"multus-additional-cni-plugins-sgsbl\" (UID: \"1869a1a5-58a8-4839-a479-679491d1f97c\") " pod="openshift-multus/multus-additional-cni-plugins-sgsbl"
Jul 27 17:27:40.244973 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.244632    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ff92db47-ab64-4850-a9a7-51267958262f-sys-fs\") pod \"aws-ebs-csi-driver-node-l2h7k\" (UID: \"ff92db47-ab64-4850-a9a7-51267958262f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l2h7k"
Jul 27 17:27:40.244973 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.244655    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1869a1a5-58a8-4839-a479-679491d1f97c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sgsbl\" (UID: \"1869a1a5-58a8-4839-a479-679491d1f97c\") " pod="openshift-multus/multus-additional-cni-plugins-sgsbl"
Jul 27 17:27:40.244973 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.244684    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ff92db47-ab64-4850-a9a7-51267958262f-socket-dir\") pod \"aws-ebs-csi-driver-node-l2h7k\" (UID: \"ff92db47-ab64-4850-a9a7-51267958262f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l2h7k"
Jul 27 17:27:40.244973 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.244711    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ff92db47-ab64-4850-a9a7-51267958262f-registration-dir\") pod \"aws-ebs-csi-driver-node-l2h7k\" (UID: \"ff92db47-ab64-4850-a9a7-51267958262f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l2h7k"
Jul 27 17:27:40.244973 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.244740    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ff92db47-ab64-4850-a9a7-51267958262f-device-dir\") pod \"aws-ebs-csi-driver-node-l2h7k\" (UID: \"ff92db47-ab64-4850-a9a7-51267958262f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l2h7k"
Jul 27 17:27:40.244973 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.244801    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5682e309-a980-4018-8f8d-c942f245b0f1-etc-sysctl-d\") pod \"tuned-rmfml\" (UID: \"5682e309-a980-4018-8f8d-c942f245b0f1\") " pod="openshift-cluster-node-tuning-operator/tuned-rmfml"
Jul 27 17:27:40.244973 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.244826    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5682e309-a980-4018-8f8d-c942f245b0f1-run\") pod \"tuned-rmfml\" (UID: \"5682e309-a980-4018-8f8d-c942f245b0f1\") " pod="openshift-cluster-node-tuning-operator/tuned-rmfml"
Jul 27 17:27:40.244973 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.244855    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5682e309-a980-4018-8f8d-c942f245b0f1-etc-tuned\") pod \"tuned-rmfml\" (UID: \"5682e309-a980-4018-8f8d-c942f245b0f1\") " pod="openshift-cluster-node-tuning-operator/tuned-rmfml"
Jul 27 17:27:40.244973 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.244887    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fedc794-294b-44d7-ba67-55dbad330693-metrics-certs\") pod \"network-metrics-daemon-zxdxx\" (UID: \"2fedc794-294b-44d7-ba67-55dbad330693\") " pod="openshift-multus/network-metrics-daemon-zxdxx"
Jul 27 17:27:40.244973 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.244920    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvb52\" (UniqueName: \"kubernetes.io/projected/d9b682b7-4f6d-4f8e-ae7a-d6a6034859d2-kube-api-access-rvb52\") pod \"iptables-alerter-qgh42\" (UID: \"d9b682b7-4f6d-4f8e-ae7a-d6a6034859d2\") " pod="openshift-network-operator/iptables-alerter-qgh42"
Jul 27 17:27:40.247848 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.245938    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-67cbg"
Jul 27 17:27:40.247848 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.245951    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-p97fd"
Jul 27 17:27:40.247848 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.247166    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 27 17:27:40.247848 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.247708    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-95qjs"
Jul 27 17:27:40.248038 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.248025    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zb877"
Jul 27 17:27:40.250792 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.250111    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 27 17:27:40.250792 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.250301    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-95v77\""
Jul 27 17:27:40.250792 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.250390    2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 27 17:27:40.250792 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.250591    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 27 17:27:40.250792 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.250633    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-n22gm\""
Jul 27 17:27:40.250792 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.250670    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 27 17:27:40.251907 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.251582    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 27 17:27:40.251907 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.251833    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-lf44s\""
Jul 27 17:27:40.252656 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.252272    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 27 17:27:40.252656 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.252443    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 27 17:27:40.252656 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.252519    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 27 17:27:40.252841 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.252740    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 27 17:27:40.253201 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.253185    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rpqnb\""
Jul 27 17:27:40.268676 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.268648    2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5vj2d"
Jul 27 17:27:40.276981 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.276954    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5vj2d"
Jul 27 17:27:40.330861 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.330838    2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 27 17:27:40.345394 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.345361    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/87dac797-f8d2-4d36-999f-f8ce855dbeff-ovn-node-metrics-cert\") pod \"ovnkube-node-vc2tz\" (UID: \"87dac797-f8d2-4d36-999f-f8ce855dbeff\") " pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.345540 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.345413    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwhzr\" (UniqueName: \"kubernetes.io/projected/1869a1a5-58a8-4839-a479-679491d1f97c-kube-api-access-mwhzr\") pod \"multus-additional-cni-plugins-sgsbl\" (UID: \"1869a1a5-58a8-4839-a479-679491d1f97c\") " pod="openshift-multus/multus-additional-cni-plugins-sgsbl"
Jul 27 17:27:40.345540 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.345446    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5682e309-a980-4018-8f8d-c942f245b0f1-etc-kubernetes\") pod \"tuned-rmfml\" (UID: \"5682e309-a980-4018-8f8d-c942f245b0f1\") " pod="openshift-cluster-node-tuning-operator/tuned-rmfml"
Jul 27 17:27:40.345540 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.345472    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmn25\" (UniqueName: \"kubernetes.io/projected/5682e309-a980-4018-8f8d-c942f245b0f1-kube-api-access-fmn25\") pod \"tuned-rmfml\" (UID: \"5682e309-a980-4018-8f8d-c942f245b0f1\") " pod="openshift-cluster-node-tuning-operator/tuned-rmfml"
Jul 27 17:27:40.345540 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.345496    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2802ee72-4be9-423a-a9bd-102c54f96d06-host\") pod \"node-ca-zb877\" (UID: \"2802ee72-4be9-423a-a9bd-102c54f96d06\") " pod="openshift-image-registry/node-ca-zb877"
Jul 27 17:27:40.345540 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.345521    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ff92db47-ab64-4850-a9a7-51267958262f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l2h7k\" (UID: \"ff92db47-ab64-4850-a9a7-51267958262f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l2h7k"
Jul 27 17:27:40.345696 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.345544    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5682e309-a980-4018-8f8d-c942f245b0f1-etc-sysconfig\") pod \"tuned-rmfml\" (UID: \"5682e309-a980-4018-8f8d-c942f245b0f1\") " pod="openshift-cluster-node-tuning-operator/tuned-rmfml"
Jul 27 17:27:40.345696 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.345558    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5682e309-a980-4018-8f8d-c942f245b0f1-etc-kubernetes\") pod \"tuned-rmfml\" (UID: \"5682e309-a980-4018-8f8d-c942f245b0f1\") " pod="openshift-cluster-node-tuning-operator/tuned-rmfml"
Jul 27 17:27:40.345696 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.345567    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5682e309-a980-4018-8f8d-c942f245b0f1-etc-sysctl-conf\") pod \"tuned-rmfml\" (UID: \"5682e309-a980-4018-8f8d-c942f245b0f1\") " pod="openshift-cluster-node-tuning-operator/tuned-rmfml"
Jul 27 17:27:40.345696 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.345612    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5682e309-a980-4018-8f8d-c942f245b0f1-etc-systemd\") pod \"tuned-rmfml\" (UID: \"5682e309-a980-4018-8f8d-c942f245b0f1\") " pod="openshift-cluster-node-tuning-operator/tuned-rmfml"
Jul 27 17:27:40.345696 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.345638    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5682e309-a980-4018-8f8d-c942f245b0f1-host\") pod \"tuned-rmfml\" (UID: \"5682e309-a980-4018-8f8d-c942f245b0f1\") " pod="openshift-cluster-node-tuning-operator/tuned-rmfml"
Jul 27 17:27:40.345696 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.345666    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87dac797-f8d2-4d36-999f-f8ce855dbeff-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vc2tz\" (UID: \"87dac797-f8d2-4d36-999f-f8ce855dbeff\") " pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.345696 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.345690    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wlmf\" (UniqueName: \"kubernetes.io/projected/2802ee72-4be9-423a-a9bd-102c54f96d06-kube-api-access-2wlmf\") pod \"node-ca-zb877\" (UID: \"2802ee72-4be9-423a-a9bd-102c54f96d06\") " pod="openshift-image-registry/node-ca-zb877"
Jul 27 17:27:40.345898 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.345716    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5682e309-a980-4018-8f8d-c942f245b0f1-etc-sysctl-conf\") pod \"tuned-rmfml\" (UID: \"5682e309-a980-4018-8f8d-c942f245b0f1\") " pod="openshift-cluster-node-tuning-operator/tuned-rmfml"
Jul 27 17:27:40.345898 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.345716    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9b682b7-4f6d-4f8e-ae7a-d6a6034859d2-host-slash\") pod \"iptables-alerter-qgh42\" (UID: \"d9b682b7-4f6d-4f8e-ae7a-d6a6034859d2\") " pod="openshift-network-operator/iptables-alerter-qgh42"
Jul 27 17:27:40.345898 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.345753    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9b682b7-4f6d-4f8e-ae7a-d6a6034859d2-host-slash\") pod \"iptables-alerter-qgh42\" (UID: \"d9b682b7-4f6d-4f8e-ae7a-d6a6034859d2\") " pod="openshift-network-operator/iptables-alerter-qgh42"
Jul 27 17:27:40.345898 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.345760    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ff92db47-ab64-4850-a9a7-51267958262f-etc-selinux\") pod \"aws-ebs-csi-driver-node-l2h7k\" (UID: \"ff92db47-ab64-4850-a9a7-51267958262f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l2h7k"
Jul 27 17:27:40.345898 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.345811    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5682e309-a980-4018-8f8d-c942f245b0f1-etc-systemd\") pod \"tuned-rmfml\" (UID: \"5682e309-a980-4018-8f8d-c942f245b0f1\") " pod="openshift-cluster-node-tuning-operator/tuned-rmfml"
Jul 27 17:27:40.345898 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.345853    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5682e309-a980-4018-8f8d-c942f245b0f1-host\") pod \"tuned-rmfml\" (UID: \"5682e309-a980-4018-8f8d-c942f245b0f1\") " pod="openshift-cluster-node-tuning-operator/tuned-rmfml"
Jul 27 17:27:40.345898 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.345883    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ff92db47-ab64-4850-a9a7-51267958262f-etc-selinux\") pod \"aws-ebs-csi-driver-node-l2h7k\" (UID: \"ff92db47-ab64-4850-a9a7-51267958262f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l2h7k"
Jul 27 17:27:40.345898 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.345895    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5682e309-a980-4018-8f8d-c942f245b0f1-sys\") pod \"tuned-rmfml\" (UID: \"5682e309-a980-4018-8f8d-c942f245b0f1\") " pod="openshift-cluster-node-tuning-operator/tuned-rmfml"
Jul 27 17:27:40.346238 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.345922    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/87dac797-f8d2-4d36-999f-f8ce855dbeff-systemd-units\") pod \"ovnkube-node-vc2tz\" (UID: \"87dac797-f8d2-4d36-999f-f8ce855dbeff\") " pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.346238 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.345949    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f3460441-b43e-430b-b5ec-21ea720fbd7f-kubelet-config\") pod \"global-pull-secret-syncer-zr64w\" (UID: \"f3460441-b43e-430b-b5ec-21ea720fbd7f\") " pod="kube-system/global-pull-secret-syncer-zr64w"
Jul 27 17:27:40.346238 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.345966    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5682e309-a980-4018-8f8d-c942f245b0f1-sys\") pod \"tuned-rmfml\" (UID: \"5682e309-a980-4018-8f8d-c942f245b0f1\") " pod="openshift-cluster-node-tuning-operator/tuned-rmfml"
Jul 27 17:27:40.346238 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.345975    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzptj\" (UniqueName: \"kubernetes.io/projected/2fedc794-294b-44d7-ba67-55dbad330693-kube-api-access-zzptj\") pod \"network-metrics-daemon-zxdxx\" (UID: \"2fedc794-294b-44d7-ba67-55dbad330693\") " pod="openshift-multus/network-metrics-daemon-zxdxx"
Jul 27 17:27:40.346238 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.345996    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ff92db47-ab64-4850-a9a7-51267958262f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l2h7k\" (UID: \"ff92db47-ab64-4850-a9a7-51267958262f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l2h7k"
Jul 27 17:27:40.346238 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.346002    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1869a1a5-58a8-4839-a479-679491d1f97c-system-cni-dir\") pod \"multus-additional-cni-plugins-sgsbl\" (UID: \"1869a1a5-58a8-4839-a479-679491d1f97c\") " pod="openshift-multus/multus-additional-cni-plugins-sgsbl"
Jul 27 17:27:40.346238 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.346019    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5682e309-a980-4018-8f8d-c942f245b0f1-etc-sysconfig\") pod \"tuned-rmfml\" (UID: \"5682e309-a980-4018-8f8d-c942f245b0f1\") " pod="openshift-cluster-node-tuning-operator/tuned-rmfml"
Jul 27 17:27:40.346238 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.346030    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1869a1a5-58a8-4839-a479-679491d1f97c-cnibin\") pod \"multus-additional-cni-plugins-sgsbl\" (UID: \"1869a1a5-58a8-4839-a479-679491d1f97c\") " pod="openshift-multus/multus-additional-cni-plugins-sgsbl"
Jul 27 17:27:40.346238 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.346058    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5682e309-a980-4018-8f8d-c942f245b0f1-etc-modprobe-d\") pod \"tuned-rmfml\" (UID: \"5682e309-a980-4018-8f8d-c942f245b0f1\") " pod="openshift-cluster-node-tuning-operator/tuned-rmfml"
Jul 27 17:27:40.346238 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.346061    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1869a1a5-58a8-4839-a479-679491d1f97c-system-cni-dir\") pod \"multus-additional-cni-plugins-sgsbl\" (UID: \"1869a1a5-58a8-4839-a479-679491d1f97c\") " pod="openshift-multus/multus-additional-cni-plugins-sgsbl"
Jul 27 17:27:40.346238 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.346087    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/87dac797-f8d2-4d36-999f-f8ce855dbeff-host-run-netns\") pod \"ovnkube-node-vc2tz\" (UID: \"87dac797-f8d2-4d36-999f-f8ce855dbeff\") " pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.346238 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.346093    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1869a1a5-58a8-4839-a479-679491d1f97c-cnibin\") pod \"multus-additional-cni-plugins-sgsbl\" (UID: \"1869a1a5-58a8-4839-a479-679491d1f97c\") " pod="openshift-multus/multus-additional-cni-plugins-sgsbl"
Jul 27 17:27:40.346238 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.346108    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cf35f819-6812-444e-a55c-b9fc24bc396e-etc-kubernetes\") pod \"multus-p97fd\" (UID: \"cf35f819-6812-444e-a55c-b9fc24bc396e\") " pod="openshift-multus/multus-p97fd"
Jul 27 17:27:40.346238 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.346152    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/87dac797-f8d2-4d36-999f-f8ce855dbeff-ovnkube-config\") pod \"ovnkube-node-vc2tz\" (UID: \"87dac797-f8d2-4d36-999f-f8ce855dbeff\") " pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.346238 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.346161    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5682e309-a980-4018-8f8d-c942f245b0f1-etc-modprobe-d\") pod \"tuned-rmfml\" (UID: \"5682e309-a980-4018-8f8d-c942f245b0f1\") " pod="openshift-cluster-node-tuning-operator/tuned-rmfml"
Jul 27 17:27:40.346238 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.346184    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1869a1a5-58a8-4839-a479-679491d1f97c-cni-binary-copy\") pod \"multus-additional-cni-plugins-sgsbl\" (UID: \"1869a1a5-58a8-4839-a479-679491d1f97c\") " pod="openshift-multus/multus-additional-cni-plugins-sgsbl"
Jul 27 17:27:40.346238 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.346224    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d9b682b7-4f6d-4f8e-ae7a-d6a6034859d2-iptables-alerter-script\") pod \"iptables-alerter-qgh42\" (UID: \"d9b682b7-4f6d-4f8e-ae7a-d6a6034859d2\") " pod="openshift-network-operator/iptables-alerter-qgh42"
Jul 27 17:27:40.347030 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.346275    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1869a1a5-58a8-4839-a479-679491d1f97c-os-release\") pod \"multus-additional-cni-plugins-sgsbl\" (UID: \"1869a1a5-58a8-4839-a479-679491d1f97c\") " pod="openshift-multus/multus-additional-cni-plugins-sgsbl"
Jul 27 17:27:40.347030 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.346300    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ff92db47-ab64-4850-a9a7-51267958262f-sys-fs\") pod \"aws-ebs-csi-driver-node-l2h7k\" (UID: \"ff92db47-ab64-4850-a9a7-51267958262f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l2h7k"
Jul 27 17:27:40.347030 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.346342    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/87dac797-f8d2-4d36-999f-f8ce855dbeff-ovnkube-script-lib\") pod \"ovnkube-node-vc2tz\" (UID: \"87dac797-f8d2-4d36-999f-f8ce855dbeff\") " pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.347030 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.346367    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f3460441-b43e-430b-b5ec-21ea720fbd7f-dbus\") pod \"global-pull-secret-syncer-zr64w\" (UID: \"f3460441-b43e-430b-b5ec-21ea720fbd7f\") " pod="kube-system/global-pull-secret-syncer-zr64w"
Jul 27 17:27:40.347030 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.346389    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cf35f819-6812-444e-a55c-b9fc24bc396e-system-cni-dir\") pod \"multus-p97fd\" (UID: \"cf35f819-6812-444e-a55c-b9fc24bc396e\") " pod="openshift-multus/multus-p97fd"
Jul 27 17:27:40.347030 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.346412    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5682e309-a980-4018-8f8d-c942f245b0f1-run\") pod \"tuned-rmfml\" (UID: \"5682e309-a980-4018-8f8d-c942f245b0f1\") " pod="openshift-cluster-node-tuning-operator/tuned-rmfml"
Jul 27 17:27:40.347030 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.346434    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87dac797-f8d2-4d36-999f-f8ce855dbeff-etc-openvswitch\") pod \"ovnkube-node-vc2tz\" (UID: \"87dac797-f8d2-4d36-999f-f8ce855dbeff\") " pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.347030 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.346459    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cf35f819-6812-444e-a55c-b9fc24bc396e-host-run-k8s-cni-cncf-io\") pod \"multus-p97fd\" (UID: \"cf35f819-6812-444e-a55c-b9fc24bc396e\") " pod="openshift-multus/multus-p97fd"
Jul 27 17:27:40.347030 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.346482    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f3460441-b43e-430b-b5ec-21ea720fbd7f-original-pull-secret\") pod \"global-pull-secret-syncer-zr64w\" (UID: \"f3460441-b43e-430b-b5ec-21ea720fbd7f\") " pod="kube-system/global-pull-secret-syncer-zr64w"
Jul 27 17:27:40.347030 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.346520    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gzxv\" (UniqueName: \"kubernetes.io/projected/e19a218b-06c1-4547-9f14-00d201dc6030-kube-api-access-8gzxv\") pod \"node-resolver-95qjs\" (UID: \"e19a218b-06c1-4547-9f14-00d201dc6030\") " pod="openshift-dns/node-resolver-95qjs"
Jul 27 17:27:40.347030 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.346545    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvb52\" (UniqueName: \"kubernetes.io/projected/d9b682b7-4f6d-4f8e-ae7a-d6a6034859d2-kube-api-access-rvb52\") pod \"iptables-alerter-qgh42\" (UID: \"d9b682b7-4f6d-4f8e-ae7a-d6a6034859d2\") " pod="openshift-network-operator/iptables-alerter-qgh42"
Jul 27 17:27:40.347030 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.346578    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvb8n\" (UniqueName: \"kubernetes.io/projected/ff92db47-ab64-4850-a9a7-51267958262f-kube-api-access-jvb8n\") pod \"aws-ebs-csi-driver-node-l2h7k\" (UID: \"ff92db47-ab64-4850-a9a7-51267958262f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l2h7k"
Jul 27 17:27:40.347030 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.346606    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cf35f819-6812-444e-a55c-b9fc24bc396e-os-release\") pod \"multus-p97fd\" (UID: \"cf35f819-6812-444e-a55c-b9fc24bc396e\") " pod="openshift-multus/multus-p97fd"
Jul 27 17:27:40.347030 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.346637    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5682e309-a980-4018-8f8d-c942f245b0f1-lib-modules\") pod \"tuned-rmfml\" (UID: \"5682e309-a980-4018-8f8d-c942f245b0f1\") " pod="openshift-cluster-node-tuning-operator/tuned-rmfml"
Jul 27 17:27:40.347030 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.346665    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5682e309-a980-4018-8f8d-c942f245b0f1-var-lib-kubelet\") pod \"tuned-rmfml\" (UID: \"5682e309-a980-4018-8f8d-c942f245b0f1\") " pod="openshift-cluster-node-tuning-operator/tuned-rmfml"
Jul 27 17:27:40.347030 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.346690    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87dac797-f8d2-4d36-999f-f8ce855dbeff-host-run-ovn-kubernetes\") pod \"ovnkube-node-vc2tz\" (UID: \"87dac797-f8d2-4d36-999f-f8ce855dbeff\") " pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.347030 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.346713    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/87dac797-f8d2-4d36-999f-f8ce855dbeff-host-cni-netd\") pod \"ovnkube-node-vc2tz\" (UID: \"87dac797-f8d2-4d36-999f-f8ce855dbeff\") " pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.347682 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.346736    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cf35f819-6812-444e-a55c-b9fc24bc396e-cnibin\") pod \"multus-p97fd\" (UID: \"cf35f819-6812-444e-a55c-b9fc24bc396e\") " pod="openshift-multus/multus-p97fd"
Jul 27 17:27:40.347682 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.346761    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/87dac797-f8d2-4d36-999f-f8ce855dbeff-host-slash\") pod \"ovnkube-node-vc2tz\" (UID: \"87dac797-f8d2-4d36-999f-f8ce855dbeff\") " pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.347682 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.346789    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d9b682b7-4f6d-4f8e-ae7a-d6a6034859d2-iptables-alerter-script\") pod \"iptables-alerter-qgh42\" (UID: \"d9b682b7-4f6d-4f8e-ae7a-d6a6034859d2\") " pod="openshift-network-operator/iptables-alerter-qgh42"
Jul 27 17:27:40.347682 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.346789    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1869a1a5-58a8-4839-a479-679491d1f97c-cni-binary-copy\") pod \"multus-additional-cni-plugins-sgsbl\" (UID: \"1869a1a5-58a8-4839-a479-679491d1f97c\") " pod="openshift-multus/multus-additional-cni-plugins-sgsbl"
Jul 27 17:27:40.347682 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.346801    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/87dac797-f8d2-4d36-999f-f8ce855dbeff-node-log\") pod \"ovnkube-node-vc2tz\" (UID: \"87dac797-f8d2-4d36-999f-f8ce855dbeff\") " pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.347682 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.346870    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1869a1a5-58a8-4839-a479-679491d1f97c-os-release\") pod \"multus-additional-cni-plugins-sgsbl\" (UID: \"1869a1a5-58a8-4839-a479-679491d1f97c\") " pod="openshift-multus/multus-additional-cni-plugins-sgsbl"
Jul 27 17:27:40.347682 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.346919    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sglr6\" (UniqueName: \"kubernetes.io/projected/87dac797-f8d2-4d36-999f-f8ce855dbeff-kube-api-access-sglr6\") pod \"ovnkube-node-vc2tz\" (UID: \"87dac797-f8d2-4d36-999f-f8ce855dbeff\") " pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.347682 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.346921    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5682e309-a980-4018-8f8d-c942f245b0f1-var-lib-kubelet\") pod \"tuned-rmfml\" (UID: \"5682e309-a980-4018-8f8d-c942f245b0f1\") " pod="openshift-cluster-node-tuning-operator/tuned-rmfml"
Jul 27 17:27:40.347682 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.346944    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ff92db47-ab64-4850-a9a7-51267958262f-sys-fs\") pod \"aws-ebs-csi-driver-node-l2h7k\" (UID: \"ff92db47-ab64-4850-a9a7-51267958262f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l2h7k"
Jul 27 17:27:40.347682 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.346950    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h76x8\" (UniqueName: \"kubernetes.io/projected/1f0e6c35-0c8a-427b-9647-00055e7fa320-kube-api-access-h76x8\") pod \"network-check-target-sqfbc\" (UID: \"1f0e6c35-0c8a-427b-9647-00055e7fa320\") " pod="openshift-network-diagnostics/network-check-target-sqfbc"
Jul 27 17:27:40.347682 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.346986    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5682e309-a980-4018-8f8d-c942f245b0f1-tmp\") pod \"tuned-rmfml\" (UID: \"5682e309-a980-4018-8f8d-c942f245b0f1\") " pod="openshift-cluster-node-tuning-operator/tuned-rmfml"
Jul 27 17:27:40.347682 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.347013    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87dac797-f8d2-4d36-999f-f8ce855dbeff-run-openvswitch\") pod \"ovnkube-node-vc2tz\" (UID: \"87dac797-f8d2-4d36-999f-f8ce855dbeff\") " pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.347682 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.347037    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cf35f819-6812-444e-a55c-b9fc24bc396e-cni-binary-copy\") pod \"multus-p97fd\" (UID: \"cf35f819-6812-444e-a55c-b9fc24bc396e\") " pod="openshift-multus/multus-p97fd"
Jul 27 17:27:40.347682 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.347067    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cf35f819-6812-444e-a55c-b9fc24bc396e-multus-socket-dir-parent\") pod \"multus-p97fd\" (UID: \"cf35f819-6812-444e-a55c-b9fc24bc396e\") " pod="openshift-multus/multus-p97fd"
Jul 27 17:27:40.347682 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.347295    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cf35f819-6812-444e-a55c-b9fc24bc396e-host-var-lib-cni-bin\") pod \"multus-p97fd\" (UID: \"cf35f819-6812-444e-a55c-b9fc24bc396e\") " pod="openshift-multus/multus-p97fd"
Jul 27 17:27:40.347682 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.347304    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5682e309-a980-4018-8f8d-c942f245b0f1-run\") pod \"tuned-rmfml\" (UID: \"5682e309-a980-4018-8f8d-c942f245b0f1\") " pod="openshift-cluster-node-tuning-operator/tuned-rmfml"
Jul 27 17:27:40.347682 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.347323    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cf35f819-6812-444e-a55c-b9fc24bc396e-host-var-lib-cni-multus\") pod \"multus-p97fd\" (UID: \"cf35f819-6812-444e-a55c-b9fc24bc396e\") " pod="openshift-multus/multus-p97fd"
Jul 27 17:27:40.348480 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.347376    2573 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 27 17:27:40.348480 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.347410    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cf35f819-6812-444e-a55c-b9fc24bc396e-hostroot\") pod \"multus-p97fd\" (UID: \"cf35f819-6812-444e-a55c-b9fc24bc396e\") " pod="openshift-multus/multus-p97fd"
Jul 27 17:27:40.348480 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.347431    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5682e309-a980-4018-8f8d-c942f245b0f1-lib-modules\") pod \"tuned-rmfml\" (UID: \"5682e309-a980-4018-8f8d-c942f245b0f1\") " pod="openshift-cluster-node-tuning-operator/tuned-rmfml"
Jul 27 17:27:40.348480 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.347435    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/87dac797-f8d2-4d36-999f-f8ce855dbeff-host-cni-bin\") pod \"ovnkube-node-vc2tz\" (UID: \"87dac797-f8d2-4d36-999f-f8ce855dbeff\") " pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.348480 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.347538    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1869a1a5-58a8-4839-a479-679491d1f97c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sgsbl\" (UID: \"1869a1a5-58a8-4839-a479-679491d1f97c\") " pod="openshift-multus/multus-additional-cni-plugins-sgsbl"
Jul 27 17:27:40.348480 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.347572    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/87dac797-f8d2-4d36-999f-f8ce855dbeff-host-kubelet\") pod \"ovnkube-node-vc2tz\" (UID: \"87dac797-f8d2-4d36-999f-f8ce855dbeff\") " pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.348480 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.347617    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87dac797-f8d2-4d36-999f-f8ce855dbeff-var-lib-openvswitch\") pod \"ovnkube-node-vc2tz\" (UID: \"87dac797-f8d2-4d36-999f-f8ce855dbeff\") " pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.348480 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.347686    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/87dac797-f8d2-4d36-999f-f8ce855dbeff-run-ovn\") pod \"ovnkube-node-vc2tz\" (UID: \"87dac797-f8d2-4d36-999f-f8ce855dbeff\") " pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.348480 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.347717    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/87dac797-f8d2-4d36-999f-f8ce855dbeff-log-socket\") pod \"ovnkube-node-vc2tz\" (UID: \"87dac797-f8d2-4d36-999f-f8ce855dbeff\") " pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.348480 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.347763    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cf35f819-6812-444e-a55c-b9fc24bc396e-multus-conf-dir\") pod \"multus-p97fd\" (UID: \"cf35f819-6812-444e-a55c-b9fc24bc396e\") " pod="openshift-multus/multus-p97fd"
Jul 27 17:27:40.348480 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.347789    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cf35f819-6812-444e-a55c-b9fc24bc396e-host-run-multus-certs\") pod \"multus-p97fd\" (UID: \"cf35f819-6812-444e-a55c-b9fc24bc396e\") " pod="openshift-multus/multus-p97fd"
Jul 27 17:27:40.348480 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.347813    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/87dac797-f8d2-4d36-999f-f8ce855dbeff-env-overrides\") pod \"ovnkube-node-vc2tz\" (UID: \"87dac797-f8d2-4d36-999f-f8ce855dbeff\") " pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.348480 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.347849    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1869a1a5-58a8-4839-a479-679491d1f97c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sgsbl\" (UID: \"1869a1a5-58a8-4839-a479-679491d1f97c\") " pod="openshift-multus/multus-additional-cni-plugins-sgsbl"
Jul 27 17:27:40.348480 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.347893    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/87dac797-f8d2-4d36-999f-f8ce855dbeff-run-systemd\") pod \"ovnkube-node-vc2tz\" (UID: \"87dac797-f8d2-4d36-999f-f8ce855dbeff\") " pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.348480 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.347923    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cf35f819-6812-444e-a55c-b9fc24bc396e-multus-cni-dir\") pod \"multus-p97fd\" (UID: \"cf35f819-6812-444e-a55c-b9fc24bc396e\") " pod="openshift-multus/multus-p97fd"
Jul 27 17:27:40.348480 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.347946    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e19a218b-06c1-4547-9f14-00d201dc6030-tmp-dir\") pod \"node-resolver-95qjs\" (UID: \"e19a218b-06c1-4547-9f14-00d201dc6030\") " pod="openshift-dns/node-resolver-95qjs"
Jul 27 17:27:40.348480 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.347975    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1869a1a5-58a8-4839-a479-679491d1f97c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sgsbl\" (UID: \"1869a1a5-58a8-4839-a479-679491d1f97c\") " pod="openshift-multus/multus-additional-cni-plugins-sgsbl"
Jul 27 17:27:40.349284 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.348000    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ff92db47-ab64-4850-a9a7-51267958262f-socket-dir\") pod \"aws-ebs-csi-driver-node-l2h7k\" (UID: \"ff92db47-ab64-4850-a9a7-51267958262f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l2h7k"
Jul 27 17:27:40.349284 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.348023    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ff92db47-ab64-4850-a9a7-51267958262f-registration-dir\") pod \"aws-ebs-csi-driver-node-l2h7k\" (UID: \"ff92db47-ab64-4850-a9a7-51267958262f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l2h7k"
Jul 27 17:27:40.349284 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.348037    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1869a1a5-58a8-4839-a479-679491d1f97c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sgsbl\" (UID: \"1869a1a5-58a8-4839-a479-679491d1f97c\") " pod="openshift-multus/multus-additional-cni-plugins-sgsbl"
Jul 27 17:27:40.349284 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.348051    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ff92db47-ab64-4850-a9a7-51267958262f-device-dir\") pod \"aws-ebs-csi-driver-node-l2h7k\" (UID: \"ff92db47-ab64-4850-a9a7-51267958262f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l2h7k"
Jul 27 17:27:40.349284 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.348077    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5682e309-a980-4018-8f8d-c942f245b0f1-etc-sysctl-d\") pod \"tuned-rmfml\" (UID: \"5682e309-a980-4018-8f8d-c942f245b0f1\") " pod="openshift-cluster-node-tuning-operator/tuned-rmfml"
Jul 27 17:27:40.349284 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.348078    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1869a1a5-58a8-4839-a479-679491d1f97c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sgsbl\" (UID: \"1869a1a5-58a8-4839-a479-679491d1f97c\") " pod="openshift-multus/multus-additional-cni-plugins-sgsbl"
Jul 27 17:27:40.349284 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.348098    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5682e309-a980-4018-8f8d-c942f245b0f1-etc-tuned\") pod \"tuned-rmfml\" (UID: \"5682e309-a980-4018-8f8d-c942f245b0f1\") " pod="openshift-cluster-node-tuning-operator/tuned-rmfml"
Jul 27 17:27:40.349284 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.348122    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e19a218b-06c1-4547-9f14-00d201dc6030-hosts-file\") pod \"node-resolver-95qjs\" (UID: \"e19a218b-06c1-4547-9f14-00d201dc6030\") " pod="openshift-dns/node-resolver-95qjs"
Jul 27 17:27:40.349284 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.348143    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ff92db47-ab64-4850-a9a7-51267958262f-registration-dir\") pod \"aws-ebs-csi-driver-node-l2h7k\" (UID: \"ff92db47-ab64-4850-a9a7-51267958262f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l2h7k"
Jul 27 17:27:40.349284 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.348307    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5682e309-a980-4018-8f8d-c942f245b0f1-etc-sysctl-d\") pod \"tuned-rmfml\" (UID: \"5682e309-a980-4018-8f8d-c942f245b0f1\") " pod="openshift-cluster-node-tuning-operator/tuned-rmfml"
Jul 27 17:27:40.349284 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.348357    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ff92db47-ab64-4850-a9a7-51267958262f-device-dir\") pod \"aws-ebs-csi-driver-node-l2h7k\" (UID: \"ff92db47-ab64-4850-a9a7-51267958262f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l2h7k"
Jul 27 17:27:40.349284 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.348409    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c6c489eb-66d0-484b-83b4-af557ac8a997-agent-certs\") pod \"konnectivity-agent-67cbg\" (UID: \"c6c489eb-66d0-484b-83b4-af557ac8a997\") " pod="kube-system/konnectivity-agent-67cbg"
Jul 27 17:27:40.349284 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.348447    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fedc794-294b-44d7-ba67-55dbad330693-metrics-certs\") pod \"network-metrics-daemon-zxdxx\" (UID: \"2fedc794-294b-44d7-ba67-55dbad330693\") " pod="openshift-multus/network-metrics-daemon-zxdxx"
Jul 27 17:27:40.349284 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.348484    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cf35f819-6812-444e-a55c-b9fc24bc396e-host-run-netns\") pod \"multus-p97fd\" (UID: \"cf35f819-6812-444e-a55c-b9fc24bc396e\") " pod="openshift-multus/multus-p97fd"
Jul 27 17:27:40.349284 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.348510    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf35f819-6812-444e-a55c-b9fc24bc396e-host-var-lib-kubelet\") pod \"multus-p97fd\" (UID: \"cf35f819-6812-444e-a55c-b9fc24bc396e\") " pod="openshift-multus/multus-p97fd"
Jul 27 17:27:40.349284 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.348532    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cf35f819-6812-444e-a55c-b9fc24bc396e-multus-daemon-config\") pod \"multus-p97fd\" (UID: \"cf35f819-6812-444e-a55c-b9fc24bc396e\") " pod="openshift-multus/multus-p97fd"
Jul 27 17:27:40.349284 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.348561    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1869a1a5-58a8-4839-a479-679491d1f97c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sgsbl\" (UID: \"1869a1a5-58a8-4839-a479-679491d1f97c\") " pod="openshift-multus/multus-additional-cni-plugins-sgsbl"
Jul 27 17:27:40.350102 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.348565    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fk88\" (UniqueName: \"kubernetes.io/projected/cf35f819-6812-444e-a55c-b9fc24bc396e-kube-api-access-5fk88\") pod \"multus-p97fd\" (UID: \"cf35f819-6812-444e-a55c-b9fc24bc396e\") " pod="openshift-multus/multus-p97fd"
Jul 27 17:27:40.350102 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.348602    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c6c489eb-66d0-484b-83b4-af557ac8a997-konnectivity-ca\") pod \"konnectivity-agent-67cbg\" (UID: \"c6c489eb-66d0-484b-83b4-af557ac8a997\") " pod="kube-system/konnectivity-agent-67cbg"
Jul 27 17:27:40.350102 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.348624    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2802ee72-4be9-423a-a9bd-102c54f96d06-serviceca\") pod \"node-ca-zb877\" (UID: \"2802ee72-4be9-423a-a9bd-102c54f96d06\") " pod="openshift-image-registry/node-ca-zb877"
Jul 27 17:27:40.350102 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:40.348639    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:27:40.350102 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:40.348713    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fedc794-294b-44d7-ba67-55dbad330693-metrics-certs podName:2fedc794-294b-44d7-ba67-55dbad330693 nodeName:}" failed. No retries permitted until 2026-07-27 17:27:40.848679717 +0000 UTC m=+2.992081750 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2fedc794-294b-44d7-ba67-55dbad330693-metrics-certs") pod "network-metrics-daemon-zxdxx" (UID: "2fedc794-294b-44d7-ba67-55dbad330693") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:27:40.350102 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.348851    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ff92db47-ab64-4850-a9a7-51267958262f-socket-dir\") pod \"aws-ebs-csi-driver-node-l2h7k\" (UID: \"ff92db47-ab64-4850-a9a7-51267958262f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l2h7k"
Jul 27 17:27:40.352240 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.352123    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5682e309-a980-4018-8f8d-c942f245b0f1-tmp\") pod \"tuned-rmfml\" (UID: \"5682e309-a980-4018-8f8d-c942f245b0f1\") " pod="openshift-cluster-node-tuning-operator/tuned-rmfml"
Jul 27 17:27:40.352240 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.352170    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5682e309-a980-4018-8f8d-c942f245b0f1-etc-tuned\") pod \"tuned-rmfml\" (UID: \"5682e309-a980-4018-8f8d-c942f245b0f1\") " pod="openshift-cluster-node-tuning-operator/tuned-rmfml"
Jul 27 17:27:40.356516 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:40.356496    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 17:27:40.356516 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:40.356517    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 17:27:40.356683 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:40.356528    2573 projected.go:196] Error preparing data for projected volume kube-api-access-h76x8 for pod openshift-network-diagnostics/network-check-target-sqfbc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:27:40.356683 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:40.356619    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f0e6c35-0c8a-427b-9647-00055e7fa320-kube-api-access-h76x8 podName:1f0e6c35-0c8a-427b-9647-00055e7fa320 nodeName:}" failed. No retries permitted until 2026-07-27 17:27:40.856601644 +0000 UTC m=+3.000003695 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-h76x8" (UniqueName: "kubernetes.io/projected/1f0e6c35-0c8a-427b-9647-00055e7fa320-kube-api-access-h76x8") pod "network-check-target-sqfbc" (UID: "1f0e6c35-0c8a-427b-9647-00055e7fa320") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:27:40.357847 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.357817    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwhzr\" (UniqueName: \"kubernetes.io/projected/1869a1a5-58a8-4839-a479-679491d1f97c-kube-api-access-mwhzr\") pod \"multus-additional-cni-plugins-sgsbl\" (UID: \"1869a1a5-58a8-4839-a479-679491d1f97c\") " pod="openshift-multus/multus-additional-cni-plugins-sgsbl"
Jul 27 17:27:40.358859 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.358834    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvb52\" (UniqueName: \"kubernetes.io/projected/d9b682b7-4f6d-4f8e-ae7a-d6a6034859d2-kube-api-access-rvb52\") pod \"iptables-alerter-qgh42\" (UID: \"d9b682b7-4f6d-4f8e-ae7a-d6a6034859d2\") " pod="openshift-network-operator/iptables-alerter-qgh42"
Jul 27 17:27:40.359432 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.359389    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvb8n\" (UniqueName: \"kubernetes.io/projected/ff92db47-ab64-4850-a9a7-51267958262f-kube-api-access-jvb8n\") pod \"aws-ebs-csi-driver-node-l2h7k\" (UID: \"ff92db47-ab64-4850-a9a7-51267958262f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l2h7k"
Jul 27 17:27:40.359632 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.359608    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmn25\" (UniqueName: \"kubernetes.io/projected/5682e309-a980-4018-8f8d-c942f245b0f1-kube-api-access-fmn25\") pod \"tuned-rmfml\" (UID: \"5682e309-a980-4018-8f8d-c942f245b0f1\") " pod="openshift-cluster-node-tuning-operator/tuned-rmfml"
Jul 27 17:27:40.361238 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.361209    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzptj\" (UniqueName: \"kubernetes.io/projected/2fedc794-294b-44d7-ba67-55dbad330693-kube-api-access-zzptj\") pod \"network-metrics-daemon-zxdxx\" (UID: \"2fedc794-294b-44d7-ba67-55dbad330693\") " pod="openshift-multus/network-metrics-daemon-zxdxx"
Jul 27 17:27:40.449393 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.449357    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/87dac797-f8d2-4d36-999f-f8ce855dbeff-ovnkube-script-lib\") pod \"ovnkube-node-vc2tz\" (UID: \"87dac797-f8d2-4d36-999f-f8ce855dbeff\") " pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.449592 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.449403    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f3460441-b43e-430b-b5ec-21ea720fbd7f-dbus\") pod \"global-pull-secret-syncer-zr64w\" (UID: \"f3460441-b43e-430b-b5ec-21ea720fbd7f\") " pod="kube-system/global-pull-secret-syncer-zr64w"
Jul 27 17:27:40.449592 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.449536    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cf35f819-6812-444e-a55c-b9fc24bc396e-system-cni-dir\") pod \"multus-p97fd\" (UID: \"cf35f819-6812-444e-a55c-b9fc24bc396e\") " pod="openshift-multus/multus-p97fd"
Jul 27 17:27:40.449592 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.449554    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f3460441-b43e-430b-b5ec-21ea720fbd7f-dbus\") pod \"global-pull-secret-syncer-zr64w\" (UID: \"f3460441-b43e-430b-b5ec-21ea720fbd7f\") " pod="kube-system/global-pull-secret-syncer-zr64w"
Jul 27 17:27:40.449592 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.449574    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87dac797-f8d2-4d36-999f-f8ce855dbeff-etc-openvswitch\") pod \"ovnkube-node-vc2tz\" (UID: \"87dac797-f8d2-4d36-999f-f8ce855dbeff\") " pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.449814 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.449600    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cf35f819-6812-444e-a55c-b9fc24bc396e-system-cni-dir\") pod \"multus-p97fd\" (UID: \"cf35f819-6812-444e-a55c-b9fc24bc396e\") " pod="openshift-multus/multus-p97fd"
Jul 27 17:27:40.449814 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.449600    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cf35f819-6812-444e-a55c-b9fc24bc396e-host-run-k8s-cni-cncf-io\") pod \"multus-p97fd\" (UID: \"cf35f819-6812-444e-a55c-b9fc24bc396e\") " pod="openshift-multus/multus-p97fd"
Jul 27 17:27:40.449814 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.449626    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f3460441-b43e-430b-b5ec-21ea720fbd7f-original-pull-secret\") pod \"global-pull-secret-syncer-zr64w\" (UID: \"f3460441-b43e-430b-b5ec-21ea720fbd7f\") " pod="kube-system/global-pull-secret-syncer-zr64w"
Jul 27 17:27:40.449814 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.449637    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87dac797-f8d2-4d36-999f-f8ce855dbeff-etc-openvswitch\") pod \"ovnkube-node-vc2tz\" (UID: \"87dac797-f8d2-4d36-999f-f8ce855dbeff\") " pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.449814 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.449670    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cf35f819-6812-444e-a55c-b9fc24bc396e-host-run-k8s-cni-cncf-io\") pod \"multus-p97fd\" (UID: \"cf35f819-6812-444e-a55c-b9fc24bc396e\") " pod="openshift-multus/multus-p97fd"
Jul 27 17:27:40.449814 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.449708    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gzxv\" (UniqueName: \"kubernetes.io/projected/e19a218b-06c1-4547-9f14-00d201dc6030-kube-api-access-8gzxv\") pod \"node-resolver-95qjs\" (UID: \"e19a218b-06c1-4547-9f14-00d201dc6030\") " pod="openshift-dns/node-resolver-95qjs"
Jul 27 17:27:40.449814 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:40.449725    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 17:27:40.449814 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.449736    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cf35f819-6812-444e-a55c-b9fc24bc396e-os-release\") pod \"multus-p97fd\" (UID: \"cf35f819-6812-444e-a55c-b9fc24bc396e\") " pod="openshift-multus/multus-p97fd"
Jul 27 17:27:40.449814 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.449764    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87dac797-f8d2-4d36-999f-f8ce855dbeff-host-run-ovn-kubernetes\") pod \"ovnkube-node-vc2tz\" (UID: \"87dac797-f8d2-4d36-999f-f8ce855dbeff\") " pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.449814 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:40.449791    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3460441-b43e-430b-b5ec-21ea720fbd7f-original-pull-secret podName:f3460441-b43e-430b-b5ec-21ea720fbd7f nodeName:}" failed. No retries permitted until 2026-07-27 17:27:40.9497721 +0000 UTC m=+3.093174134 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f3460441-b43e-430b-b5ec-21ea720fbd7f-original-pull-secret") pod "global-pull-secret-syncer-zr64w" (UID: "f3460441-b43e-430b-b5ec-21ea720fbd7f") : object "kube-system"/"original-pull-secret" not registered
Jul 27 17:27:40.449814 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.449797    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87dac797-f8d2-4d36-999f-f8ce855dbeff-host-run-ovn-kubernetes\") pod \"ovnkube-node-vc2tz\" (UID: \"87dac797-f8d2-4d36-999f-f8ce855dbeff\") " pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.450392 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.449824    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/87dac797-f8d2-4d36-999f-f8ce855dbeff-host-cni-netd\") pod \"ovnkube-node-vc2tz\" (UID: \"87dac797-f8d2-4d36-999f-f8ce855dbeff\") " pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.450392 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.449847    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cf35f819-6812-444e-a55c-b9fc24bc396e-cnibin\") pod \"multus-p97fd\" (UID: \"cf35f819-6812-444e-a55c-b9fc24bc396e\") " pod="openshift-multus/multus-p97fd"
Jul 27 17:27:40.450392 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.449859    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cf35f819-6812-444e-a55c-b9fc24bc396e-os-release\") pod \"multus-p97fd\" (UID: \"cf35f819-6812-444e-a55c-b9fc24bc396e\") " pod="openshift-multus/multus-p97fd"
Jul 27 17:27:40.450392 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.449869    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/87dac797-f8d2-4d36-999f-f8ce855dbeff-host-slash\") pod \"ovnkube-node-vc2tz\" (UID: \"87dac797-f8d2-4d36-999f-f8ce855dbeff\") " pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.450392 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.449898    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/87dac797-f8d2-4d36-999f-f8ce855dbeff-host-slash\") pod \"ovnkube-node-vc2tz\" (UID: \"87dac797-f8d2-4d36-999f-f8ce855dbeff\") " pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.450392 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.449914    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/87dac797-f8d2-4d36-999f-f8ce855dbeff-node-log\") pod \"ovnkube-node-vc2tz\" (UID: \"87dac797-f8d2-4d36-999f-f8ce855dbeff\") " pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.450392 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.449924    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/87dac797-f8d2-4d36-999f-f8ce855dbeff-host-cni-netd\") pod \"ovnkube-node-vc2tz\" (UID: \"87dac797-f8d2-4d36-999f-f8ce855dbeff\") " pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.450392 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.449941    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sglr6\" (UniqueName: \"kubernetes.io/projected/87dac797-f8d2-4d36-999f-f8ce855dbeff-kube-api-access-sglr6\") pod \"ovnkube-node-vc2tz\" (UID: \"87dac797-f8d2-4d36-999f-f8ce855dbeff\") " pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.450392 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.449947    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cf35f819-6812-444e-a55c-b9fc24bc396e-cnibin\") pod \"multus-p97fd\" (UID: \"cf35f819-6812-444e-a55c-b9fc24bc396e\") " pod="openshift-multus/multus-p97fd"
Jul 27 17:27:40.450392 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.449998    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87dac797-f8d2-4d36-999f-f8ce855dbeff-run-openvswitch\") pod \"ovnkube-node-vc2tz\" (UID: \"87dac797-f8d2-4d36-999f-f8ce855dbeff\") " pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.450392 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.450013    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/87dac797-f8d2-4d36-999f-f8ce855dbeff-node-log\") pod \"ovnkube-node-vc2tz\" (UID: \"87dac797-f8d2-4d36-999f-f8ce855dbeff\") " pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.450392 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.450035    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cf35f819-6812-444e-a55c-b9fc24bc396e-cni-binary-copy\") pod \"multus-p97fd\" (UID: \"cf35f819-6812-444e-a55c-b9fc24bc396e\") " pod="openshift-multus/multus-p97fd"
Jul 27 17:27:40.450392 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.450058    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87dac797-f8d2-4d36-999f-f8ce855dbeff-run-openvswitch\") pod \"ovnkube-node-vc2tz\" (UID: \"87dac797-f8d2-4d36-999f-f8ce855dbeff\") " pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.450392 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.450061    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cf35f819-6812-444e-a55c-b9fc24bc396e-multus-socket-dir-parent\") pod \"multus-p97fd\" (UID: \"cf35f819-6812-444e-a55c-b9fc24bc396e\") " pod="openshift-multus/multus-p97fd"
Jul 27 17:27:40.450392 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.450074    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/87dac797-f8d2-4d36-999f-f8ce855dbeff-ovnkube-script-lib\") pod \"ovnkube-node-vc2tz\" (UID: \"87dac797-f8d2-4d36-999f-f8ce855dbeff\") " pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.450392 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.450099    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cf35f819-6812-444e-a55c-b9fc24bc396e-host-var-lib-cni-bin\") pod \"multus-p97fd\" (UID: \"cf35f819-6812-444e-a55c-b9fc24bc396e\") " pod="openshift-multus/multus-p97fd"
Jul 27 17:27:40.450392 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.450110    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cf35f819-6812-444e-a55c-b9fc24bc396e-multus-socket-dir-parent\") pod \"multus-p97fd\" (UID: \"cf35f819-6812-444e-a55c-b9fc24bc396e\") " pod="openshift-multus/multus-p97fd"
Jul 27 17:27:40.450392 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.450123    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cf35f819-6812-444e-a55c-b9fc24bc396e-host-var-lib-cni-multus\") pod \"multus-p97fd\" (UID: \"cf35f819-6812-444e-a55c-b9fc24bc396e\") " pod="openshift-multus/multus-p97fd"
Jul 27 17:27:40.451223 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.450162    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cf35f819-6812-444e-a55c-b9fc24bc396e-host-var-lib-cni-bin\") pod \"multus-p97fd\" (UID: \"cf35f819-6812-444e-a55c-b9fc24bc396e\") " pod="openshift-multus/multus-p97fd"
Jul 27 17:27:40.451223 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.450162    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cf35f819-6812-444e-a55c-b9fc24bc396e-host-var-lib-cni-multus\") pod \"multus-p97fd\" (UID: \"cf35f819-6812-444e-a55c-b9fc24bc396e\") " pod="openshift-multus/multus-p97fd"
Jul 27 17:27:40.451223 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.450181    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cf35f819-6812-444e-a55c-b9fc24bc396e-hostroot\") pod \"multus-p97fd\" (UID: \"cf35f819-6812-444e-a55c-b9fc24bc396e\") " pod="openshift-multus/multus-p97fd"
Jul 27 17:27:40.451223 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.450204    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/87dac797-f8d2-4d36-999f-f8ce855dbeff-host-cni-bin\") pod \"ovnkube-node-vc2tz\" (UID: \"87dac797-f8d2-4d36-999f-f8ce855dbeff\") " pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.451223 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.450225    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cf35f819-6812-444e-a55c-b9fc24bc396e-hostroot\") pod \"multus-p97fd\" (UID: \"cf35f819-6812-444e-a55c-b9fc24bc396e\") " pod="openshift-multus/multus-p97fd"
Jul 27 17:27:40.451223 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.450227    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/87dac797-f8d2-4d36-999f-f8ce855dbeff-host-kubelet\") pod \"ovnkube-node-vc2tz\" (UID: \"87dac797-f8d2-4d36-999f-f8ce855dbeff\") " pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.451223 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.450273    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87dac797-f8d2-4d36-999f-f8ce855dbeff-var-lib-openvswitch\") pod \"ovnkube-node-vc2tz\" (UID: \"87dac797-f8d2-4d36-999f-f8ce855dbeff\") " pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.451223 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.450282    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/87dac797-f8d2-4d36-999f-f8ce855dbeff-host-kubelet\") pod \"ovnkube-node-vc2tz\" (UID: \"87dac797-f8d2-4d36-999f-f8ce855dbeff\") " pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.451223 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.450297    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/87dac797-f8d2-4d36-999f-f8ce855dbeff-run-ovn\") pod \"ovnkube-node-vc2tz\" (UID: \"87dac797-f8d2-4d36-999f-f8ce855dbeff\") " pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.451223 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.450298    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/87dac797-f8d2-4d36-999f-f8ce855dbeff-host-cni-bin\") pod \"ovnkube-node-vc2tz\" (UID: \"87dac797-f8d2-4d36-999f-f8ce855dbeff\") " pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.451223 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.450319    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/87dac797-f8d2-4d36-999f-f8ce855dbeff-log-socket\") pod \"ovnkube-node-vc2tz\" (UID: \"87dac797-f8d2-4d36-999f-f8ce855dbeff\") " pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.451223 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.450320    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87dac797-f8d2-4d36-999f-f8ce855dbeff-var-lib-openvswitch\") pod \"ovnkube-node-vc2tz\" (UID: \"87dac797-f8d2-4d36-999f-f8ce855dbeff\") " pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.451223 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.450836    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/87dac797-f8d2-4d36-999f-f8ce855dbeff-run-ovn\") pod \"ovnkube-node-vc2tz\" (UID: \"87dac797-f8d2-4d36-999f-f8ce855dbeff\") " pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.451223 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.450354    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/87dac797-f8d2-4d36-999f-f8ce855dbeff-log-socket\") pod \"ovnkube-node-vc2tz\" (UID: \"87dac797-f8d2-4d36-999f-f8ce855dbeff\") " pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.451223 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.451036    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cf35f819-6812-444e-a55c-b9fc24bc396e-multus-conf-dir\") pod \"multus-p97fd\" (UID: \"cf35f819-6812-444e-a55c-b9fc24bc396e\") " pod="openshift-multus/multus-p97fd"
Jul 27 17:27:40.451223 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.451085    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cf35f819-6812-444e-a55c-b9fc24bc396e-host-run-multus-certs\") pod \"multus-p97fd\" (UID: \"cf35f819-6812-444e-a55c-b9fc24bc396e\") " pod="openshift-multus/multus-p97fd"
Jul 27 17:27:40.451223 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.451117    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/87dac797-f8d2-4d36-999f-f8ce855dbeff-env-overrides\") pod \"ovnkube-node-vc2tz\" (UID: \"87dac797-f8d2-4d36-999f-f8ce855dbeff\") " pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.451223 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.451144    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/87dac797-f8d2-4d36-999f-f8ce855dbeff-run-systemd\") pod \"ovnkube-node-vc2tz\" (UID: \"87dac797-f8d2-4d36-999f-f8ce855dbeff\") " pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.452054 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.451191    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cf35f819-6812-444e-a55c-b9fc24bc396e-host-run-multus-certs\") pod \"multus-p97fd\" (UID: \"cf35f819-6812-444e-a55c-b9fc24bc396e\") " pod="openshift-multus/multus-p97fd"
Jul 27 17:27:40.452054 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.451236    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cf35f819-6812-444e-a55c-b9fc24bc396e-cni-binary-copy\") pod \"multus-p97fd\" (UID: \"cf35f819-6812-444e-a55c-b9fc24bc396e\") " pod="openshift-multus/multus-p97fd"
Jul 27 17:27:40.452054 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.451287    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cf35f819-6812-444e-a55c-b9fc24bc396e-multus-conf-dir\") pod \"multus-p97fd\" (UID: \"cf35f819-6812-444e-a55c-b9fc24bc396e\") " pod="openshift-multus/multus-p97fd"
Jul 27 17:27:40.452054 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.451413    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/87dac797-f8d2-4d36-999f-f8ce855dbeff-run-systemd\") pod \"ovnkube-node-vc2tz\" (UID: \"87dac797-f8d2-4d36-999f-f8ce855dbeff\") " pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.452054 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.451660    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/87dac797-f8d2-4d36-999f-f8ce855dbeff-env-overrides\") pod \"ovnkube-node-vc2tz\" (UID: \"87dac797-f8d2-4d36-999f-f8ce855dbeff\") " pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.452054 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.451767    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cf35f819-6812-444e-a55c-b9fc24bc396e-multus-cni-dir\") pod \"multus-p97fd\" (UID: \"cf35f819-6812-444e-a55c-b9fc24bc396e\") " pod="openshift-multus/multus-p97fd"
Jul 27 17:27:40.452054 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.451810    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e19a218b-06c1-4547-9f14-00d201dc6030-tmp-dir\") pod \"node-resolver-95qjs\" (UID: \"e19a218b-06c1-4547-9f14-00d201dc6030\") " pod="openshift-dns/node-resolver-95qjs"
Jul 27 17:27:40.452054 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.451851    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e19a218b-06c1-4547-9f14-00d201dc6030-hosts-file\") pod \"node-resolver-95qjs\" (UID: \"e19a218b-06c1-4547-9f14-00d201dc6030\") " pod="openshift-dns/node-resolver-95qjs"
Jul 27 17:27:40.452054 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.451893    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c6c489eb-66d0-484b-83b4-af557ac8a997-agent-certs\") pod \"konnectivity-agent-67cbg\" (UID: \"c6c489eb-66d0-484b-83b4-af557ac8a997\") " pod="kube-system/konnectivity-agent-67cbg"
Jul 27 17:27:40.452054 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.451939    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cf35f819-6812-444e-a55c-b9fc24bc396e-host-run-netns\") pod \"multus-p97fd\" (UID: \"cf35f819-6812-444e-a55c-b9fc24bc396e\") " pod="openshift-multus/multus-p97fd"
Jul 27 17:27:40.452054 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.451969    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf35f819-6812-444e-a55c-b9fc24bc396e-host-var-lib-kubelet\") pod \"multus-p97fd\" (UID: \"cf35f819-6812-444e-a55c-b9fc24bc396e\") " pod="openshift-multus/multus-p97fd"
Jul 27 17:27:40.452054 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.451998    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cf35f819-6812-444e-a55c-b9fc24bc396e-multus-daemon-config\") pod \"multus-p97fd\" (UID: \"cf35f819-6812-444e-a55c-b9fc24bc396e\") " pod="openshift-multus/multus-p97fd"
Jul 27 17:27:40.452054 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.452022    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fk88\" (UniqueName: \"kubernetes.io/projected/cf35f819-6812-444e-a55c-b9fc24bc396e-kube-api-access-5fk88\") pod \"multus-p97fd\" (UID: \"cf35f819-6812-444e-a55c-b9fc24bc396e\") " pod="openshift-multus/multus-p97fd"
Jul 27 17:27:40.452054 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.452051    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c6c489eb-66d0-484b-83b4-af557ac8a997-konnectivity-ca\") pod \"konnectivity-agent-67cbg\" (UID: \"c6c489eb-66d0-484b-83b4-af557ac8a997\") " pod="kube-system/konnectivity-agent-67cbg"
Jul 27 17:27:40.452666 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.452078    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2802ee72-4be9-423a-a9bd-102c54f96d06-serviceca\") pod \"node-ca-zb877\" (UID: \"2802ee72-4be9-423a-a9bd-102c54f96d06\") " pod="openshift-image-registry/node-ca-zb877"
Jul 27 17:27:40.452666 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.452105    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/87dac797-f8d2-4d36-999f-f8ce855dbeff-ovn-node-metrics-cert\") pod \"ovnkube-node-vc2tz\" (UID: \"87dac797-f8d2-4d36-999f-f8ce855dbeff\") " pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.452666 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.452137    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2802ee72-4be9-423a-a9bd-102c54f96d06-host\") pod \"node-ca-zb877\" (UID: \"2802ee72-4be9-423a-a9bd-102c54f96d06\") " pod="openshift-image-registry/node-ca-zb877"
Jul 27 17:27:40.452666 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.452168    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87dac797-f8d2-4d36-999f-f8ce855dbeff-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vc2tz\" (UID: \"87dac797-f8d2-4d36-999f-f8ce855dbeff\") " pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.452666 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.452200    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wlmf\" (UniqueName: \"kubernetes.io/projected/2802ee72-4be9-423a-a9bd-102c54f96d06-kube-api-access-2wlmf\") pod \"node-ca-zb877\" (UID: \"2802ee72-4be9-423a-a9bd-102c54f96d06\") " pod="openshift-image-registry/node-ca-zb877"
Jul 27 17:27:40.452666 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.452234    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/87dac797-f8d2-4d36-999f-f8ce855dbeff-systemd-units\") pod \"ovnkube-node-vc2tz\" (UID: \"87dac797-f8d2-4d36-999f-f8ce855dbeff\") " pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.452666 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.452281    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f3460441-b43e-430b-b5ec-21ea720fbd7f-kubelet-config\") pod \"global-pull-secret-syncer-zr64w\" (UID: \"f3460441-b43e-430b-b5ec-21ea720fbd7f\") " pod="kube-system/global-pull-secret-syncer-zr64w"
Jul 27 17:27:40.452666 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.452316    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/87dac797-f8d2-4d36-999f-f8ce855dbeff-host-run-netns\") pod \"ovnkube-node-vc2tz\" (UID: \"87dac797-f8d2-4d36-999f-f8ce855dbeff\") " pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.452666 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.452353    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cf35f819-6812-444e-a55c-b9fc24bc396e-etc-kubernetes\") pod \"multus-p97fd\" (UID: \"cf35f819-6812-444e-a55c-b9fc24bc396e\") " pod="openshift-multus/multus-p97fd"
Jul 27 17:27:40.452666 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.452384    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/87dac797-f8d2-4d36-999f-f8ce855dbeff-ovnkube-config\") pod \"ovnkube-node-vc2tz\" (UID: \"87dac797-f8d2-4d36-999f-f8ce855dbeff\") " pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.453157 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.452871    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/87dac797-f8d2-4d36-999f-f8ce855dbeff-ovnkube-config\") pod \"ovnkube-node-vc2tz\" (UID: \"87dac797-f8d2-4d36-999f-f8ce855dbeff\") " pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.453157 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.452988    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cf35f819-6812-444e-a55c-b9fc24bc396e-multus-cni-dir\") pod \"multus-p97fd\" (UID: \"cf35f819-6812-444e-a55c-b9fc24bc396e\") " pod="openshift-multus/multus-p97fd"
Jul 27 17:27:40.453814 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.453791    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e19a218b-06c1-4547-9f14-00d201dc6030-tmp-dir\") pod \"node-resolver-95qjs\" (UID: \"e19a218b-06c1-4547-9f14-00d201dc6030\") " pod="openshift-dns/node-resolver-95qjs"
Jul 27 17:27:40.453933 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.453818    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2802ee72-4be9-423a-a9bd-102c54f96d06-serviceca\") pod \"node-ca-zb877\" (UID: \"2802ee72-4be9-423a-a9bd-102c54f96d06\") " pod="openshift-image-registry/node-ca-zb877"
Jul 27 17:27:40.453933 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.453874    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87dac797-f8d2-4d36-999f-f8ce855dbeff-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vc2tz\" (UID: \"87dac797-f8d2-4d36-999f-f8ce855dbeff\") " pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.453933 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.453882    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2802ee72-4be9-423a-a9bd-102c54f96d06-host\") pod \"node-ca-zb877\" (UID: \"2802ee72-4be9-423a-a9bd-102c54f96d06\") " pod="openshift-image-registry/node-ca-zb877"
Jul 27 17:27:40.454075 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.453938    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cf35f819-6812-444e-a55c-b9fc24bc396e-host-var-lib-kubelet\") pod \"multus-p97fd\" (UID: \"cf35f819-6812-444e-a55c-b9fc24bc396e\") " pod="openshift-multus/multus-p97fd"
Jul 27 17:27:40.454075 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.454020    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/87dac797-f8d2-4d36-999f-f8ce855dbeff-systemd-units\") pod \"ovnkube-node-vc2tz\" (UID: \"87dac797-f8d2-4d36-999f-f8ce855dbeff\") " pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.454178 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.454072    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/87dac797-f8d2-4d36-999f-f8ce855dbeff-host-run-netns\") pod \"ovnkube-node-vc2tz\" (UID: \"87dac797-f8d2-4d36-999f-f8ce855dbeff\") " pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.454178 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.454132    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f3460441-b43e-430b-b5ec-21ea720fbd7f-kubelet-config\") pod \"global-pull-secret-syncer-zr64w\" (UID: \"f3460441-b43e-430b-b5ec-21ea720fbd7f\") " pod="kube-system/global-pull-secret-syncer-zr64w"
Jul 27 17:27:40.454303 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.454221    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cf35f819-6812-444e-a55c-b9fc24bc396e-host-run-netns\") pod \"multus-p97fd\" (UID: \"cf35f819-6812-444e-a55c-b9fc24bc396e\") " pod="openshift-multus/multus-p97fd"
Jul 27 17:27:40.454303 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.454238    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cf35f819-6812-444e-a55c-b9fc24bc396e-etc-kubernetes\") pod \"multus-p97fd\" (UID: \"cf35f819-6812-444e-a55c-b9fc24bc396e\") " pod="openshift-multus/multus-p97fd"
Jul 27 17:27:40.454412 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.454329    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e19a218b-06c1-4547-9f14-00d201dc6030-hosts-file\") pod \"node-resolver-95qjs\" (UID: \"e19a218b-06c1-4547-9f14-00d201dc6030\") " pod="openshift-dns/node-resolver-95qjs"
Jul 27 17:27:40.454712 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.454543    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cf35f819-6812-444e-a55c-b9fc24bc396e-multus-daemon-config\") pod \"multus-p97fd\" (UID: \"cf35f819-6812-444e-a55c-b9fc24bc396e\") " pod="openshift-multus/multus-p97fd"
Jul 27 17:27:40.454820 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.454750    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c6c489eb-66d0-484b-83b4-af557ac8a997-konnectivity-ca\") pod \"konnectivity-agent-67cbg\" (UID: \"c6c489eb-66d0-484b-83b4-af557ac8a997\") " pod="kube-system/konnectivity-agent-67cbg"
Jul 27 17:27:40.457298 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.457223    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/87dac797-f8d2-4d36-999f-f8ce855dbeff-ovn-node-metrics-cert\") pod \"ovnkube-node-vc2tz\" (UID: \"87dac797-f8d2-4d36-999f-f8ce855dbeff\") " pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.459451 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.459273    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c6c489eb-66d0-484b-83b4-af557ac8a997-agent-certs\") pod \"konnectivity-agent-67cbg\" (UID: \"c6c489eb-66d0-484b-83b4-af557ac8a997\") " pod="kube-system/konnectivity-agent-67cbg"
Jul 27 17:27:40.459559 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.459526    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sglr6\" (UniqueName: \"kubernetes.io/projected/87dac797-f8d2-4d36-999f-f8ce855dbeff-kube-api-access-sglr6\") pod \"ovnkube-node-vc2tz\" (UID: \"87dac797-f8d2-4d36-999f-f8ce855dbeff\") " pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.460507 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.460468    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gzxv\" (UniqueName: \"kubernetes.io/projected/e19a218b-06c1-4547-9f14-00d201dc6030-kube-api-access-8gzxv\") pod \"node-resolver-95qjs\" (UID: \"e19a218b-06c1-4547-9f14-00d201dc6030\") " pod="openshift-dns/node-resolver-95qjs"
Jul 27 17:27:40.462195 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.462169    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fk88\" (UniqueName: \"kubernetes.io/projected/cf35f819-6812-444e-a55c-b9fc24bc396e-kube-api-access-5fk88\") pod \"multus-p97fd\" (UID: \"cf35f819-6812-444e-a55c-b9fc24bc396e\") " pod="openshift-multus/multus-p97fd"
Jul 27 17:27:40.462571 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.462528    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wlmf\" (UniqueName: \"kubernetes.io/projected/2802ee72-4be9-423a-a9bd-102c54f96d06-kube-api-access-2wlmf\") pod \"node-ca-zb877\" (UID: \"2802ee72-4be9-423a-a9bd-102c54f96d06\") " pod="openshift-image-registry/node-ca-zb877"
Jul 27 17:27:40.531971 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.531933    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sgsbl"
Jul 27 17:27:40.548180 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.548148    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l2h7k"
Jul 27 17:27:40.565290 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.565265    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rmfml"
Jul 27 17:27:40.582171 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.582141    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qgh42"
Jul 27 17:27:40.589041 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.589017    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:27:40.598924 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.598895    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-p97fd"
Jul 27 17:27:40.606714 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.606690    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-67cbg"
Jul 27 17:27:40.614466 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.614435    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-95qjs"
Jul 27 17:27:40.621188 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.621155    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zb877"
Jul 27 17:27:40.699789 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.699753    2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 17:27:40.855693 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.855614    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fedc794-294b-44d7-ba67-55dbad330693-metrics-certs\") pod \"network-metrics-daemon-zxdxx\" (UID: \"2fedc794-294b-44d7-ba67-55dbad330693\") " pod="openshift-multus/network-metrics-daemon-zxdxx"
Jul 27 17:27:40.855856 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:40.855749    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:27:40.855856 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:40.855815    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fedc794-294b-44d7-ba67-55dbad330693-metrics-certs podName:2fedc794-294b-44d7-ba67-55dbad330693 nodeName:}" failed. No retries permitted until 2026-07-27 17:27:41.855795517 +0000 UTC m=+3.999197548 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2fedc794-294b-44d7-ba67-55dbad330693-metrics-certs") pod "network-metrics-daemon-zxdxx" (UID: "2fedc794-294b-44d7-ba67-55dbad330693") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:27:40.956620 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.956588    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f3460441-b43e-430b-b5ec-21ea720fbd7f-original-pull-secret\") pod \"global-pull-secret-syncer-zr64w\" (UID: \"f3460441-b43e-430b-b5ec-21ea720fbd7f\") " pod="kube-system/global-pull-secret-syncer-zr64w"
Jul 27 17:27:40.956780 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:40.956631    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h76x8\" (UniqueName: \"kubernetes.io/projected/1f0e6c35-0c8a-427b-9647-00055e7fa320-kube-api-access-h76x8\") pod \"network-check-target-sqfbc\" (UID: \"1f0e6c35-0c8a-427b-9647-00055e7fa320\") " pod="openshift-network-diagnostics/network-check-target-sqfbc"
Jul 27 17:27:40.956780 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:40.956709    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 17:27:40.956780 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:40.956740    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 17:27:40.956780 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:40.956752    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 17:27:40.956780 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:40.956761    2573 projected.go:196] Error preparing data for projected volume kube-api-access-h76x8 for pod openshift-network-diagnostics/network-check-target-sqfbc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:27:40.956780 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:40.956768    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3460441-b43e-430b-b5ec-21ea720fbd7f-original-pull-secret podName:f3460441-b43e-430b-b5ec-21ea720fbd7f nodeName:}" failed. No retries permitted until 2026-07-27 17:27:41.9567542 +0000 UTC m=+4.100156229 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f3460441-b43e-430b-b5ec-21ea720fbd7f-original-pull-secret") pod "global-pull-secret-syncer-zr64w" (UID: "f3460441-b43e-430b-b5ec-21ea720fbd7f") : object "kube-system"/"original-pull-secret" not registered
Jul 27 17:27:40.956988 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:40.956801    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f0e6c35-0c8a-427b-9647-00055e7fa320-kube-api-access-h76x8 podName:1f0e6c35-0c8a-427b-9647-00055e7fa320 nodeName:}" failed. No retries permitted until 2026-07-27 17:27:41.956790709 +0000 UTC m=+4.100192737 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-h76x8" (UniqueName: "kubernetes.io/projected/1f0e6c35-0c8a-427b-9647-00055e7fa320-kube-api-access-h76x8") pod "network-check-target-sqfbc" (UID: "1f0e6c35-0c8a-427b-9647-00055e7fa320") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:27:41.280571 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:41.280476    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 17:22:40 +0000 UTC" deadline="2028-01-17 18:52:23.746769722 +0000 UTC"
Jul 27 17:27:41.280571 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:41.280527    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12937h24m42.466246555s"
Jul 27 17:27:41.305737 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:41.305706    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9b682b7_4f6d_4f8e_ae7a_d6a6034859d2.slice/crio-a689eb3f43aa406530583f5d7715d90bb1115a91bd1f05cef07b73fe10d7d673 WatchSource:0}: Error finding container a689eb3f43aa406530583f5d7715d90bb1115a91bd1f05cef07b73fe10d7d673: Status 404 returned error can't find the container with id a689eb3f43aa406530583f5d7715d90bb1115a91bd1f05cef07b73fe10d7d673
Jul 27 17:27:41.308186 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:41.308147    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87dac797_f8d2_4d36_999f_f8ce855dbeff.slice/crio-ec411293ef451a6ae45379b0cf9089c32353e2ec64dba1032c6027d72ffcb655 WatchSource:0}: Error finding container ec411293ef451a6ae45379b0cf9089c32353e2ec64dba1032c6027d72ffcb655: Status 404 returned error can't find the container with id ec411293ef451a6ae45379b0cf9089c32353e2ec64dba1032c6027d72ffcb655
Jul 27 17:27:41.311151 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:41.311131    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode19a218b_06c1_4547_9f14_00d201dc6030.slice/crio-c5880b0316bb2bfc13556199bf671f51ea766c3ac6c0e4b807cb4ffe1326b6eb WatchSource:0}: Error finding container c5880b0316bb2bfc13556199bf671f51ea766c3ac6c0e4b807cb4ffe1326b6eb: Status 404 returned error can't find the container with id c5880b0316bb2bfc13556199bf671f51ea766c3ac6c0e4b807cb4ffe1326b6eb
Jul 27 17:27:41.332306 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:41.332033    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6c489eb_66d0_484b_83b4_af557ac8a997.slice/crio-b1f4778451b24a40ad05f35bf29b2393f7b89e5f583f63239ac84543bf831286 WatchSource:0}: Error finding container b1f4778451b24a40ad05f35bf29b2393f7b89e5f583f63239ac84543bf831286: Status 404 returned error can't find the container with id b1f4778451b24a40ad05f35bf29b2393f7b89e5f583f63239ac84543bf831286
Jul 27 17:27:41.333343 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:27:41.333216    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1869a1a5_58a8_4839_a479_679491d1f97c.slice/crio-2838cef611ec8467131ef3a80d377b98ab7a9aebdbfd0926efe44051d94d1868 WatchSource:0}: Error finding container 2838cef611ec8467131ef3a80d377b98ab7a9aebdbfd0926efe44051d94d1868: Status 404 returned error can't find the container with id 2838cef611ec8467131ef3a80d377b98ab7a9aebdbfd0926efe44051d94d1868
Jul 27 17:27:41.463556 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:41.463351    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sqfbc"
Jul 27 17:27:41.463712 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:41.463638    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sqfbc" podUID="1f0e6c35-0c8a-427b-9647-00055e7fa320"
Jul 27 17:27:41.472860 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:41.472800    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-0.ec2.internal" event={"ID":"99d4260fbe93baf5233f885e6c10aaab","Type":"ContainerStarted","Data":"b9721325159f577feb8a96a981a595c3acc430644c1027cf708103526fca8f8d"}
Jul 27 17:27:41.473980 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:41.473953    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-p97fd" event={"ID":"cf35f819-6812-444e-a55c-b9fc24bc396e","Type":"ContainerStarted","Data":"e13293aa02443359b7a6f185cedcf21936af5d12d27b4bc63dbcf8f6ea166d12"}
Jul 27 17:27:41.475037 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:41.475002    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l2h7k" event={"ID":"ff92db47-ab64-4850-a9a7-51267958262f","Type":"ContainerStarted","Data":"8f55925df22c2188e65a7b25c904c9dedd999281db222ff3d9277e7a486c5cfa"}
Jul 27 17:27:41.476117 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:41.476096    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zb877" event={"ID":"2802ee72-4be9-423a-a9bd-102c54f96d06","Type":"ContainerStarted","Data":"3f4ba9b8ed7d59352520595c8bbfe446bd783ee2dc0376f284cdf12b434bb1f5"}
Jul 27 17:27:41.477375 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:41.477349    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sgsbl" event={"ID":"1869a1a5-58a8-4839-a479-679491d1f97c","Type":"ContainerStarted","Data":"2838cef611ec8467131ef3a80d377b98ab7a9aebdbfd0926efe44051d94d1868"}
Jul 27 17:27:41.478426 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:41.478407    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-67cbg" event={"ID":"c6c489eb-66d0-484b-83b4-af557ac8a997","Type":"ContainerStarted","Data":"b1f4778451b24a40ad05f35bf29b2393f7b89e5f583f63239ac84543bf831286"}
Jul 27 17:27:41.479497 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:41.479461    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-95qjs" event={"ID":"e19a218b-06c1-4547-9f14-00d201dc6030","Type":"ContainerStarted","Data":"c5880b0316bb2bfc13556199bf671f51ea766c3ac6c0e4b807cb4ffe1326b6eb"}
Jul 27 17:27:41.480643 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:41.480617    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rmfml" event={"ID":"5682e309-a980-4018-8f8d-c942f245b0f1","Type":"ContainerStarted","Data":"beca4cb02c04038d8078b302699b5027c955a30ba3b27abbcb243cb9dd38cddf"}
Jul 27 17:27:41.481723 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:41.481697    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz" event={"ID":"87dac797-f8d2-4d36-999f-f8ce855dbeff","Type":"ContainerStarted","Data":"ec411293ef451a6ae45379b0cf9089c32353e2ec64dba1032c6027d72ffcb655"}
Jul 27 17:27:41.485367 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:41.485339    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qgh42" event={"ID":"d9b682b7-4f6d-4f8e-ae7a-d6a6034859d2","Type":"ContainerStarted","Data":"a689eb3f43aa406530583f5d7715d90bb1115a91bd1f05cef07b73fe10d7d673"}
Jul 27 17:27:41.488948 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:41.488909    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-0.ec2.internal" podStartSLOduration=2.488896812 podStartE2EDuration="2.488896812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:27:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:27:41.488378858 +0000 UTC m=+3.631780908" watchObservedRunningTime="2026-07-27 17:27:41.488896812 +0000 UTC m=+3.632298862"
Jul 27 17:27:41.864991 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:41.864880    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fedc794-294b-44d7-ba67-55dbad330693-metrics-certs\") pod \"network-metrics-daemon-zxdxx\" (UID: \"2fedc794-294b-44d7-ba67-55dbad330693\") " pod="openshift-multus/network-metrics-daemon-zxdxx"
Jul 27 17:27:41.865126 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:41.865019    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:27:41.865126 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:41.865104    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fedc794-294b-44d7-ba67-55dbad330693-metrics-certs podName:2fedc794-294b-44d7-ba67-55dbad330693 nodeName:}" failed. No retries permitted until 2026-07-27 17:27:43.865082618 +0000 UTC m=+6.008484665 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2fedc794-294b-44d7-ba67-55dbad330693-metrics-certs") pod "network-metrics-daemon-zxdxx" (UID: "2fedc794-294b-44d7-ba67-55dbad330693") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:27:41.966876 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:41.966226    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f3460441-b43e-430b-b5ec-21ea720fbd7f-original-pull-secret\") pod \"global-pull-secret-syncer-zr64w\" (UID: \"f3460441-b43e-430b-b5ec-21ea720fbd7f\") " pod="kube-system/global-pull-secret-syncer-zr64w"
Jul 27 17:27:41.966876 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:41.966318    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h76x8\" (UniqueName: \"kubernetes.io/projected/1f0e6c35-0c8a-427b-9647-00055e7fa320-kube-api-access-h76x8\") pod \"network-check-target-sqfbc\" (UID: \"1f0e6c35-0c8a-427b-9647-00055e7fa320\") " pod="openshift-network-diagnostics/network-check-target-sqfbc"
Jul 27 17:27:41.966876 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:41.966458    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 17:27:41.966876 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:41.966540    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3460441-b43e-430b-b5ec-21ea720fbd7f-original-pull-secret podName:f3460441-b43e-430b-b5ec-21ea720fbd7f nodeName:}" failed. No retries permitted until 2026-07-27 17:27:43.966519394 +0000 UTC m=+6.109921436 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f3460441-b43e-430b-b5ec-21ea720fbd7f-original-pull-secret") pod "global-pull-secret-syncer-zr64w" (UID: "f3460441-b43e-430b-b5ec-21ea720fbd7f") : object "kube-system"/"original-pull-secret" not registered
Jul 27 17:27:41.966876 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:41.966463    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 17:27:41.966876 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:41.966581    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 17:27:41.966876 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:41.966597    2573 projected.go:196] Error preparing data for projected volume kube-api-access-h76x8 for pod openshift-network-diagnostics/network-check-target-sqfbc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:27:41.966876 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:41.966649    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f0e6c35-0c8a-427b-9647-00055e7fa320-kube-api-access-h76x8 podName:1f0e6c35-0c8a-427b-9647-00055e7fa320 nodeName:}" failed. No retries permitted until 2026-07-27 17:27:43.966632535 +0000 UTC m=+6.110034567 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-h76x8" (UniqueName: "kubernetes.io/projected/1f0e6c35-0c8a-427b-9647-00055e7fa320-kube-api-access-h76x8") pod "network-check-target-sqfbc" (UID: "1f0e6c35-0c8a-427b-9647-00055e7fa320") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:27:42.280886 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:42.280756    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 17:22:40 +0000 UTC" deadline="2028-04-20 07:37:01.484347178 +0000 UTC"
Jul 27 17:27:42.280886 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:42.280810    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15182h9m19.20354137s"
Jul 27 17:27:42.465168 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:42.464459    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zr64w"
Jul 27 17:27:42.465168 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:42.464599    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zr64w" podUID="f3460441-b43e-430b-b5ec-21ea720fbd7f"
Jul 27 17:27:42.465702 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:42.465502    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxdxx"
Jul 27 17:27:42.465702 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:42.465623    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxdxx" podUID="2fedc794-294b-44d7-ba67-55dbad330693"
Jul 27 17:27:42.496273 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:42.495447    2573 generic.go:356] "Generic (PLEG): container finished" podID="f3bba782fd46ee2e0fd21ec3413b04ad" containerID="53adac0d2bd59da1b9e32dc889cec30ffb0431b8c7e9418e0b10096610615be5" exitCode=0
Jul 27 17:27:42.496273 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:42.496132    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-0.ec2.internal" event={"ID":"f3bba782fd46ee2e0fd21ec3413b04ad","Type":"ContainerDied","Data":"53adac0d2bd59da1b9e32dc889cec30ffb0431b8c7e9418e0b10096610615be5"}
Jul 27 17:27:43.463881 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:43.463836    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sqfbc"
Jul 27 17:27:43.464360 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:43.463962    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sqfbc" podUID="1f0e6c35-0c8a-427b-9647-00055e7fa320"
Jul 27 17:27:43.506511 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:43.505802    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-0.ec2.internal" event={"ID":"f3bba782fd46ee2e0fd21ec3413b04ad","Type":"ContainerStarted","Data":"e41aac5d9fef64ffa3b70f5c2b6ba88d599b0627bb4784d5c35d0ffc4fca8424"}
Jul 27 17:27:43.543240 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:43.542805    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-0.ec2.internal" podStartSLOduration=4.54278397 podStartE2EDuration="4.54278397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:27:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:27:43.541678257 +0000 UTC m=+5.685080308" watchObservedRunningTime="2026-07-27 17:27:43.54278397 +0000 UTC m=+5.686186022"
Jul 27 17:27:43.882561 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:43.881846    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fedc794-294b-44d7-ba67-55dbad330693-metrics-certs\") pod \"network-metrics-daemon-zxdxx\" (UID: \"2fedc794-294b-44d7-ba67-55dbad330693\") " pod="openshift-multus/network-metrics-daemon-zxdxx"
Jul 27 17:27:43.882561 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:43.882004    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:27:43.882561 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:43.882101    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fedc794-294b-44d7-ba67-55dbad330693-metrics-certs podName:2fedc794-294b-44d7-ba67-55dbad330693 nodeName:}" failed. No retries permitted until 2026-07-27 17:27:47.88205373 +0000 UTC m=+10.025455762 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2fedc794-294b-44d7-ba67-55dbad330693-metrics-certs") pod "network-metrics-daemon-zxdxx" (UID: "2fedc794-294b-44d7-ba67-55dbad330693") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:27:43.982854 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:43.982819    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f3460441-b43e-430b-b5ec-21ea720fbd7f-original-pull-secret\") pod \"global-pull-secret-syncer-zr64w\" (UID: \"f3460441-b43e-430b-b5ec-21ea720fbd7f\") " pod="kube-system/global-pull-secret-syncer-zr64w"
Jul 27 17:27:43.983052 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:43.982870    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h76x8\" (UniqueName: \"kubernetes.io/projected/1f0e6c35-0c8a-427b-9647-00055e7fa320-kube-api-access-h76x8\") pod \"network-check-target-sqfbc\" (UID: \"1f0e6c35-0c8a-427b-9647-00055e7fa320\") " pod="openshift-network-diagnostics/network-check-target-sqfbc"
Jul 27 17:27:43.983052 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:43.983023    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 17:27:43.983052 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:43.983044    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 17:27:43.983210 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:43.983058    2573 projected.go:196] Error preparing data for projected volume kube-api-access-h76x8 for pod openshift-network-diagnostics/network-check-target-sqfbc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:27:43.983210 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:43.983117    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f0e6c35-0c8a-427b-9647-00055e7fa320-kube-api-access-h76x8 podName:1f0e6c35-0c8a-427b-9647-00055e7fa320 nodeName:}" failed. No retries permitted until 2026-07-27 17:27:47.983099215 +0000 UTC m=+10.126501258 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-h76x8" (UniqueName: "kubernetes.io/projected/1f0e6c35-0c8a-427b-9647-00055e7fa320-kube-api-access-h76x8") pod "network-check-target-sqfbc" (UID: "1f0e6c35-0c8a-427b-9647-00055e7fa320") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:27:43.983725 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:43.983378    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 17:27:43.983725 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:43.983537    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3460441-b43e-430b-b5ec-21ea720fbd7f-original-pull-secret podName:f3460441-b43e-430b-b5ec-21ea720fbd7f nodeName:}" failed. No retries permitted until 2026-07-27 17:27:47.983487309 +0000 UTC m=+10.126889341 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f3460441-b43e-430b-b5ec-21ea720fbd7f-original-pull-secret") pod "global-pull-secret-syncer-zr64w" (UID: "f3460441-b43e-430b-b5ec-21ea720fbd7f") : object "kube-system"/"original-pull-secret" not registered
Jul 27 17:27:44.480069 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:44.479024    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxdxx"
Jul 27 17:27:44.480069 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:44.479344    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxdxx" podUID="2fedc794-294b-44d7-ba67-55dbad330693"
Jul 27 17:27:44.480069 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:44.479763    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zr64w"
Jul 27 17:27:44.480069 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:44.479896    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zr64w" podUID="f3460441-b43e-430b-b5ec-21ea720fbd7f"
Jul 27 17:27:45.464336 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:45.463860    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sqfbc"
Jul 27 17:27:45.464336 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:45.463989    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sqfbc" podUID="1f0e6c35-0c8a-427b-9647-00055e7fa320"
Jul 27 17:27:46.464901 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:46.464351    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zr64w"
Jul 27 17:27:46.464901 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:46.464482    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zr64w" podUID="f3460441-b43e-430b-b5ec-21ea720fbd7f"
Jul 27 17:27:46.465540 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:46.465514    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxdxx"
Jul 27 17:27:46.465664 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:46.465636    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxdxx" podUID="2fedc794-294b-44d7-ba67-55dbad330693"
Jul 27 17:27:47.463982 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:47.463950    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sqfbc"
Jul 27 17:27:47.464173 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:47.464093    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sqfbc" podUID="1f0e6c35-0c8a-427b-9647-00055e7fa320"
Jul 27 17:27:47.916483 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:47.916296    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fedc794-294b-44d7-ba67-55dbad330693-metrics-certs\") pod \"network-metrics-daemon-zxdxx\" (UID: \"2fedc794-294b-44d7-ba67-55dbad330693\") " pod="openshift-multus/network-metrics-daemon-zxdxx"
Jul 27 17:27:47.916483 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:47.916445    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:27:47.916983 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:47.916515    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fedc794-294b-44d7-ba67-55dbad330693-metrics-certs podName:2fedc794-294b-44d7-ba67-55dbad330693 nodeName:}" failed. No retries permitted until 2026-07-27 17:27:55.916495716 +0000 UTC m=+18.059897745 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2fedc794-294b-44d7-ba67-55dbad330693-metrics-certs") pod "network-metrics-daemon-zxdxx" (UID: "2fedc794-294b-44d7-ba67-55dbad330693") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:27:48.017288 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:48.017235    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f3460441-b43e-430b-b5ec-21ea720fbd7f-original-pull-secret\") pod \"global-pull-secret-syncer-zr64w\" (UID: \"f3460441-b43e-430b-b5ec-21ea720fbd7f\") " pod="kube-system/global-pull-secret-syncer-zr64w"
Jul 27 17:27:48.017489 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:48.017306    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h76x8\" (UniqueName: \"kubernetes.io/projected/1f0e6c35-0c8a-427b-9647-00055e7fa320-kube-api-access-h76x8\") pod \"network-check-target-sqfbc\" (UID: \"1f0e6c35-0c8a-427b-9647-00055e7fa320\") " pod="openshift-network-diagnostics/network-check-target-sqfbc"
Jul 27 17:27:48.017489 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:48.017411    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 17:27:48.017489 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:48.017449    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 17:27:48.017489 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:48.017466    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 17:27:48.017489 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:48.017479    2573 projected.go:196] Error preparing data for projected volume kube-api-access-h76x8 for pod openshift-network-diagnostics/network-check-target-sqfbc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:27:48.017489 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:48.017483    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3460441-b43e-430b-b5ec-21ea720fbd7f-original-pull-secret podName:f3460441-b43e-430b-b5ec-21ea720fbd7f nodeName:}" failed. No retries permitted until 2026-07-27 17:27:56.017464231 +0000 UTC m=+18.160866280 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f3460441-b43e-430b-b5ec-21ea720fbd7f-original-pull-secret") pod "global-pull-secret-syncer-zr64w" (UID: "f3460441-b43e-430b-b5ec-21ea720fbd7f") : object "kube-system"/"original-pull-secret" not registered
Jul 27 17:27:48.017758 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:48.017513    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f0e6c35-0c8a-427b-9647-00055e7fa320-kube-api-access-h76x8 podName:1f0e6c35-0c8a-427b-9647-00055e7fa320 nodeName:}" failed. No retries permitted until 2026-07-27 17:27:56.01750452 +0000 UTC m=+18.160906550 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-h76x8" (UniqueName: "kubernetes.io/projected/1f0e6c35-0c8a-427b-9647-00055e7fa320-kube-api-access-h76x8") pod "network-check-target-sqfbc" (UID: "1f0e6c35-0c8a-427b-9647-00055e7fa320") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:27:48.467270 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:48.467226    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zr64w"
Jul 27 17:27:48.467450 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:48.467382    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zr64w" podUID="f3460441-b43e-430b-b5ec-21ea720fbd7f"
Jul 27 17:27:48.467513 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:48.467459    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxdxx"
Jul 27 17:27:48.467619 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:48.467601    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxdxx" podUID="2fedc794-294b-44d7-ba67-55dbad330693"
Jul 27 17:27:49.464313 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:49.463930    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sqfbc"
Jul 27 17:27:49.464313 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:49.464143    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sqfbc" podUID="1f0e6c35-0c8a-427b-9647-00055e7fa320"
Jul 27 17:27:50.463725 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:50.463689    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxdxx"
Jul 27 17:27:50.463725 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:50.463713    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zr64w"
Jul 27 17:27:50.463987 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:50.463836    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zr64w" podUID="f3460441-b43e-430b-b5ec-21ea720fbd7f"
Jul 27 17:27:50.464045 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:50.463999    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxdxx" podUID="2fedc794-294b-44d7-ba67-55dbad330693"
Jul 27 17:27:51.463712 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:51.463676    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sqfbc"
Jul 27 17:27:51.464062 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:51.463784    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sqfbc" podUID="1f0e6c35-0c8a-427b-9647-00055e7fa320"
Jul 27 17:27:52.463728 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:52.463638    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxdxx"
Jul 27 17:27:52.464182 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:52.463651    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zr64w"
Jul 27 17:27:52.464182 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:52.463785    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxdxx" podUID="2fedc794-294b-44d7-ba67-55dbad330693"
Jul 27 17:27:52.464182 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:52.463851    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zr64w" podUID="f3460441-b43e-430b-b5ec-21ea720fbd7f"
Jul 27 17:27:53.463461 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:53.463424    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sqfbc"
Jul 27 17:27:53.463663 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:53.463556    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sqfbc" podUID="1f0e6c35-0c8a-427b-9647-00055e7fa320"
Jul 27 17:27:54.463336 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:54.463292    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxdxx"
Jul 27 17:27:54.463855 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:54.463292    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zr64w"
Jul 27 17:27:54.463855 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:54.463435    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxdxx" podUID="2fedc794-294b-44d7-ba67-55dbad330693"
Jul 27 17:27:54.463855 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:54.463516    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zr64w" podUID="f3460441-b43e-430b-b5ec-21ea720fbd7f"
Jul 27 17:27:55.463859 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:55.463825    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sqfbc"
Jul 27 17:27:55.464342 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:55.463951    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sqfbc" podUID="1f0e6c35-0c8a-427b-9647-00055e7fa320"
Jul 27 17:27:55.974750 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:55.974714    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fedc794-294b-44d7-ba67-55dbad330693-metrics-certs\") pod \"network-metrics-daemon-zxdxx\" (UID: \"2fedc794-294b-44d7-ba67-55dbad330693\") " pod="openshift-multus/network-metrics-daemon-zxdxx"
Jul 27 17:27:55.974907 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:55.974865    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:27:55.974946 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:55.974930    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fedc794-294b-44d7-ba67-55dbad330693-metrics-certs podName:2fedc794-294b-44d7-ba67-55dbad330693 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:11.974915047 +0000 UTC m=+34.118317076 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2fedc794-294b-44d7-ba67-55dbad330693-metrics-certs") pod "network-metrics-daemon-zxdxx" (UID: "2fedc794-294b-44d7-ba67-55dbad330693") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:27:56.075033 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:56.074996    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f3460441-b43e-430b-b5ec-21ea720fbd7f-original-pull-secret\") pod \"global-pull-secret-syncer-zr64w\" (UID: \"f3460441-b43e-430b-b5ec-21ea720fbd7f\") " pod="kube-system/global-pull-secret-syncer-zr64w"
Jul 27 17:27:56.075033 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:56.075040    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h76x8\" (UniqueName: \"kubernetes.io/projected/1f0e6c35-0c8a-427b-9647-00055e7fa320-kube-api-access-h76x8\") pod \"network-check-target-sqfbc\" (UID: \"1f0e6c35-0c8a-427b-9647-00055e7fa320\") " pod="openshift-network-diagnostics/network-check-target-sqfbc"
Jul 27 17:27:56.075317 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:56.075175    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 17:27:56.075317 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:56.075180    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 17:27:56.075317 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:56.075289    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 17:27:56.075317 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:56.075273    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3460441-b43e-430b-b5ec-21ea720fbd7f-original-pull-secret podName:f3460441-b43e-430b-b5ec-21ea720fbd7f nodeName:}" failed. No retries permitted until 2026-07-27 17:28:12.075235926 +0000 UTC m=+34.218637955 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f3460441-b43e-430b-b5ec-21ea720fbd7f-original-pull-secret") pod "global-pull-secret-syncer-zr64w" (UID: "f3460441-b43e-430b-b5ec-21ea720fbd7f") : object "kube-system"/"original-pull-secret" not registered
Jul 27 17:27:56.075317 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:56.075303    2573 projected.go:196] Error preparing data for projected volume kube-api-access-h76x8 for pod openshift-network-diagnostics/network-check-target-sqfbc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:27:56.075563 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:56.075371    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f0e6c35-0c8a-427b-9647-00055e7fa320-kube-api-access-h76x8 podName:1f0e6c35-0c8a-427b-9647-00055e7fa320 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:12.075357337 +0000 UTC m=+34.218759365 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-h76x8" (UniqueName: "kubernetes.io/projected/1f0e6c35-0c8a-427b-9647-00055e7fa320-kube-api-access-h76x8") pod "network-check-target-sqfbc" (UID: "1f0e6c35-0c8a-427b-9647-00055e7fa320") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:27:56.463881 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:56.463798    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zr64w"
Jul 27 17:27:56.464371 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:56.463927    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zr64w" podUID="f3460441-b43e-430b-b5ec-21ea720fbd7f"
Jul 27 17:27:56.464371 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:56.464004    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxdxx"
Jul 27 17:27:56.464371 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:56.464098    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxdxx" podUID="2fedc794-294b-44d7-ba67-55dbad330693"
Jul 27 17:27:57.463534 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:57.463489    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sqfbc"
Jul 27 17:27:57.463712 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:57.463629    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sqfbc" podUID="1f0e6c35-0c8a-427b-9647-00055e7fa320"
Jul 27 17:27:58.464956 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:58.464911    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxdxx"
Jul 27 17:27:58.465379 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:58.465032    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxdxx" podUID="2fedc794-294b-44d7-ba67-55dbad330693"
Jul 27 17:27:58.465379 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:58.465084    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zr64w"
Jul 27 17:27:58.465379 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:58.465176    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zr64w" podUID="f3460441-b43e-430b-b5ec-21ea720fbd7f"
Jul 27 17:27:59.464230 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:27:59.464183    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sqfbc"
Jul 27 17:27:59.464440 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:27:59.464322    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sqfbc" podUID="1f0e6c35-0c8a-427b-9647-00055e7fa320"
Jul 27 17:28:00.463702 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:00.463533    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxdxx"
Jul 27 17:28:00.464141 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:00.463605    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zr64w"
Jul 27 17:28:00.464141 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:28:00.463782    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxdxx" podUID="2fedc794-294b-44d7-ba67-55dbad330693"
Jul 27 17:28:00.464141 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:28:00.463874    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zr64w" podUID="f3460441-b43e-430b-b5ec-21ea720fbd7f"
Jul 27 17:28:00.541830 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:00.541768    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-67cbg" event={"ID":"c6c489eb-66d0-484b-83b4-af557ac8a997","Type":"ContainerStarted","Data":"c4fdb8b2b5fcd01c68277855084d8c61866ee5f70b257edffd300544cabc4981"}
Jul 27 17:28:00.543635 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:00.543428    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-95qjs" event={"ID":"e19a218b-06c1-4547-9f14-00d201dc6030","Type":"ContainerStarted","Data":"1d12a72237096e6ce2005eb1655281fd9a7e787b3f583bab1f28041974ceb77c"}
Jul 27 17:28:00.545027 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:00.544999    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rmfml" event={"ID":"5682e309-a980-4018-8f8d-c942f245b0f1","Type":"ContainerStarted","Data":"355676c500d9940fb24ce6ca545f23dc1228ee3c04b6123d9cebbda19b25ec68"}
Jul 27 17:28:00.546620 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:00.546599    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-p97fd" event={"ID":"cf35f819-6812-444e-a55c-b9fc24bc396e","Type":"ContainerStarted","Data":"6c6a59d64245102617b68ca7d6423ebd3af21b6b3398562a3608b3a227604fb4"}
Jul 27 17:28:00.549293 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:00.549223    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l2h7k" event={"ID":"ff92db47-ab64-4850-a9a7-51267958262f","Type":"ContainerStarted","Data":"8c8e6d137c62324bb887839f015578057066a3b81f26f1eca87f29e890aab187"}
Jul 27 17:28:00.551913 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:00.551889    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zb877" event={"ID":"2802ee72-4be9-423a-a9bd-102c54f96d06","Type":"ContainerStarted","Data":"4cc96b4087b2f01b875b29629ea7c25f54014dae0219d6c4246739ec7a717767"}
Jul 27 17:28:00.553636 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:00.553617    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sgsbl" event={"ID":"1869a1a5-58a8-4839-a479-679491d1f97c","Type":"ContainerStarted","Data":"f02eed77aa8397f9b0d681a2f1988ebd86a105ed6d1e2fa2032f920caa46018c"}
Jul 27 17:28:00.557730 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:00.557688    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-67cbg" podStartSLOduration=3.73113689 podStartE2EDuration="22.557676216s" totalImagesPullingTime="18.826539326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:27:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:00.557332754 +0000 UTC m=+22.700734806" watchObservedRunningTime="2026-07-27 17:28:00.557676216 +0000 UTC m=+22.701078289"
Jul 27 17:28:00.575306 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:00.575232    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-95qjs" podStartSLOduration=3.723586451 podStartE2EDuration="22.575215822s" totalImagesPullingTime="18.851629371s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:27:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:00.574879909 +0000 UTC m=+22.718281959" watchObservedRunningTime="2026-07-27 17:28:00.575215822 +0000 UTC m=+22.718617873"
Jul 27 17:28:00.616968 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:00.616920    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zb877" podStartSLOduration=4.123533278 podStartE2EDuration="22.616905125s" totalImagesPullingTime="18.493371847s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:27:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:00.589684985 +0000 UTC m=+22.733087036" watchObservedRunningTime="2026-07-27 17:28:00.616905125 +0000 UTC m=+22.760307176"
Jul 27 17:28:00.637509 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:00.637466    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rmfml" podStartSLOduration=3.79912466 podStartE2EDuration="22.637450709s" totalImagesPullingTime="18.838326049s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:27:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:00.636923719 +0000 UTC m=+22.780325771" watchObservedRunningTime="2026-07-27 17:28:00.637450709 +0000 UTC m=+22.780852768"
Jul 27 17:28:01.387311 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:01.387114    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-67cbg"
Jul 27 17:28:01.463635 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:01.463604    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sqfbc"
Jul 27 17:28:01.463791 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:28:01.463710    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sqfbc" podUID="1f0e6c35-0c8a-427b-9647-00055e7fa320"
Jul 27 17:28:01.558157 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:01.558119    2573 generic.go:356] "Generic (PLEG): container finished" podID="1869a1a5-58a8-4839-a479-679491d1f97c" containerID="f02eed77aa8397f9b0d681a2f1988ebd86a105ed6d1e2fa2032f920caa46018c" exitCode=0
Jul 27 17:28:01.558374 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:01.558162    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sgsbl" event={"ID":"1869a1a5-58a8-4839-a479-679491d1f97c","Type":"ContainerDied","Data":"f02eed77aa8397f9b0d681a2f1988ebd86a105ed6d1e2fa2032f920caa46018c"}
Jul 27 17:28:01.562405 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:01.562378    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vc2tz_87dac797-f8d2-4d36-999f-f8ce855dbeff/ovn-acl-logging/0.log"
Jul 27 17:28:01.563293 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:01.562870    2573 generic.go:356] "Generic (PLEG): container finished" podID="87dac797-f8d2-4d36-999f-f8ce855dbeff" containerID="5e78d8edaeed2c6ad7900a50b6f1d6c40d153daecd070d1b77ea0f7b05286fb3" exitCode=1
Jul 27 17:28:01.563293 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:01.562952    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz" event={"ID":"87dac797-f8d2-4d36-999f-f8ce855dbeff","Type":"ContainerStarted","Data":"102065db0d4b113a20fc94fc86332f0e1e8a0f3dd28c56572fcf94cab33c4f79"}
Jul 27 17:28:01.563293 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:01.563026    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz" event={"ID":"87dac797-f8d2-4d36-999f-f8ce855dbeff","Type":"ContainerStarted","Data":"86e75523782bd6ed723b37bf2e526f30c9d31024f0b5659b157c38ae056af4a2"}
Jul 27 17:28:01.563293 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:01.563071    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz" event={"ID":"87dac797-f8d2-4d36-999f-f8ce855dbeff","Type":"ContainerStarted","Data":"cf912ef4a8b1928fea62ac3b1310092adeec2a11c80baa902fc90cd1cd901923"}
Jul 27 17:28:01.563293 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:01.563115    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz" event={"ID":"87dac797-f8d2-4d36-999f-f8ce855dbeff","Type":"ContainerStarted","Data":"65554057aa0740c4f4150944e8eae605739ac6f705e9a50b6b60d585f554dbb0"}
Jul 27 17:28:01.563293 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:01.563161    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz" event={"ID":"87dac797-f8d2-4d36-999f-f8ce855dbeff","Type":"ContainerDied","Data":"5e78d8edaeed2c6ad7900a50b6f1d6c40d153daecd070d1b77ea0f7b05286fb3"}
Jul 27 17:28:01.563293 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:01.563207    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz" event={"ID":"87dac797-f8d2-4d36-999f-f8ce855dbeff","Type":"ContainerStarted","Data":"c78d43f3af3c01579cccd8304e7d4504bc8894d41b34f33201f197379b2c6a8a"}
Jul 27 17:28:01.565627 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:01.565602    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qgh42" event={"ID":"d9b682b7-4f6d-4f8e-ae7a-d6a6034859d2","Type":"ContainerStarted","Data":"7286502b60fecd3c23c9ac29a4eba4442842dc9030312dc9b598ff6079846276"}
Jul 27 17:28:01.582263 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:01.582208    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-p97fd" podStartSLOduration=4.729112503 podStartE2EDuration="23.582190445s" totalImagesPullingTime="18.853077942s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:27:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:00.665827323 +0000 UTC m=+22.809229375" watchObservedRunningTime="2026-07-27 17:28:01.582190445 +0000 UTC m=+23.725592496"
Jul 27 17:28:01.582693 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:01.582662    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qgh42" podStartSLOduration=4.746420824 podStartE2EDuration="23.582652752s" totalImagesPullingTime="18.836231928s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:27:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:01.582073885 +0000 UTC m=+23.725475935" watchObservedRunningTime="2026-07-27 17:28:01.582652752 +0000 UTC m=+23.726054802"
Jul 27 17:28:01.644761 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:01.644730    2573 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 27 17:28:02.304803 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:02.304673    2573 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-27T17:28:01.644754556Z","UUID":"8b1edf45-97f1-47ff-bf74-34109b998b47","Handler":null,"Name":"","Endpoint":""}
Jul 27 17:28:02.306532 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:02.306497    2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 27 17:28:02.306674 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:02.306553    2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 27 17:28:02.463387 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:02.463354    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zr64w"
Jul 27 17:28:02.463577 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:28:02.463484    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zr64w" podUID="f3460441-b43e-430b-b5ec-21ea720fbd7f"
Jul 27 17:28:02.463577 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:02.463535    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxdxx"
Jul 27 17:28:02.463752 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:28:02.463656    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxdxx" podUID="2fedc794-294b-44d7-ba67-55dbad330693"
Jul 27 17:28:02.571372 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:02.571287    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l2h7k" event={"ID":"ff92db47-ab64-4850-a9a7-51267958262f","Type":"ContainerStarted","Data":"bf5c2aadea03541ce65fab1d4bb570cb8758f36eaedd3d744add2f7db2537d7c"}
Jul 27 17:28:03.464107 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:03.464076    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sqfbc"
Jul 27 17:28:03.464236 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:28:03.464188    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sqfbc" podUID="1f0e6c35-0c8a-427b-9647-00055e7fa320"
Jul 27 17:28:03.576217 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:03.576179    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l2h7k" event={"ID":"ff92db47-ab64-4850-a9a7-51267958262f","Type":"ContainerStarted","Data":"1c3d870c1fa5490f6f1c4d5e4f17a1e64ff32938e8eede1553893f65f6c529fc"}
Jul 27 17:28:03.580154 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:03.580130    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vc2tz_87dac797-f8d2-4d36-999f-f8ce855dbeff/ovn-acl-logging/0.log"
Jul 27 17:28:03.580618 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:03.580586    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz" event={"ID":"87dac797-f8d2-4d36-999f-f8ce855dbeff","Type":"ContainerStarted","Data":"1bc502888222e9cac95bb6feda821bbf6fbe54bcbce9d486d288066bdb40414d"}
Jul 27 17:28:03.597110 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:03.597048    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l2h7k" podStartSLOduration=4.486977622 podStartE2EDuration="25.597031032s" totalImagesPullingTime="21.11005341s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:27:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:03.596336443 +0000 UTC m=+25.739738495" watchObservedRunningTime="2026-07-27 17:28:03.597031032 +0000 UTC m=+25.740433083"
Jul 27 17:28:04.050948 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:04.050702    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-67cbg"
Jul 27 17:28:04.051467 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:04.051450    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-67cbg"
Jul 27 17:28:04.463780 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:04.463748    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zr64w"
Jul 27 17:28:04.463971 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:04.463753    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxdxx"
Jul 27 17:28:04.463971 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:28:04.463871    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zr64w" podUID="f3460441-b43e-430b-b5ec-21ea720fbd7f"
Jul 27 17:28:04.464107 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:28:04.463975    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxdxx" podUID="2fedc794-294b-44d7-ba67-55dbad330693"
Jul 27 17:28:04.583726 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:04.583696    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-67cbg"
Jul 27 17:28:05.463629 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:05.463597    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sqfbc"
Jul 27 17:28:05.463768 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:28:05.463734    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sqfbc" podUID="1f0e6c35-0c8a-427b-9647-00055e7fa320"
Jul 27 17:28:05.586961 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:05.586933    2573 generic.go:356] "Generic (PLEG): container finished" podID="1869a1a5-58a8-4839-a479-679491d1f97c" containerID="491e1dd527c01c060aa5a1017049024328dd5c6614d8a1ee6f37f20685f3a5fa" exitCode=0
Jul 27 17:28:05.587642 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:05.587004    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sgsbl" event={"ID":"1869a1a5-58a8-4839-a479-679491d1f97c","Type":"ContainerDied","Data":"491e1dd527c01c060aa5a1017049024328dd5c6614d8a1ee6f37f20685f3a5fa"}
Jul 27 17:28:06.464142 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:06.463964    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zr64w"
Jul 27 17:28:06.464359 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:06.464045    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxdxx"
Jul 27 17:28:06.464359 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:28:06.464210    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zr64w" podUID="f3460441-b43e-430b-b5ec-21ea720fbd7f"
Jul 27 17:28:06.464359 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:28:06.464313    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxdxx" podUID="2fedc794-294b-44d7-ba67-55dbad330693"
Jul 27 17:28:06.592853 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:06.592827    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vc2tz_87dac797-f8d2-4d36-999f-f8ce855dbeff/ovn-acl-logging/0.log"
Jul 27 17:28:06.593424 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:06.593177    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz" event={"ID":"87dac797-f8d2-4d36-999f-f8ce855dbeff","Type":"ContainerStarted","Data":"e428d915a6faec6da6cb8f34e70fd2c4d9093cc5b9d608aa2a600d0813b706b6"}
Jul 27 17:28:06.593424 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:06.593401    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:28:06.593523 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:06.593455    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:28:06.593523 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:06.593502    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:28:06.593713 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:06.593696    2573 scope.go:117] "RemoveContainer" containerID="5e78d8edaeed2c6ad7900a50b6f1d6c40d153daecd070d1b77ea0f7b05286fb3"
Jul 27 17:28:06.609157 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:06.609136    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:28:06.610585 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:06.610568    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:28:07.464224 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:07.464189    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sqfbc"
Jul 27 17:28:07.464406 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:28:07.464356    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sqfbc" podUID="1f0e6c35-0c8a-427b-9647-00055e7fa320"
Jul 27 17:28:07.600071 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:07.600044    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vc2tz_87dac797-f8d2-4d36-999f-f8ce855dbeff/ovn-acl-logging/0.log"
Jul 27 17:28:07.600541 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:07.600511    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz" event={"ID":"87dac797-f8d2-4d36-999f-f8ce855dbeff","Type":"ContainerStarted","Data":"e6711ae7fc2952fd694182b810baa092d138e3c9ce8fb9e7c7d061763d6c183b"}
Jul 27 17:28:07.632297 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:07.632230    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz" podStartSLOduration=10.436358703 podStartE2EDuration="29.632209968s" totalImagesPullingTime="19.195851265s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:27:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:07.631942388 +0000 UTC m=+29.775344443" watchObservedRunningTime="2026-07-27 17:28:07.632209968 +0000 UTC m=+29.775612023"
Jul 27 17:28:07.909884 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:07.909661    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zxdxx"]
Jul 27 17:28:07.909999 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:07.909985    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxdxx"
Jul 27 17:28:07.910103 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:28:07.910085    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxdxx" podUID="2fedc794-294b-44d7-ba67-55dbad330693"
Jul 27 17:28:07.912377 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:07.912350    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zr64w"]
Jul 27 17:28:07.912518 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:07.912448    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zr64w"
Jul 27 17:28:07.912577 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:28:07.912548    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zr64w" podUID="f3460441-b43e-430b-b5ec-21ea720fbd7f"
Jul 27 17:28:07.912990 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:07.912971    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sqfbc"]
Jul 27 17:28:07.913076 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:07.913052    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sqfbc"
Jul 27 17:28:07.913162 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:28:07.913141    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sqfbc" podUID="1f0e6c35-0c8a-427b-9647-00055e7fa320"
Jul 27 17:28:08.604724 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:08.604689    2573 generic.go:356] "Generic (PLEG): container finished" podID="1869a1a5-58a8-4839-a479-679491d1f97c" containerID="f507052f2e143631a5492437344b4699339a6529b37d7e3ed0026d43bd6fac2c" exitCode=0
Jul 27 17:28:08.605174 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:08.604768    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sgsbl" event={"ID":"1869a1a5-58a8-4839-a479-679491d1f97c","Type":"ContainerDied","Data":"f507052f2e143631a5492437344b4699339a6529b37d7e3ed0026d43bd6fac2c"}
Jul 27 17:28:09.463374 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:09.463339    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zr64w"
Jul 27 17:28:09.463374 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:09.463355    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sqfbc"
Jul 27 17:28:09.463646 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:28:09.463444    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zr64w" podUID="f3460441-b43e-430b-b5ec-21ea720fbd7f"
Jul 27 17:28:09.463646 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:28:09.463523    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sqfbc" podUID="1f0e6c35-0c8a-427b-9647-00055e7fa320"
Jul 27 17:28:09.463646 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:09.463584    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxdxx"
Jul 27 17:28:09.463774 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:28:09.463655    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxdxx" podUID="2fedc794-294b-44d7-ba67-55dbad330693"
Jul 27 17:28:10.611723 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:10.611635    2573 generic.go:356] "Generic (PLEG): container finished" podID="1869a1a5-58a8-4839-a479-679491d1f97c" containerID="ca7fb3b400f546428327fb53e319746930729ff45dc91ee50a35380166fd7f83" exitCode=0
Jul 27 17:28:10.611723 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:10.611702    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sgsbl" event={"ID":"1869a1a5-58a8-4839-a479-679491d1f97c","Type":"ContainerDied","Data":"ca7fb3b400f546428327fb53e319746930729ff45dc91ee50a35380166fd7f83"}
Jul 27 17:28:11.464231 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:11.464196    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxdxx"
Jul 27 17:28:11.464415 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:11.464196    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sqfbc"
Jul 27 17:28:11.464415 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:28:11.464336    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxdxx" podUID="2fedc794-294b-44d7-ba67-55dbad330693"
Jul 27 17:28:11.464415 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:11.464196    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zr64w"
Jul 27 17:28:11.464598 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:28:11.464424    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sqfbc" podUID="1f0e6c35-0c8a-427b-9647-00055e7fa320"
Jul 27 17:28:11.464598 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:28:11.464491    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zr64w" podUID="f3460441-b43e-430b-b5ec-21ea720fbd7f"
Jul 27 17:28:11.989456 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:11.989413    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fedc794-294b-44d7-ba67-55dbad330693-metrics-certs\") pod \"network-metrics-daemon-zxdxx\" (UID: \"2fedc794-294b-44d7-ba67-55dbad330693\") " pod="openshift-multus/network-metrics-daemon-zxdxx"
Jul 27 17:28:11.989840 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:28:11.989560    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:28:11.989840 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:28:11.989625    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fedc794-294b-44d7-ba67-55dbad330693-metrics-certs podName:2fedc794-294b-44d7-ba67-55dbad330693 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:43.989606337 +0000 UTC m=+66.133008368 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2fedc794-294b-44d7-ba67-55dbad330693-metrics-certs") pod "network-metrics-daemon-zxdxx" (UID: "2fedc794-294b-44d7-ba67-55dbad330693") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:28:12.089922 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:12.089886    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f3460441-b43e-430b-b5ec-21ea720fbd7f-original-pull-secret\") pod \"global-pull-secret-syncer-zr64w\" (UID: \"f3460441-b43e-430b-b5ec-21ea720fbd7f\") " pod="kube-system/global-pull-secret-syncer-zr64w"
Jul 27 17:28:12.089922 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:12.089930    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h76x8\" (UniqueName: \"kubernetes.io/projected/1f0e6c35-0c8a-427b-9647-00055e7fa320-kube-api-access-h76x8\") pod \"network-check-target-sqfbc\" (UID: \"1f0e6c35-0c8a-427b-9647-00055e7fa320\") " pod="openshift-network-diagnostics/network-check-target-sqfbc"
Jul 27 17:28:12.090151 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:28:12.090047    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 17:28:12.090151 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:28:12.090124    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3460441-b43e-430b-b5ec-21ea720fbd7f-original-pull-secret podName:f3460441-b43e-430b-b5ec-21ea720fbd7f nodeName:}" failed. No retries permitted until 2026-07-27 17:28:44.090104951 +0000 UTC m=+66.233506994 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f3460441-b43e-430b-b5ec-21ea720fbd7f-original-pull-secret") pod "global-pull-secret-syncer-zr64w" (UID: "f3460441-b43e-430b-b5ec-21ea720fbd7f") : object "kube-system"/"original-pull-secret" not registered
Jul 27 17:28:12.090151 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:28:12.090052    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 17:28:12.090318 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:28:12.090173    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 17:28:12.090318 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:28:12.090188    2573 projected.go:196] Error preparing data for projected volume kube-api-access-h76x8 for pod openshift-network-diagnostics/network-check-target-sqfbc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:28:12.090318 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:28:12.090243    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1f0e6c35-0c8a-427b-9647-00055e7fa320-kube-api-access-h76x8 podName:1f0e6c35-0c8a-427b-9647-00055e7fa320 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:44.090227869 +0000 UTC m=+66.233629916 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-h76x8" (UniqueName: "kubernetes.io/projected/1f0e6c35-0c8a-427b-9647-00055e7fa320-kube-api-access-h76x8") pod "network-check-target-sqfbc" (UID: "1f0e6c35-0c8a-427b-9647-00055e7fa320") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:28:13.463998 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:13.463967    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zr64w"
Jul 27 17:28:13.464543 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:13.464085    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sqfbc"
Jul 27 17:28:13.464543 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:28:13.464096    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zr64w" podUID="f3460441-b43e-430b-b5ec-21ea720fbd7f"
Jul 27 17:28:13.464543 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:13.464115    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxdxx"
Jul 27 17:28:13.464543 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:28:13.464207    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sqfbc" podUID="1f0e6c35-0c8a-427b-9647-00055e7fa320"
Jul 27 17:28:13.464543 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:28:13.464309    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zxdxx" podUID="2fedc794-294b-44d7-ba67-55dbad330693"
Jul 27 17:28:14.081397 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:14.081329    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-0.ec2.internal" event="NodeReady"
Jul 27 17:28:14.081552 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:14.081462    2573 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 27 17:28:14.147837 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:14.144356    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-php8z"]
Jul 27 17:28:14.150470 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:14.150444    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-php8z"
Jul 27 17:28:14.154156 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:14.153842    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 27 17:28:14.154156 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:14.154021    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 27 17:28:14.154400 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:14.154322    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-swd8n\""
Jul 27 17:28:14.156160 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:14.156139    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6nssx"]
Jul 27 17:28:14.159241 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:14.159220    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6nssx"
Jul 27 17:28:14.163694 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:14.163671    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 27 17:28:14.164319 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:14.163957    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-z94n7\""
Jul 27 17:28:14.164770 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:14.164738    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-php8z"]
Jul 27 17:28:14.165676 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:14.165503    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 27 17:28:14.165676 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:14.165579    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 27 17:28:14.178950 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:14.178921    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6nssx"]
Jul 27 17:28:14.303339 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:14.303300    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg84v\" (UniqueName: \"kubernetes.io/projected/b4f69c89-1bf8-483d-8854-324c9a551e60-kube-api-access-dg84v\") pod \"dns-default-php8z\" (UID: \"b4f69c89-1bf8-483d-8854-324c9a551e60\") " pod="openshift-dns/dns-default-php8z"
Jul 27 17:28:14.303507 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:14.303392    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds66c\" (UniqueName: \"kubernetes.io/projected/567f3aaf-ffed-47b5-97cf-2451d54b0db7-kube-api-access-ds66c\") pod \"ingress-canary-6nssx\" (UID: \"567f3aaf-ffed-47b5-97cf-2451d54b0db7\") " pod="openshift-ingress-canary/ingress-canary-6nssx"
Jul 27 17:28:14.303507 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:14.303424    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4f69c89-1bf8-483d-8854-324c9a551e60-metrics-tls\") pod \"dns-default-php8z\" (UID: \"b4f69c89-1bf8-483d-8854-324c9a551e60\") " pod="openshift-dns/dns-default-php8z"
Jul 27 17:28:14.303507 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:14.303474    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4f69c89-1bf8-483d-8854-324c9a551e60-tmp-dir\") pod \"dns-default-php8z\" (UID: \"b4f69c89-1bf8-483d-8854-324c9a551e60\") " pod="openshift-dns/dns-default-php8z"
Jul 27 17:28:14.303655 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:14.303525    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b4f69c89-1bf8-483d-8854-324c9a551e60-config-volume\") pod \"dns-default-php8z\" (UID: \"b4f69c89-1bf8-483d-8854-324c9a551e60\") " pod="openshift-dns/dns-default-php8z"
Jul 27 17:28:14.303655 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:14.303621    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/567f3aaf-ffed-47b5-97cf-2451d54b0db7-cert\") pod \"ingress-canary-6nssx\" (UID: \"567f3aaf-ffed-47b5-97cf-2451d54b0db7\") " pod="openshift-ingress-canary/ingress-canary-6nssx"
Jul 27 17:28:14.404833 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:14.404805    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds66c\" (UniqueName: \"kubernetes.io/projected/567f3aaf-ffed-47b5-97cf-2451d54b0db7-kube-api-access-ds66c\") pod \"ingress-canary-6nssx\" (UID: \"567f3aaf-ffed-47b5-97cf-2451d54b0db7\") " pod="openshift-ingress-canary/ingress-canary-6nssx"
Jul 27 17:28:14.405018 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:14.404838    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4f69c89-1bf8-483d-8854-324c9a551e60-metrics-tls\") pod \"dns-default-php8z\" (UID: \"b4f69c89-1bf8-483d-8854-324c9a551e60\") " pod="openshift-dns/dns-default-php8z"
Jul 27 17:28:14.405018 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:14.404888    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4f69c89-1bf8-483d-8854-324c9a551e60-tmp-dir\") pod \"dns-default-php8z\" (UID: \"b4f69c89-1bf8-483d-8854-324c9a551e60\") " pod="openshift-dns/dns-default-php8z"
Jul 27 17:28:14.405018 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:14.404932    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b4f69c89-1bf8-483d-8854-324c9a551e60-config-volume\") pod \"dns-default-php8z\" (UID: \"b4f69c89-1bf8-483d-8854-324c9a551e60\") " pod="openshift-dns/dns-default-php8z"
Jul 27 17:28:14.405018 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:14.404967    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/567f3aaf-ffed-47b5-97cf-2451d54b0db7-cert\") pod \"ingress-canary-6nssx\" (UID: \"567f3aaf-ffed-47b5-97cf-2451d54b0db7\") " pod="openshift-ingress-canary/ingress-canary-6nssx"
Jul 27 17:28:14.405018 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:14.405005    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg84v\" (UniqueName: \"kubernetes.io/projected/b4f69c89-1bf8-483d-8854-324c9a551e60-kube-api-access-dg84v\") pod \"dns-default-php8z\" (UID: \"b4f69c89-1bf8-483d-8854-324c9a551e60\") " pod="openshift-dns/dns-default-php8z"
Jul 27 17:28:14.405297 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:28:14.405041    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 17:28:14.405297 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:28:14.405104    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4f69c89-1bf8-483d-8854-324c9a551e60-metrics-tls podName:b4f69c89-1bf8-483d-8854-324c9a551e60 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:14.905085025 +0000 UTC m=+37.048487053 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b4f69c89-1bf8-483d-8854-324c9a551e60-metrics-tls") pod "dns-default-php8z" (UID: "b4f69c89-1bf8-483d-8854-324c9a551e60") : secret "dns-default-metrics-tls" not found
Jul 27 17:28:14.405420 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:14.405304    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4f69c89-1bf8-483d-8854-324c9a551e60-tmp-dir\") pod \"dns-default-php8z\" (UID: \"b4f69c89-1bf8-483d-8854-324c9a551e60\") " pod="openshift-dns/dns-default-php8z"
Jul 27 17:28:14.405758 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:14.405733    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b4f69c89-1bf8-483d-8854-324c9a551e60-config-volume\") pod \"dns-default-php8z\" (UID: \"b4f69c89-1bf8-483d-8854-324c9a551e60\") " pod="openshift-dns/dns-default-php8z"
Jul 27 17:28:14.410069 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:14.410049    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/567f3aaf-ffed-47b5-97cf-2451d54b0db7-cert\") pod \"ingress-canary-6nssx\" (UID: \"567f3aaf-ffed-47b5-97cf-2451d54b0db7\") " pod="openshift-ingress-canary/ingress-canary-6nssx"
Jul 27 17:28:14.418120 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:14.418096    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds66c\" (UniqueName: \"kubernetes.io/projected/567f3aaf-ffed-47b5-97cf-2451d54b0db7-kube-api-access-ds66c\") pod \"ingress-canary-6nssx\" (UID: \"567f3aaf-ffed-47b5-97cf-2451d54b0db7\") " pod="openshift-ingress-canary/ingress-canary-6nssx"
Jul 27 17:28:14.418225 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:14.418181    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg84v\" (UniqueName: \"kubernetes.io/projected/b4f69c89-1bf8-483d-8854-324c9a551e60-kube-api-access-dg84v\") pod \"dns-default-php8z\" (UID: \"b4f69c89-1bf8-483d-8854-324c9a551e60\") " pod="openshift-dns/dns-default-php8z"
Jul 27 17:28:14.471429 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:14.471392    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6nssx"
Jul 27 17:28:14.912375 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:14.911373    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4f69c89-1bf8-483d-8854-324c9a551e60-metrics-tls\") pod \"dns-default-php8z\" (UID: \"b4f69c89-1bf8-483d-8854-324c9a551e60\") " pod="openshift-dns/dns-default-php8z"
Jul 27 17:28:14.912375 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:28:14.911524    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 17:28:14.912375 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:28:14.911587    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4f69c89-1bf8-483d-8854-324c9a551e60-metrics-tls podName:b4f69c89-1bf8-483d-8854-324c9a551e60 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:15.911567 +0000 UTC m=+38.054969029 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b4f69c89-1bf8-483d-8854-324c9a551e60-metrics-tls") pod "dns-default-php8z" (UID: "b4f69c89-1bf8-483d-8854-324c9a551e60") : secret "dns-default-metrics-tls" not found
Jul 27 17:28:14.992086 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:14.992058    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6nssx"]
Jul 27 17:28:14.995967 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:28:14.995942    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod567f3aaf_ffed_47b5_97cf_2451d54b0db7.slice/crio-45d9f4b1d32b46896eacd0706d8a685710a009c7b97c29787cdb14a17efbe7a9 WatchSource:0}: Error finding container 45d9f4b1d32b46896eacd0706d8a685710a009c7b97c29787cdb14a17efbe7a9: Status 404 returned error can't find the container with id 45d9f4b1d32b46896eacd0706d8a685710a009c7b97c29787cdb14a17efbe7a9
Jul 27 17:28:15.463690 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:15.463653    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxdxx"
Jul 27 17:28:15.463878 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:15.463659    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sqfbc"
Jul 27 17:28:15.463878 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:15.463784    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zr64w"
Jul 27 17:28:15.475141 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:15.475112    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 27 17:28:15.476601 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:15.476564    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 27 17:28:15.476752 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:15.476626    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-p44n4\""
Jul 27 17:28:15.476752 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:15.476638    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 27 17:28:15.476752 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:15.476649    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 27 17:28:15.476752 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:15.476630    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jhb7r\""
Jul 27 17:28:15.627210 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:15.627180    2573 generic.go:356] "Generic (PLEG): container finished" podID="1869a1a5-58a8-4839-a479-679491d1f97c" containerID="e2cd321fa46bdf8e68d7a675ed1023cc7ae3f7d44241969f42358d06489a4dad" exitCode=0
Jul 27 17:28:15.627404 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:15.627223    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sgsbl" event={"ID":"1869a1a5-58a8-4839-a479-679491d1f97c","Type":"ContainerDied","Data":"e2cd321fa46bdf8e68d7a675ed1023cc7ae3f7d44241969f42358d06489a4dad"}
Jul 27 17:28:15.628459 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:15.628437    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6nssx" event={"ID":"567f3aaf-ffed-47b5-97cf-2451d54b0db7","Type":"ContainerStarted","Data":"45d9f4b1d32b46896eacd0706d8a685710a009c7b97c29787cdb14a17efbe7a9"}
Jul 27 17:28:15.919586 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:15.919371    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4f69c89-1bf8-483d-8854-324c9a551e60-metrics-tls\") pod \"dns-default-php8z\" (UID: \"b4f69c89-1bf8-483d-8854-324c9a551e60\") " pod="openshift-dns/dns-default-php8z"
Jul 27 17:28:15.919775 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:28:15.919549    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 17:28:15.919775 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:28:15.919696    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4f69c89-1bf8-483d-8854-324c9a551e60-metrics-tls podName:b4f69c89-1bf8-483d-8854-324c9a551e60 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:17.919677971 +0000 UTC m=+40.063079999 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b4f69c89-1bf8-483d-8854-324c9a551e60-metrics-tls") pod "dns-default-php8z" (UID: "b4f69c89-1bf8-483d-8854-324c9a551e60") : secret "dns-default-metrics-tls" not found
Jul 27 17:28:16.634849 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:16.634808    2573 generic.go:356] "Generic (PLEG): container finished" podID="1869a1a5-58a8-4839-a479-679491d1f97c" containerID="8ac73cb832b11fca9db74f5797b68aeed40379cbff3892d8e46cb1b63b97a07d" exitCode=0
Jul 27 17:28:16.635305 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:16.634891    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sgsbl" event={"ID":"1869a1a5-58a8-4839-a479-679491d1f97c","Type":"ContainerDied","Data":"8ac73cb832b11fca9db74f5797b68aeed40379cbff3892d8e46cb1b63b97a07d"}
Jul 27 17:28:16.883930 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:16.883897    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:28:16.979623 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:16.979580    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:28:17.640449 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:17.640415    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sgsbl" event={"ID":"1869a1a5-58a8-4839-a479-679491d1f97c","Type":"ContainerStarted","Data":"6b77c09dd69768182aa9091ab007018e2a527aa7112a39dcaab80a2cc428f913"}
Jul 27 17:28:17.665881 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:17.665835    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-sgsbl" podStartSLOduration=7.880634408 podStartE2EDuration="39.665820348s" totalImagesPullingTime="31.78518594s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:27:38 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:17.665708656 +0000 UTC m=+39.809110707" watchObservedRunningTime="2026-07-27 17:28:17.665820348 +0000 UTC m=+39.809222398"
Jul 27 17:28:17.800188 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:17.800152    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:28:17.875795 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:17.875753    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:28:17.936266 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:17.936223    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4f69c89-1bf8-483d-8854-324c9a551e60-metrics-tls\") pod \"dns-default-php8z\" (UID: \"b4f69c89-1bf8-483d-8854-324c9a551e60\") " pod="openshift-dns/dns-default-php8z"
Jul 27 17:28:17.936466 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:28:17.936448    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 17:28:17.936537 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:28:17.936514    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4f69c89-1bf8-483d-8854-324c9a551e60-metrics-tls podName:b4f69c89-1bf8-483d-8854-324c9a551e60 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:21.936495204 +0000 UTC m=+44.079897233 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b4f69c89-1bf8-483d-8854-324c9a551e60-metrics-tls") pod "dns-default-php8z" (UID: "b4f69c89-1bf8-483d-8854-324c9a551e60") : secret "dns-default-metrics-tls" not found
Jul 27 17:28:17.962732 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:17.962695    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:28:18.423725 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:18.423693    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:28:18.527084 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:18.527047    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:28:18.644442 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:18.644407    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6nssx" event={"ID":"567f3aaf-ffed-47b5-97cf-2451d54b0db7","Type":"ContainerStarted","Data":"5453aafc38eddb8b3c37844e55b6e83405e04ed73a5f9a826cfd47079f7556a6"}
Jul 27 17:28:18.663112 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:18.663064    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6nssx" podStartSLOduration=1.7232718550000001 podStartE2EDuration="4.663049269s" totalImagesPullingTime="2.939777414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:28:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:18.662495785 +0000 UTC m=+40.805897836" watchObservedRunningTime="2026-07-27 17:28:18.663049269 +0000 UTC m=+40.806451321"
Jul 27 17:28:21.963355 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:21.963308    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4f69c89-1bf8-483d-8854-324c9a551e60-metrics-tls\") pod \"dns-default-php8z\" (UID: \"b4f69c89-1bf8-483d-8854-324c9a551e60\") " pod="openshift-dns/dns-default-php8z"
Jul 27 17:28:21.963752 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:28:21.963482    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 17:28:21.963752 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:28:21.963549    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4f69c89-1bf8-483d-8854-324c9a551e60-metrics-tls podName:b4f69c89-1bf8-483d-8854-324c9a551e60 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:29.963532249 +0000 UTC m=+52.106934278 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b4f69c89-1bf8-483d-8854-324c9a551e60-metrics-tls") pod "dns-default-php8z" (UID: "b4f69c89-1bf8-483d-8854-324c9a551e60") : secret "dns-default-metrics-tls" not found
Jul 27 17:28:22.288214 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:22.288117    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:28:22.378135 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:22.378097    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:28:23.936182 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:23.936144    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:28:23.997621 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:23.997588    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:28:25.094499 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:25.094453    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:28:25.153385 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:25.153319    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:28:30.020183 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:30.020137    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4f69c89-1bf8-483d-8854-324c9a551e60-metrics-tls\") pod \"dns-default-php8z\" (UID: \"b4f69c89-1bf8-483d-8854-324c9a551e60\") " pod="openshift-dns/dns-default-php8z"
Jul 27 17:28:30.020600 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:28:30.020316    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 17:28:30.020600 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:28:30.020384    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4f69c89-1bf8-483d-8854-324c9a551e60-metrics-tls podName:b4f69c89-1bf8-483d-8854-324c9a551e60 nodeName:}" failed. No retries permitted until 2026-07-27 17:28:46.020367361 +0000 UTC m=+68.163769389 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b4f69c89-1bf8-483d-8854-324c9a551e60-metrics-tls") pod "dns-default-php8z" (UID: "b4f69c89-1bf8-483d-8854-324c9a551e60") : secret "dns-default-metrics-tls" not found
Jul 27 17:28:37.497765 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:37.497723    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:28:38.623411 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:38.623383    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vc2tz"
Jul 27 17:28:40.413039 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:40.413006    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:28:43.897045 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:43.897004    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:28:43.960830 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:43.960794    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:28:44.019160 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:44.019125    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fedc794-294b-44d7-ba67-55dbad330693-metrics-certs\") pod \"network-metrics-daemon-zxdxx\" (UID: \"2fedc794-294b-44d7-ba67-55dbad330693\") " pod="openshift-multus/network-metrics-daemon-zxdxx"
Jul 27 17:28:44.022239 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:44.022221    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 27 17:28:44.030025 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:28:44.030004    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 27 17:28:44.030138 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:28:44.030077    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fedc794-294b-44d7-ba67-55dbad330693-metrics-certs podName:2fedc794-294b-44d7-ba67-55dbad330693 nodeName:}" failed. No retries permitted until 2026-07-27 17:29:48.030061361 +0000 UTC m=+130.173463389 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2fedc794-294b-44d7-ba67-55dbad330693-metrics-certs") pod "network-metrics-daemon-zxdxx" (UID: "2fedc794-294b-44d7-ba67-55dbad330693") : secret "metrics-daemon-secret" not found
Jul 27 17:28:44.052952 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:44.052916    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:28:44.120142 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:44.120104    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f3460441-b43e-430b-b5ec-21ea720fbd7f-original-pull-secret\") pod \"global-pull-secret-syncer-zr64w\" (UID: \"f3460441-b43e-430b-b5ec-21ea720fbd7f\") " pod="kube-system/global-pull-secret-syncer-zr64w"
Jul 27 17:28:44.120344 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:44.120165    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h76x8\" (UniqueName: \"kubernetes.io/projected/1f0e6c35-0c8a-427b-9647-00055e7fa320-kube-api-access-h76x8\") pod \"network-check-target-sqfbc\" (UID: \"1f0e6c35-0c8a-427b-9647-00055e7fa320\") " pod="openshift-network-diagnostics/network-check-target-sqfbc"
Jul 27 17:28:44.123342 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:44.123320    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 27 17:28:44.123433 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:44.123320    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 27 17:28:44.133833 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:44.133811    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 27 17:28:44.134165 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:44.134147    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f3460441-b43e-430b-b5ec-21ea720fbd7f-original-pull-secret\") pod \"global-pull-secret-syncer-zr64w\" (UID: \"f3460441-b43e-430b-b5ec-21ea720fbd7f\") " pod="kube-system/global-pull-secret-syncer-zr64w"
Jul 27 17:28:44.144092 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:44.144065    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h76x8\" (UniqueName: \"kubernetes.io/projected/1f0e6c35-0c8a-427b-9647-00055e7fa320-kube-api-access-h76x8\") pod \"network-check-target-sqfbc\" (UID: \"1f0e6c35-0c8a-427b-9647-00055e7fa320\") " pod="openshift-network-diagnostics/network-check-target-sqfbc"
Jul 27 17:28:44.282819 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:44.282740    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jhb7r\""
Jul 27 17:28:44.284815 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:44.284797    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zr64w"
Jul 27 17:28:44.290537 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:44.290517    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sqfbc"
Jul 27 17:28:44.432597 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:44.432567    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zr64w"]
Jul 27 17:28:44.435993 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:28:44.435966    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3460441_b43e_430b_b5ec_21ea720fbd7f.slice/crio-50ac907eed91535f8da0bddd1b1efadff6df87dab342b6343fb263061aadf29a WatchSource:0}: Error finding container 50ac907eed91535f8da0bddd1b1efadff6df87dab342b6343fb263061aadf29a: Status 404 returned error can't find the container with id 50ac907eed91535f8da0bddd1b1efadff6df87dab342b6343fb263061aadf29a
Jul 27 17:28:44.444978 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:44.444950    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sqfbc"]
Jul 27 17:28:44.448226 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:28:44.448206    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f0e6c35_0c8a_427b_9647_00055e7fa320.slice/crio-81de3d8b2f7a71709e6c4be64b13028873c8aeab68c5710ad792b6531c2353df WatchSource:0}: Error finding container 81de3d8b2f7a71709e6c4be64b13028873c8aeab68c5710ad792b6531c2353df: Status 404 returned error can't find the container with id 81de3d8b2f7a71709e6c4be64b13028873c8aeab68c5710ad792b6531c2353df
Jul 27 17:28:44.707396 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:44.707360    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sqfbc" event={"ID":"1f0e6c35-0c8a-427b-9647-00055e7fa320","Type":"ContainerStarted","Data":"81de3d8b2f7a71709e6c4be64b13028873c8aeab68c5710ad792b6531c2353df"}
Jul 27 17:28:44.708470 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:44.708435    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zr64w" event={"ID":"f3460441-b43e-430b-b5ec-21ea720fbd7f","Type":"ContainerStarted","Data":"50ac907eed91535f8da0bddd1b1efadff6df87dab342b6343fb263061aadf29a"}
Jul 27 17:28:46.034099 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:46.034051    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4f69c89-1bf8-483d-8854-324c9a551e60-metrics-tls\") pod \"dns-default-php8z\" (UID: \"b4f69c89-1bf8-483d-8854-324c9a551e60\") " pod="openshift-dns/dns-default-php8z"
Jul 27 17:28:46.034581 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:28:46.034196    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 17:28:46.034581 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:28:46.034283    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4f69c89-1bf8-483d-8854-324c9a551e60-metrics-tls podName:b4f69c89-1bf8-483d-8854-324c9a551e60 nodeName:}" failed. No retries permitted until 2026-07-27 17:29:18.034262031 +0000 UTC m=+100.177664075 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b4f69c89-1bf8-483d-8854-324c9a551e60-metrics-tls") pod "dns-default-php8z" (UID: "b4f69c89-1bf8-483d-8854-324c9a551e60") : secret "dns-default-metrics-tls" not found
Jul 27 17:28:49.724596 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:49.724561    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sqfbc" event={"ID":"1f0e6c35-0c8a-427b-9647-00055e7fa320","Type":"ContainerStarted","Data":"d7cfa013af85033e0d7846f9444a0cd332c9a3c27962e07c55526dd07a27b383"}
Jul 27 17:28:49.725051 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:49.724684    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-sqfbc"
Jul 27 17:28:49.725999 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:49.725977    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zr64w" event={"ID":"f3460441-b43e-430b-b5ec-21ea720fbd7f","Type":"ContainerStarted","Data":"9224a9b6f34c96b285c78f10daa2acdb84db63950c6b9301bb434ee8b7cc12e8"}
Jul 27 17:28:49.773160 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:49.773109    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-sqfbc" podStartSLOduration=67.388795366 podStartE2EDuration="1m11.773095015s" totalImagesPullingTime="4.384299649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:27:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:49.772664009 +0000 UTC m=+71.916066059" watchObservedRunningTime="2026-07-27 17:28:49.773095015 +0000 UTC m=+71.916497077"
Jul 27 17:28:49.784946 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:49.784900    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:28:49.807081 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:49.807011    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-zr64w" podStartSLOduration=67.406921501 podStartE2EDuration="1m11.80699223s" totalImagesPullingTime="4.400070729s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:27:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:49.806260854 +0000 UTC m=+71.949662896" watchObservedRunningTime="2026-07-27 17:28:49.80699223 +0000 UTC m=+71.950394282"
Jul 27 17:28:49.877943 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:49.877900    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:28:51.088166 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:28:51.088112    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:11.183122 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:11.183084    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-vx9kx"]
Jul 27 17:29:11.187475 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:11.187454    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vx9kx"
Jul 27 17:29:11.192073 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:11.192052    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 27 17:29:11.192200 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:11.192111    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 27 17:29:11.193678 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:11.193656    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 27 17:29:11.193791 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:11.193772    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 27 17:29:11.193946 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:11.193922    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-zsk9v\""
Jul 27 17:29:11.196827 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:11.196805    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-vx9kx"]
Jul 27 17:29:11.287886 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:11.287854    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-fvcgl"]
Jul 27 17:29:11.290559 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:11.290544    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-fvcgl"
Jul 27 17:29:11.293597 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:11.293568    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-m98bp\""
Jul 27 17:29:11.294365 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:11.294346    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6db8876459-vdgh6"]
Jul 27 17:29:11.297063 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:11.297045    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6db8876459-vdgh6"
Jul 27 17:29:11.299761 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:11.299742    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 27 17:29:11.299951 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:11.299931    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 27 17:29:11.300052 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:11.300036    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-h7n9r\""
Jul 27 17:29:11.300830 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:11.300811    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 27 17:29:11.301596 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:11.301576    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-fvcgl"]
Jul 27 17:29:11.302110 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:11.302090    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ab3e588-fe39-4747-8d1f-9b1ce44ffed4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-vx9kx\" (UID: \"7ab3e588-fe39-4747-8d1f-9b1ce44ffed4\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vx9kx"
Jul 27 17:29:11.302207 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:11.302189    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7ab3e588-fe39-4747-8d1f-9b1ce44ffed4-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-vx9kx\" (UID: \"7ab3e588-fe39-4747-8d1f-9b1ce44ffed4\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vx9kx"
Jul 27 17:29:11.302301 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:11.302224    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frhsm\" (UniqueName: \"kubernetes.io/projected/7ab3e588-fe39-4747-8d1f-9b1ce44ffed4-kube-api-access-frhsm\") pod \"cluster-monitoring-operator-548b489cb8-vx9kx\" (UID: \"7ab3e588-fe39-4747-8d1f-9b1ce44ffed4\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vx9kx"
Jul 27 17:29:11.305904 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:11.305885    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 27 17:29:11.315657 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:11.315635    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6db8876459-vdgh6"]
Jul 27 17:29:11.402820 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:11.402785    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6f2becbf-a41c-4215-9435-9f735c60822e-bound-sa-token\") pod \"image-registry-6db8876459-vdgh6\" (UID: \"6f2becbf-a41c-4215-9435-9f735c60822e\") " pod="openshift-image-registry/image-registry-6db8876459-vdgh6"
Jul 27 17:29:11.402999 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:11.402837    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6f2becbf-a41c-4215-9435-9f735c60822e-registry-tls\") pod \"image-registry-6db8876459-vdgh6\" (UID: \"6f2becbf-a41c-4215-9435-9f735c60822e\") " pod="openshift-image-registry/image-registry-6db8876459-vdgh6"
Jul 27 17:29:11.402999 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:11.402858    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6f2becbf-a41c-4215-9435-9f735c60822e-registry-certificates\") pod \"image-registry-6db8876459-vdgh6\" (UID: \"6f2becbf-a41c-4215-9435-9f735c60822e\") " pod="openshift-image-registry/image-registry-6db8876459-vdgh6"
Jul 27 17:29:11.402999 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:11.402875    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6f2becbf-a41c-4215-9435-9f735c60822e-trusted-ca\") pod \"image-registry-6db8876459-vdgh6\" (UID: \"6f2becbf-a41c-4215-9435-9f735c60822e\") " pod="openshift-image-registry/image-registry-6db8876459-vdgh6"
Jul 27 17:29:11.402999 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:11.402910    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7ab3e588-fe39-4747-8d1f-9b1ce44ffed4-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-vx9kx\" (UID: \"7ab3e588-fe39-4747-8d1f-9b1ce44ffed4\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vx9kx"
Jul 27 17:29:11.402999 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:11.402954    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frhsm\" (UniqueName: \"kubernetes.io/projected/7ab3e588-fe39-4747-8d1f-9b1ce44ffed4-kube-api-access-frhsm\") pod \"cluster-monitoring-operator-548b489cb8-vx9kx\" (UID: \"7ab3e588-fe39-4747-8d1f-9b1ce44ffed4\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vx9kx"
Jul 27 17:29:11.402999 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:11.402990    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6f2becbf-a41c-4215-9435-9f735c60822e-installation-pull-secrets\") pod \"image-registry-6db8876459-vdgh6\" (UID: \"6f2becbf-a41c-4215-9435-9f735c60822e\") " pod="openshift-image-registry/image-registry-6db8876459-vdgh6"
Jul 27 17:29:11.403188 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:11.403022    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6f2becbf-a41c-4215-9435-9f735c60822e-image-registry-private-configuration\") pod \"image-registry-6db8876459-vdgh6\" (UID: \"6f2becbf-a41c-4215-9435-9f735c60822e\") " pod="openshift-image-registry/image-registry-6db8876459-vdgh6"
Jul 27 17:29:11.403188 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:11.403047    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6f2becbf-a41c-4215-9435-9f735c60822e-ca-trust-extracted\") pod \"image-registry-6db8876459-vdgh6\" (UID: \"6f2becbf-a41c-4215-9435-9f735c60822e\") " pod="openshift-image-registry/image-registry-6db8876459-vdgh6"
Jul 27 17:29:11.403188 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:11.403067    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ab3e588-fe39-4747-8d1f-9b1ce44ffed4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-vx9kx\" (UID: \"7ab3e588-fe39-4747-8d1f-9b1ce44ffed4\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vx9kx"
Jul 27 17:29:11.403188 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:11.403092    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvbjl\" (UniqueName: \"kubernetes.io/projected/953d5e73-7bd2-454e-a2c5-90c411bcbd7d-kube-api-access-nvbjl\") pod \"network-check-source-7d88cb55d8-fvcgl\" (UID: \"953d5e73-7bd2-454e-a2c5-90c411bcbd7d\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-fvcgl"
Jul 27 17:29:11.403188 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:11.403151    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zptp5\" (UniqueName: \"kubernetes.io/projected/6f2becbf-a41c-4215-9435-9f735c60822e-kube-api-access-zptp5\") pod \"image-registry-6db8876459-vdgh6\" (UID: \"6f2becbf-a41c-4215-9435-9f735c60822e\") " pod="openshift-image-registry/image-registry-6db8876459-vdgh6"
Jul 27 17:29:11.403188 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:29:11.403173    2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 27 17:29:11.403426 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:29:11.403268    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ab3e588-fe39-4747-8d1f-9b1ce44ffed4-cluster-monitoring-operator-tls podName:7ab3e588-fe39-4747-8d1f-9b1ce44ffed4 nodeName:}" failed. No retries permitted until 2026-07-27 17:29:11.903228473 +0000 UTC m=+94.046630521 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7ab3e588-fe39-4747-8d1f-9b1ce44ffed4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-vx9kx" (UID: "7ab3e588-fe39-4747-8d1f-9b1ce44ffed4") : secret "cluster-monitoring-operator-tls" not found
Jul 27 17:29:11.403603 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:11.403587    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7ab3e588-fe39-4747-8d1f-9b1ce44ffed4-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-vx9kx\" (UID: \"7ab3e588-fe39-4747-8d1f-9b1ce44ffed4\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vx9kx"
Jul 27 17:29:11.415548 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:11.415522    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frhsm\" (UniqueName: \"kubernetes.io/projected/7ab3e588-fe39-4747-8d1f-9b1ce44ffed4-kube-api-access-frhsm\") pod \"cluster-monitoring-operator-548b489cb8-vx9kx\" (UID: \"7ab3e588-fe39-4747-8d1f-9b1ce44ffed4\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vx9kx"
Jul 27 17:29:11.503839 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:11.503735    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6f2becbf-a41c-4215-9435-9f735c60822e-installation-pull-secrets\") pod \"image-registry-6db8876459-vdgh6\" (UID: \"6f2becbf-a41c-4215-9435-9f735c60822e\") " pod="openshift-image-registry/image-registry-6db8876459-vdgh6"
Jul 27 17:29:11.503839 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:11.503785    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6f2becbf-a41c-4215-9435-9f735c60822e-image-registry-private-configuration\") pod \"image-registry-6db8876459-vdgh6\" (UID: \"6f2becbf-a41c-4215-9435-9f735c60822e\") " pod="openshift-image-registry/image-registry-6db8876459-vdgh6"
Jul 27 17:29:11.503839 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:11.503817    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6f2becbf-a41c-4215-9435-9f735c60822e-ca-trust-extracted\") pod \"image-registry-6db8876459-vdgh6\" (UID: \"6f2becbf-a41c-4215-9435-9f735c60822e\") " pod="openshift-image-registry/image-registry-6db8876459-vdgh6"
Jul 27 17:29:11.504123 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:11.503856    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvbjl\" (UniqueName: \"kubernetes.io/projected/953d5e73-7bd2-454e-a2c5-90c411bcbd7d-kube-api-access-nvbjl\") pod \"network-check-source-7d88cb55d8-fvcgl\" (UID: \"953d5e73-7bd2-454e-a2c5-90c411bcbd7d\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-fvcgl"
Jul 27 17:29:11.504123 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:11.503878    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zptp5\" (UniqueName: \"kubernetes.io/projected/6f2becbf-a41c-4215-9435-9f735c60822e-kube-api-access-zptp5\") pod \"image-registry-6db8876459-vdgh6\" (UID: \"6f2becbf-a41c-4215-9435-9f735c60822e\") " pod="openshift-image-registry/image-registry-6db8876459-vdgh6"
Jul 27 17:29:11.504123 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:11.503933    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6f2becbf-a41c-4215-9435-9f735c60822e-bound-sa-token\") pod \"image-registry-6db8876459-vdgh6\" (UID: \"6f2becbf-a41c-4215-9435-9f735c60822e\") " pod="openshift-image-registry/image-registry-6db8876459-vdgh6"
Jul 27 17:29:11.504123 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:11.503972    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6f2becbf-a41c-4215-9435-9f735c60822e-registry-tls\") pod \"image-registry-6db8876459-vdgh6\" (UID: \"6f2becbf-a41c-4215-9435-9f735c60822e\") " pod="openshift-image-registry/image-registry-6db8876459-vdgh6"
Jul 27 17:29:11.504123 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:11.503995    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6f2becbf-a41c-4215-9435-9f735c60822e-registry-certificates\") pod \"image-registry-6db8876459-vdgh6\" (UID: \"6f2becbf-a41c-4215-9435-9f735c60822e\") " pod="openshift-image-registry/image-registry-6db8876459-vdgh6"
Jul 27 17:29:11.504123 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:11.504018    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6f2becbf-a41c-4215-9435-9f735c60822e-trusted-ca\") pod \"image-registry-6db8876459-vdgh6\" (UID: \"6f2becbf-a41c-4215-9435-9f735c60822e\") " pod="openshift-image-registry/image-registry-6db8876459-vdgh6"
Jul 27 17:29:11.504458 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:29:11.504137    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 17:29:11.504458 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:29:11.504157    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6db8876459-vdgh6: secret "image-registry-tls" not found
Jul 27 17:29:11.504458 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:29:11.504222    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f2becbf-a41c-4215-9435-9f735c60822e-registry-tls podName:6f2becbf-a41c-4215-9435-9f735c60822e nodeName:}" failed. No retries permitted until 2026-07-27 17:29:12.004202571 +0000 UTC m=+94.147604615 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6f2becbf-a41c-4215-9435-9f735c60822e-registry-tls") pod "image-registry-6db8876459-vdgh6" (UID: "6f2becbf-a41c-4215-9435-9f735c60822e") : secret "image-registry-tls" not found
Jul 27 17:29:11.504458 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:11.504349    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6f2becbf-a41c-4215-9435-9f735c60822e-ca-trust-extracted\") pod \"image-registry-6db8876459-vdgh6\" (UID: \"6f2becbf-a41c-4215-9435-9f735c60822e\") " pod="openshift-image-registry/image-registry-6db8876459-vdgh6"
Jul 27 17:29:11.504669 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:11.504646    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6f2becbf-a41c-4215-9435-9f735c60822e-registry-certificates\") pod \"image-registry-6db8876459-vdgh6\" (UID: \"6f2becbf-a41c-4215-9435-9f735c60822e\") " pod="openshift-image-registry/image-registry-6db8876459-vdgh6"
Jul 27 17:29:11.504984 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:11.504962    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6f2becbf-a41c-4215-9435-9f735c60822e-trusted-ca\") pod \"image-registry-6db8876459-vdgh6\" (UID: \"6f2becbf-a41c-4215-9435-9f735c60822e\") " pod="openshift-image-registry/image-registry-6db8876459-vdgh6"
Jul 27 17:29:11.506765 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:11.506739    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6f2becbf-a41c-4215-9435-9f735c60822e-image-registry-private-configuration\") pod \"image-registry-6db8876459-vdgh6\" (UID: \"6f2becbf-a41c-4215-9435-9f735c60822e\") " pod="openshift-image-registry/image-registry-6db8876459-vdgh6"
Jul 27 17:29:11.506861 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:11.506792    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6f2becbf-a41c-4215-9435-9f735c60822e-installation-pull-secrets\") pod \"image-registry-6db8876459-vdgh6\" (UID: \"6f2becbf-a41c-4215-9435-9f735c60822e\") " pod="openshift-image-registry/image-registry-6db8876459-vdgh6"
Jul 27 17:29:11.513924 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:11.513904    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6f2becbf-a41c-4215-9435-9f735c60822e-bound-sa-token\") pod \"image-registry-6db8876459-vdgh6\" (UID: \"6f2becbf-a41c-4215-9435-9f735c60822e\") " pod="openshift-image-registry/image-registry-6db8876459-vdgh6"
Jul 27 17:29:11.514143 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:11.514122    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zptp5\" (UniqueName: \"kubernetes.io/projected/6f2becbf-a41c-4215-9435-9f735c60822e-kube-api-access-zptp5\") pod \"image-registry-6db8876459-vdgh6\" (UID: \"6f2becbf-a41c-4215-9435-9f735c60822e\") " pod="openshift-image-registry/image-registry-6db8876459-vdgh6"
Jul 27 17:29:11.514301 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:11.514279    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvbjl\" (UniqueName: \"kubernetes.io/projected/953d5e73-7bd2-454e-a2c5-90c411bcbd7d-kube-api-access-nvbjl\") pod \"network-check-source-7d88cb55d8-fvcgl\" (UID: \"953d5e73-7bd2-454e-a2c5-90c411bcbd7d\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-fvcgl"
Jul 27 17:29:11.541145 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:11.541108    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:11.599849 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:11.599816    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-fvcgl"
Jul 27 17:29:11.642577 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:11.642540    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:11.725714 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:11.725578    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-fvcgl"]
Jul 27 17:29:11.728132 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:29:11.728110    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod953d5e73_7bd2_454e_a2c5_90c411bcbd7d.slice/crio-435a94dc54af34a0f649f3ca395561047c8849f8c7538f83c460de3a7e677904 WatchSource:0}: Error finding container 435a94dc54af34a0f649f3ca395561047c8849f8c7538f83c460de3a7e677904: Status 404 returned error can't find the container with id 435a94dc54af34a0f649f3ca395561047c8849f8c7538f83c460de3a7e677904
Jul 27 17:29:11.780131 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:11.780106    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-fvcgl" event={"ID":"953d5e73-7bd2-454e-a2c5-90c411bcbd7d","Type":"ContainerStarted","Data":"7f90807836c5f1a8ded85c00045d1e8f3e39de7a8559de4aee09a44b048e64d8"}
Jul 27 17:29:11.780227 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:11.780158    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-fvcgl" event={"ID":"953d5e73-7bd2-454e-a2c5-90c411bcbd7d","Type":"ContainerStarted","Data":"435a94dc54af34a0f649f3ca395561047c8849f8c7538f83c460de3a7e677904"}
Jul 27 17:29:11.907823 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:11.907786    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ab3e588-fe39-4747-8d1f-9b1ce44ffed4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-vx9kx\" (UID: \"7ab3e588-fe39-4747-8d1f-9b1ce44ffed4\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vx9kx"
Jul 27 17:29:11.907981 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:29:11.907936    2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 27 17:29:11.908044 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:29:11.908010    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ab3e588-fe39-4747-8d1f-9b1ce44ffed4-cluster-monitoring-operator-tls podName:7ab3e588-fe39-4747-8d1f-9b1ce44ffed4 nodeName:}" failed. No retries permitted until 2026-07-27 17:29:12.907993808 +0000 UTC m=+95.051395839 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7ab3e588-fe39-4747-8d1f-9b1ce44ffed4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-vx9kx" (UID: "7ab3e588-fe39-4747-8d1f-9b1ce44ffed4") : secret "cluster-monitoring-operator-tls" not found
Jul 27 17:29:12.009283 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:12.009187    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6f2becbf-a41c-4215-9435-9f735c60822e-registry-tls\") pod \"image-registry-6db8876459-vdgh6\" (UID: \"6f2becbf-a41c-4215-9435-9f735c60822e\") " pod="openshift-image-registry/image-registry-6db8876459-vdgh6"
Jul 27 17:29:12.009434 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:29:12.009364    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 17:29:12.009434 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:29:12.009387    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6db8876459-vdgh6: secret "image-registry-tls" not found
Jul 27 17:29:12.009539 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:29:12.009457    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f2becbf-a41c-4215-9435-9f735c60822e-registry-tls podName:6f2becbf-a41c-4215-9435-9f735c60822e nodeName:}" failed. No retries permitted until 2026-07-27 17:29:13.009436503 +0000 UTC m=+95.152838531 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6f2becbf-a41c-4215-9435-9f735c60822e-registry-tls") pod "image-registry-6db8876459-vdgh6" (UID: "6f2becbf-a41c-4215-9435-9f735c60822e") : secret "image-registry-tls" not found
Jul 27 17:29:12.802370 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:12.802322    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-fvcgl" podStartSLOduration=1.802304934 podStartE2EDuration="1.802304934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:29:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:12.801016616 +0000 UTC m=+94.944418668" watchObservedRunningTime="2026-07-27 17:29:12.802304934 +0000 UTC m=+94.945706985"
Jul 27 17:29:12.916587 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:12.916549    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ab3e588-fe39-4747-8d1f-9b1ce44ffed4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-vx9kx\" (UID: \"7ab3e588-fe39-4747-8d1f-9b1ce44ffed4\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vx9kx"
Jul 27 17:29:12.916720 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:29:12.916698    2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 27 17:29:12.916776 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:29:12.916769    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ab3e588-fe39-4747-8d1f-9b1ce44ffed4-cluster-monitoring-operator-tls podName:7ab3e588-fe39-4747-8d1f-9b1ce44ffed4 nodeName:}" failed. No retries permitted until 2026-07-27 17:29:14.916752905 +0000 UTC m=+97.060154933 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7ab3e588-fe39-4747-8d1f-9b1ce44ffed4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-vx9kx" (UID: "7ab3e588-fe39-4747-8d1f-9b1ce44ffed4") : secret "cluster-monitoring-operator-tls" not found
Jul 27 17:29:13.017742 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:13.017715    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6f2becbf-a41c-4215-9435-9f735c60822e-registry-tls\") pod \"image-registry-6db8876459-vdgh6\" (UID: \"6f2becbf-a41c-4215-9435-9f735c60822e\") " pod="openshift-image-registry/image-registry-6db8876459-vdgh6"
Jul 27 17:29:13.017888 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:29:13.017869    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 17:29:13.017928 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:29:13.017890    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6db8876459-vdgh6: secret "image-registry-tls" not found
Jul 27 17:29:13.017962 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:29:13.017942    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f2becbf-a41c-4215-9435-9f735c60822e-registry-tls podName:6f2becbf-a41c-4215-9435-9f735c60822e nodeName:}" failed. No retries permitted until 2026-07-27 17:29:15.01792803 +0000 UTC m=+97.161330063 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6f2becbf-a41c-4215-9435-9f735c60822e-registry-tls") pod "image-registry-6db8876459-vdgh6" (UID: "6f2becbf-a41c-4215-9435-9f735c60822e") : secret "image-registry-tls" not found
Jul 27 17:29:14.931594 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:14.931557    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ab3e588-fe39-4747-8d1f-9b1ce44ffed4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-vx9kx\" (UID: \"7ab3e588-fe39-4747-8d1f-9b1ce44ffed4\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vx9kx"
Jul 27 17:29:14.931974 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:29:14.931673    2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 27 17:29:14.931974 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:29:14.931729    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ab3e588-fe39-4747-8d1f-9b1ce44ffed4-cluster-monitoring-operator-tls podName:7ab3e588-fe39-4747-8d1f-9b1ce44ffed4 nodeName:}" failed. No retries permitted until 2026-07-27 17:29:18.931715659 +0000 UTC m=+101.075117688 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7ab3e588-fe39-4747-8d1f-9b1ce44ffed4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-vx9kx" (UID: "7ab3e588-fe39-4747-8d1f-9b1ce44ffed4") : secret "cluster-monitoring-operator-tls" not found
Jul 27 17:29:15.027898 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:15.027866    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:15.032475 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:15.032453    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6f2becbf-a41c-4215-9435-9f735c60822e-registry-tls\") pod \"image-registry-6db8876459-vdgh6\" (UID: \"6f2becbf-a41c-4215-9435-9f735c60822e\") " pod="openshift-image-registry/image-registry-6db8876459-vdgh6"
Jul 27 17:29:15.032590 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:29:15.032576    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 17:29:15.032634 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:29:15.032592    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6db8876459-vdgh6: secret "image-registry-tls" not found
Jul 27 17:29:15.032669 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:29:15.032637    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f2becbf-a41c-4215-9435-9f735c60822e-registry-tls podName:6f2becbf-a41c-4215-9435-9f735c60822e nodeName:}" failed. No retries permitted until 2026-07-27 17:29:19.032623049 +0000 UTC m=+101.176025078 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6f2becbf-a41c-4215-9435-9f735c60822e-registry-tls") pod "image-registry-6db8876459-vdgh6" (UID: "6f2becbf-a41c-4215-9435-9f735c60822e") : secret "image-registry-tls" not found
Jul 27 17:29:15.127422 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:15.127384    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:15.435789 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:15.435752    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8ctdw"]
Jul 27 17:29:15.438887 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:15.438865    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8ctdw"
Jul 27 17:29:15.443597 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:15.443564    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 27 17:29:15.443701 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:15.443666    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 27 17:29:15.443796 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:15.443779    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 27 17:29:15.452698 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:15.452678    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 27 17:29:15.458029 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:15.458013    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-cgk6t\""
Jul 27 17:29:15.465627 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:15.465604    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8ctdw"]
Jul 27 17:29:15.535908 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:15.535871    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69v6j\" (UniqueName: \"kubernetes.io/projected/fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4-kube-api-access-69v6j\") pod \"insights-runtime-extractor-8ctdw\" (UID: \"fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4\") " pod="openshift-insights/insights-runtime-extractor-8ctdw"
Jul 27 17:29:15.535908 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:15.535908    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8ctdw\" (UID: \"fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4\") " pod="openshift-insights/insights-runtime-extractor-8ctdw"
Jul 27 17:29:15.536219 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:15.535936    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4-crio-socket\") pod \"insights-runtime-extractor-8ctdw\" (UID: \"fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4\") " pod="openshift-insights/insights-runtime-extractor-8ctdw"
Jul 27 17:29:15.536219 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:15.536025    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4-data-volume\") pod \"insights-runtime-extractor-8ctdw\" (UID: \"fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4\") " pod="openshift-insights/insights-runtime-extractor-8ctdw"
Jul 27 17:29:15.536219 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:15.536065    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8ctdw\" (UID: \"fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4\") " pod="openshift-insights/insights-runtime-extractor-8ctdw"
Jul 27 17:29:15.637177 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:15.637142    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69v6j\" (UniqueName: \"kubernetes.io/projected/fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4-kube-api-access-69v6j\") pod \"insights-runtime-extractor-8ctdw\" (UID: \"fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4\") " pod="openshift-insights/insights-runtime-extractor-8ctdw"
Jul 27 17:29:15.637333 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:15.637184    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8ctdw\" (UID: \"fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4\") " pod="openshift-insights/insights-runtime-extractor-8ctdw"
Jul 27 17:29:15.637333 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:15.637210    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4-crio-socket\") pod \"insights-runtime-extractor-8ctdw\" (UID: \"fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4\") " pod="openshift-insights/insights-runtime-extractor-8ctdw"
Jul 27 17:29:15.637333 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:15.637234    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4-data-volume\") pod \"insights-runtime-extractor-8ctdw\" (UID: \"fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4\") " pod="openshift-insights/insights-runtime-extractor-8ctdw"
Jul 27 17:29:15.637333 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:15.637284    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8ctdw\" (UID: \"fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4\") " pod="openshift-insights/insights-runtime-extractor-8ctdw"
Jul 27 17:29:15.637554 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:15.637326    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4-crio-socket\") pod \"insights-runtime-extractor-8ctdw\" (UID: \"fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4\") " pod="openshift-insights/insights-runtime-extractor-8ctdw"
Jul 27 17:29:15.637554 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:29:15.637352    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 17:29:15.637554 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:29:15.637437    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4-insights-runtime-extractor-tls podName:fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4 nodeName:}" failed. No retries permitted until 2026-07-27 17:29:16.137415014 +0000 UTC m=+98.280817045 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8ctdw" (UID: "fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4") : secret "insights-runtime-extractor-tls" not found
Jul 27 17:29:15.637554 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:29:15.637363    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 17:29:15.637554 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:29:15.637492    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4-kube-rbac-proxy-cm podName:fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4 nodeName:}" failed. No retries permitted until 2026-07-27 17:29:16.137481932 +0000 UTC m=+98.280883962 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8ctdw" (UID: "fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4") : configmap "kube-rbac-proxy" not found
Jul 27 17:29:15.637741 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:15.637629    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4-data-volume\") pod \"insights-runtime-extractor-8ctdw\" (UID: \"fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4\") " pod="openshift-insights/insights-runtime-extractor-8ctdw"
Jul 27 17:29:15.647552 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:15.647530    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69v6j\" (UniqueName: \"kubernetes.io/projected/fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4-kube-api-access-69v6j\") pod \"insights-runtime-extractor-8ctdw\" (UID: \"fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4\") " pod="openshift-insights/insights-runtime-extractor-8ctdw"
Jul 27 17:29:16.141007 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:16.140967    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8ctdw\" (UID: \"fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4\") " pod="openshift-insights/insights-runtime-extractor-8ctdw"
Jul 27 17:29:16.141471 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:16.141061    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8ctdw\" (UID: \"fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4\") " pod="openshift-insights/insights-runtime-extractor-8ctdw"
Jul 27 17:29:16.141471 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:29:16.141099    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 17:29:16.141471 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:29:16.141177    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4-kube-rbac-proxy-cm podName:fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4 nodeName:}" failed. No retries permitted until 2026-07-27 17:29:17.141161685 +0000 UTC m=+99.284563714 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8ctdw" (UID: "fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4") : configmap "kube-rbac-proxy" not found
Jul 27 17:29:16.141471 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:29:16.141179    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 17:29:16.141471 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:29:16.141232    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4-insights-runtime-extractor-tls podName:fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4 nodeName:}" failed. No retries permitted until 2026-07-27 17:29:17.141216382 +0000 UTC m=+99.284618417 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8ctdw" (UID: "fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4") : secret "insights-runtime-extractor-tls" not found
Jul 27 17:29:17.148467 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:17.148426    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8ctdw\" (UID: \"fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4\") " pod="openshift-insights/insights-runtime-extractor-8ctdw"
Jul 27 17:29:17.148888 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:17.148490    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8ctdw\" (UID: \"fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4\") " pod="openshift-insights/insights-runtime-extractor-8ctdw"
Jul 27 17:29:17.148888 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:29:17.148575    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 17:29:17.148888 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:29:17.148649    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4-insights-runtime-extractor-tls podName:fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4 nodeName:}" failed. No retries permitted until 2026-07-27 17:29:19.148634632 +0000 UTC m=+101.292036661 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8ctdw" (UID: "fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4") : secret "insights-runtime-extractor-tls" not found
Jul 27 17:29:17.148888 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:29:17.148582    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 17:29:17.148888 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:29:17.148708    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4-kube-rbac-proxy-cm podName:fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4 nodeName:}" failed. No retries permitted until 2026-07-27 17:29:19.148695602 +0000 UTC m=+101.292097631 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8ctdw" (UID: "fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4") : configmap "kube-rbac-proxy" not found
Jul 27 17:29:17.203379 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:17.203346    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:17.307204 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:17.307170    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-dcbmp"]
Jul 27 17:29:17.311504 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:17.311485    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-dcbmp"
Jul 27 17:29:17.314528 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:17.314510    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-2ptpk\""
Jul 27 17:29:17.315791 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:17.315773    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 27 17:29:17.315893 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:17.315822    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 27 17:29:17.319546 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:17.319518    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:17.325716 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:17.325693    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-dcbmp"]
Jul 27 17:29:17.350371 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:17.350333    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vvlk\" (UniqueName: \"kubernetes.io/projected/93ed2402-e28d-4671-b8bd-f7993a1afdd0-kube-api-access-5vvlk\") pod \"migrator-6865c9967-dcbmp\" (UID: \"93ed2402-e28d-4671-b8bd-f7993a1afdd0\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-dcbmp"
Jul 27 17:29:17.400826 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:17.400749    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-95qjs_e19a218b-06c1-4547-9f14-00d201dc6030/dns-node-resolver/0.log"
Jul 27 17:29:17.407848 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:17.407821    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:17.451496 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:17.451465    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vvlk\" (UniqueName: \"kubernetes.io/projected/93ed2402-e28d-4671-b8bd-f7993a1afdd0-kube-api-access-5vvlk\") pod \"migrator-6865c9967-dcbmp\" (UID: \"93ed2402-e28d-4671-b8bd-f7993a1afdd0\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-dcbmp"
Jul 27 17:29:17.461328 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:17.461304    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vvlk\" (UniqueName: \"kubernetes.io/projected/93ed2402-e28d-4671-b8bd-f7993a1afdd0-kube-api-access-5vvlk\") pod \"migrator-6865c9967-dcbmp\" (UID: \"93ed2402-e28d-4671-b8bd-f7993a1afdd0\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-dcbmp"
Jul 27 17:29:17.469858 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:17.469818    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:17.621283 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:17.621230    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-dcbmp"
Jul 27 17:29:17.751724 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:17.751691    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-dcbmp"]
Jul 27 17:29:17.754575 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:29:17.754544    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93ed2402_e28d_4671_b8bd_f7993a1afdd0.slice/crio-5643fae36727330b33157014f8203617dd8ff5ce6f5c977b4d10af982a77a084 WatchSource:0}: Error finding container 5643fae36727330b33157014f8203617dd8ff5ce6f5c977b4d10af982a77a084: Status 404 returned error can't find the container with id 5643fae36727330b33157014f8203617dd8ff5ce6f5c977b4d10af982a77a084
Jul 27 17:29:17.794801 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:17.794764    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-dcbmp" event={"ID":"93ed2402-e28d-4671-b8bd-f7993a1afdd0","Type":"ContainerStarted","Data":"5643fae36727330b33157014f8203617dd8ff5ce6f5c977b4d10af982a77a084"}
Jul 27 17:29:17.984477 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:17.984390    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:18.056181 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:18.056139    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4f69c89-1bf8-483d-8854-324c9a551e60-metrics-tls\") pod \"dns-default-php8z\" (UID: \"b4f69c89-1bf8-483d-8854-324c9a551e60\") " pod="openshift-dns/dns-default-php8z"
Jul 27 17:29:18.056369 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:29:18.056313    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 17:29:18.056417 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:29:18.056379    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4f69c89-1bf8-483d-8854-324c9a551e60-metrics-tls podName:b4f69c89-1bf8-483d-8854-324c9a551e60 nodeName:}" failed. No retries permitted until 2026-07-27 17:30:22.056363878 +0000 UTC m=+164.199765906 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b4f69c89-1bf8-483d-8854-324c9a551e60-metrics-tls") pod "dns-default-php8z" (UID: "b4f69c89-1bf8-483d-8854-324c9a551e60") : secret "dns-default-metrics-tls" not found
Jul 27 17:29:18.594037 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:18.594010    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zb877_2802ee72-4be9-423a-a9bd-102c54f96d06/node-ca/0.log"
Jul 27 17:29:18.963957 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:18.963890    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ab3e588-fe39-4747-8d1f-9b1ce44ffed4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-vx9kx\" (UID: \"7ab3e588-fe39-4747-8d1f-9b1ce44ffed4\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vx9kx"
Jul 27 17:29:18.964141 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:29:18.964083    2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 27 17:29:18.964205 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:29:18.964170    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ab3e588-fe39-4747-8d1f-9b1ce44ffed4-cluster-monitoring-operator-tls podName:7ab3e588-fe39-4747-8d1f-9b1ce44ffed4 nodeName:}" failed. No retries permitted until 2026-07-27 17:29:26.964147328 +0000 UTC m=+109.107549362 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7ab3e588-fe39-4747-8d1f-9b1ce44ffed4-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-vx9kx" (UID: "7ab3e588-fe39-4747-8d1f-9b1ce44ffed4") : secret "cluster-monitoring-operator-tls" not found
Jul 27 17:29:18.966595 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:18.966563    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:18.994907 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:18.994878    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6nssx_567f3aaf-ffed-47b5-97cf-2451d54b0db7/serve-healthcheck-canary/0.log"
Jul 27 17:29:19.065153 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:19.065112    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6f2becbf-a41c-4215-9435-9f735c60822e-registry-tls\") pod \"image-registry-6db8876459-vdgh6\" (UID: \"6f2becbf-a41c-4215-9435-9f735c60822e\") " pod="openshift-image-registry/image-registry-6db8876459-vdgh6"
Jul 27 17:29:19.065356 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:29:19.065289    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 27 17:29:19.065356 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:29:19.065311    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6db8876459-vdgh6: secret "image-registry-tls" not found
Jul 27 17:29:19.065442 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:29:19.065387    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f2becbf-a41c-4215-9435-9f735c60822e-registry-tls podName:6f2becbf-a41c-4215-9435-9f735c60822e nodeName:}" failed. No retries permitted until 2026-07-27 17:29:27.065370238 +0000 UTC m=+109.208772266 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6f2becbf-a41c-4215-9435-9f735c60822e-registry-tls") pod "image-registry-6db8876459-vdgh6" (UID: "6f2becbf-a41c-4215-9435-9f735c60822e") : secret "image-registry-tls" not found
Jul 27 17:29:19.067460 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:19.067377    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:19.166294 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:19.166233    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8ctdw\" (UID: \"fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4\") " pod="openshift-insights/insights-runtime-extractor-8ctdw"
Jul 27 17:29:19.166498 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:19.166334    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8ctdw\" (UID: \"fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4\") " pod="openshift-insights/insights-runtime-extractor-8ctdw"
Jul 27 17:29:19.166498 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:29:19.166388    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 17:29:19.166498 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:29:19.166423    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 17:29:19.166498 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:29:19.166455    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4-insights-runtime-extractor-tls podName:fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4 nodeName:}" failed. No retries permitted until 2026-07-27 17:29:23.166438813 +0000 UTC m=+105.309840842 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8ctdw" (UID: "fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4") : secret "insights-runtime-extractor-tls" not found
Jul 27 17:29:19.166498 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:29:19.166485    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4-kube-rbac-proxy-cm podName:fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4 nodeName:}" failed. No retries permitted until 2026-07-27 17:29:23.166469467 +0000 UTC m=+105.309871503 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8ctdw" (UID: "fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4") : configmap "kube-rbac-proxy" not found
Jul 27 17:29:19.267621 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:19.267544    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:19.359184 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:19.359144    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:19.572519 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:19.572470    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:19.687745 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:19.687700    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:19.802669 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:19.802581    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-dcbmp" event={"ID":"93ed2402-e28d-4671-b8bd-f7993a1afdd0","Type":"ContainerStarted","Data":"b94c93798ca79dc8539a198294a01266ee554976f716e034dfc0307fabe3dc0f"}
Jul 27 17:29:19.802669 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:19.802645    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-dcbmp" event={"ID":"93ed2402-e28d-4671-b8bd-f7993a1afdd0","Type":"ContainerStarted","Data":"b608a42b2c9f95fb92f507dd67fef2aab1cbd5bdde597cf942396d686eda2b5f"}
Jul 27 17:29:19.825502 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:19.825444    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-dcbmp" podStartSLOduration=1.102994831 podStartE2EDuration="2.825426798s" totalImagesPullingTime="1.722431967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:29:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:19.825273536 +0000 UTC m=+101.968675587" watchObservedRunningTime="2026-07-27 17:29:19.825426798 +0000 UTC m=+101.968828846"
Jul 27 17:29:19.826460 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:19.826394    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:19.902515 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:19.902488    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:19.960776 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:19.960748    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:20.062612 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:20.062518    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:20.267323 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:20.267286    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:20.731568 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:20.731534    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-sqfbc"
Jul 27 17:29:21.460909 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:21.459572    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:21.657669 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:21.657635    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:21.857590 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:21.857495    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:22.121943 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:22.121859    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:22.271762 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:22.271728    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:22.642566 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:22.642534    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:23.197642 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:23.197601    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8ctdw\" (UID: \"fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4\") " pod="openshift-insights/insights-runtime-extractor-8ctdw"
Jul 27 17:29:23.198092 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:23.197671    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8ctdw\" (UID: \"fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4\") " pod="openshift-insights/insights-runtime-extractor-8ctdw"
Jul 27 17:29:23.198092 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:29:23.197734    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 17:29:23.198092 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:29:23.197777    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 17:29:23.198092 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:29:23.197798    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4-kube-rbac-proxy-cm podName:fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4 nodeName:}" failed. No retries permitted until 2026-07-27 17:29:31.197784685 +0000 UTC m=+113.341186713 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8ctdw" (UID: "fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4") : configmap "kube-rbac-proxy" not found
Jul 27 17:29:23.198092 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:29:23.197865    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4-insights-runtime-extractor-tls podName:fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4 nodeName:}" failed. No retries permitted until 2026-07-27 17:29:31.197846532 +0000 UTC m=+113.341248561 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8ctdw" (UID: "fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4") : secret "insights-runtime-extractor-tls" not found
Jul 27 17:29:23.282437 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:23.282395    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:23.350204 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:23.350172    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:23.664360 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:23.664328    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:23.858997 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:23.858967    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:24.062299 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:24.062180    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:24.800099 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:24.800053    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:24.901523 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:24.901488    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:25.058079 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:25.057990    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:25.452359 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:25.452324    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:25.517347 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:25.517315    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:25.867227 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:25.867147    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:26.068184 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:26.068145    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:26.460634 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:26.460602    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:26.858771 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:26.858672    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:27.030351 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:27.030315    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ab3e588-fe39-4747-8d1f-9b1ce44ffed4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-vx9kx\" (UID: \"7ab3e588-fe39-4747-8d1f-9b1ce44ffed4\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vx9kx"
Jul 27 17:29:27.032866 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:27.032829    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7ab3e588-fe39-4747-8d1f-9b1ce44ffed4-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-vx9kx\" (UID: \"7ab3e588-fe39-4747-8d1f-9b1ce44ffed4\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vx9kx"
Jul 27 17:29:27.095909 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:27.095872    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vx9kx"
Jul 27 17:29:27.131464 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:27.131374    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6f2becbf-a41c-4215-9435-9f735c60822e-registry-tls\") pod \"image-registry-6db8876459-vdgh6\" (UID: \"6f2becbf-a41c-4215-9435-9f735c60822e\") " pod="openshift-image-registry/image-registry-6db8876459-vdgh6"
Jul 27 17:29:27.134045 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:27.134007    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6f2becbf-a41c-4215-9435-9f735c60822e-registry-tls\") pod \"image-registry-6db8876459-vdgh6\" (UID: \"6f2becbf-a41c-4215-9435-9f735c60822e\") " pod="openshift-image-registry/image-registry-6db8876459-vdgh6"
Jul 27 17:29:27.207986 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:27.207952    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6db8876459-vdgh6"
Jul 27 17:29:27.232393 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:27.232360    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-vx9kx"]
Jul 27 17:29:27.235074 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:29:27.235036    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ab3e588_fe39_4747_8d1f_9b1ce44ffed4.slice/crio-75304ea7434de3898ce82a857d989e9bc306b39eff687b1011c848ec2998ed9f WatchSource:0}: Error finding container 75304ea7434de3898ce82a857d989e9bc306b39eff687b1011c848ec2998ed9f: Status 404 returned error can't find the container with id 75304ea7434de3898ce82a857d989e9bc306b39eff687b1011c848ec2998ed9f
Jul 27 17:29:27.271826 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:27.271767    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:27.342367 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:27.342335    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6db8876459-vdgh6"]
Jul 27 17:29:27.345371 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:29:27.345342    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f2becbf_a41c_4215_9435_9f735c60822e.slice/crio-633b71016060ff13344a7000dacb6a1558b5d85ce105c990c18b19eb558466fe WatchSource:0}: Error finding container 633b71016060ff13344a7000dacb6a1558b5d85ce105c990c18b19eb558466fe: Status 404 returned error can't find the container with id 633b71016060ff13344a7000dacb6a1558b5d85ce105c990c18b19eb558466fe
Jul 27 17:29:27.832030 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:27.831990    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6db8876459-vdgh6" event={"ID":"6f2becbf-a41c-4215-9435-9f735c60822e","Type":"ContainerStarted","Data":"12119fa4a86bbac817678b8c8914b14f52a77a8593ebe191c1213155c79dee62"}
Jul 27 17:29:27.832230 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:27.832079    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6db8876459-vdgh6"
Jul 27 17:29:27.832230 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:27.832101    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6db8876459-vdgh6" event={"ID":"6f2becbf-a41c-4215-9435-9f735c60822e","Type":"ContainerStarted","Data":"633b71016060ff13344a7000dacb6a1558b5d85ce105c990c18b19eb558466fe"}
Jul 27 17:29:27.833172 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:27.833145    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vx9kx" event={"ID":"7ab3e588-fe39-4747-8d1f-9b1ce44ffed4","Type":"ContainerStarted","Data":"75304ea7434de3898ce82a857d989e9bc306b39eff687b1011c848ec2998ed9f"}
Jul 27 17:29:27.855211 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:27.855161    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6db8876459-vdgh6" podStartSLOduration=16.855145489999998 podStartE2EDuration="16.85514549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:29:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:27.854112224 +0000 UTC m=+109.997514276" watchObservedRunningTime="2026-07-27 17:29:27.85514549 +0000 UTC m=+109.998547541"
Jul 27 17:29:27.860577 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:27.860552    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:28.263064 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:28.263030    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:28.467102 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:28.467057    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:28.663098 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:28.663056    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:29.107594 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:29.107513    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:29.268710 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:29.268675    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:29.808979 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:29.808902    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:29.840762 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:29.840730    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vx9kx" event={"ID":"7ab3e588-fe39-4747-8d1f-9b1ce44ffed4","Type":"ContainerStarted","Data":"3dc2971aa714a12611d31ff54dfb333109e007a48ee92bbb8575cfac29ce08d2"}
Jul 27 17:29:29.860621 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:29.860568    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-vx9kx" podStartSLOduration=16.643279007 podStartE2EDuration="18.860551141s" totalImagesPullingTime="2.217272134s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:29:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:29.860057275 +0000 UTC m=+112.003459328" watchObservedRunningTime="2026-07-27 17:29:29.860551141 +0000 UTC m=+112.003953211"
Jul 27 17:29:29.879651 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:29.879615    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:29.970813 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:29.970778    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:30.070914 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:30.070806    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:30.267582 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:30.267548    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:30.664511 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:30.664479    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:31.075633 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:31.075545    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:31.265085 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:31.265045    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8ctdw\" (UID: \"fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4\") " pod="openshift-insights/insights-runtime-extractor-8ctdw"
Jul 27 17:29:31.265085 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:31.265093    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8ctdw\" (UID: \"fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4\") " pod="openshift-insights/insights-runtime-extractor-8ctdw"
Jul 27 17:29:31.265575 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:31.265559    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8ctdw\" (UID: \"fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4\") " pod="openshift-insights/insights-runtime-extractor-8ctdw"
Jul 27 17:29:31.267345 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:31.267326    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8ctdw\" (UID: \"fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4\") " pod="openshift-insights/insights-runtime-extractor-8ctdw"
Jul 27 17:29:31.347694 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:31.347586    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8ctdw"
Jul 27 17:29:31.477490 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:31.477458    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8ctdw"]
Jul 27 17:29:31.480219 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:29:31.480189    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd6ca9f1_b1b1_4258_964a_7a1c1276b3c4.slice/crio-16323fc5d24149fc42fb1b9dcfe69fec3bcea98bd5753b0db324d503005a1f50 WatchSource:0}: Error finding container 16323fc5d24149fc42fb1b9dcfe69fec3bcea98bd5753b0db324d503005a1f50: Status 404 returned error can't find the container with id 16323fc5d24149fc42fb1b9dcfe69fec3bcea98bd5753b0db324d503005a1f50
Jul 27 17:29:31.851360 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:31.851322    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8ctdw" event={"ID":"fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4","Type":"ContainerStarted","Data":"16323fc5d24149fc42fb1b9dcfe69fec3bcea98bd5753b0db324d503005a1f50"}
Jul 27 17:29:35.162179 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:35.162141    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:35.425560 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:35.425447    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:35.487868 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:35.487832    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:35.659022 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:35.658974    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:35.756408 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:35.756324    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:35.858098 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:35.858060    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:36.494609 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:36.494564    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:36.554193 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:36.554157    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:36.648452 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:36.648419    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:36.720132 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:36.720088    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:36.838584 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:36.838489    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:36.946745 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:36.946632    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:37.041691 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:37.041661    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:37.661054 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:37.661019    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:37.719113 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:37.719030    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:37.873689 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:37.873650    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8ctdw" event={"ID":"fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4","Type":"ContainerStarted","Data":"0738c275311427ccf1690d371150bc41be1b76350ed2a52549dff7ed18e0d1ed"}
Jul 27 17:29:39.292214 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:39.292171    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c547d977-nwxxl"]
Jul 27 17:29:39.303684 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:39.303656    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c547d977-nwxxl"
Jul 27 17:29:39.306630 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:39.306601    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 27 17:29:39.306775 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:39.306612    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 27 17:29:39.306775 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:39.306710    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 27 17:29:39.308129 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:39.308107    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 27 17:29:39.324422 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:39.324393    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c547d977-nwxxl"]
Jul 27 17:29:39.325014 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:39.324989    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9xsht"]
Jul 27 17:29:39.328971 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:39.328775    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9xsht"
Jul 27 17:29:39.331832 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:39.331811    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-wqrd8\""
Jul 27 17:29:39.332841 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:39.332819    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 27 17:29:39.336078 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:39.336055    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6db8876459-vdgh6"]
Jul 27 17:29:39.343138 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:39.343111    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9xsht"]
Jul 27 17:29:39.367032 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:39.366995    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-74648bb95f-zz4vg"]
Jul 27 17:29:39.370233 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:39.370209    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74648bb95f-zz4vg"
Jul 27 17:29:39.389648 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:39.389621    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-74648bb95f-zz4vg"]
Jul 27 17:29:39.403345 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:39.403307    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:39.430880 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:39.430842    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/da10c088-ba88-4668-8e2b-eda1f18d6203-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-9xsht\" (UID: \"da10c088-ba88-4668-8e2b-eda1f18d6203\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9xsht"
Jul 27 17:29:39.431033 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:39.430939    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjfl9\" (UniqueName: \"kubernetes.io/projected/ab4044fe-ea94-4f18-9569-571d8fcd95df-kube-api-access-sjfl9\") pod \"klusterlet-addon-workmgr-6c547d977-nwxxl\" (UID: \"ab4044fe-ea94-4f18-9569-571d8fcd95df\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c547d977-nwxxl"
Jul 27 17:29:39.431033 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:39.430989    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ab4044fe-ea94-4f18-9569-571d8fcd95df-klusterlet-config\") pod \"klusterlet-addon-workmgr-6c547d977-nwxxl\" (UID: \"ab4044fe-ea94-4f18-9569-571d8fcd95df\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c547d977-nwxxl"
Jul 27 17:29:39.431033 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:39.431019    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ab4044fe-ea94-4f18-9569-571d8fcd95df-tmp\") pod \"klusterlet-addon-workmgr-6c547d977-nwxxl\" (UID: \"ab4044fe-ea94-4f18-9569-571d8fcd95df\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c547d977-nwxxl"
Jul 27 17:29:39.508917 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:39.508885    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:39.532400 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:39.532354    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0e5f0900-1fec-4aa8-9372-45cb418b7bdd-registry-certificates\") pod \"image-registry-74648bb95f-zz4vg\" (UID: \"0e5f0900-1fec-4aa8-9372-45cb418b7bdd\") " pod="openshift-image-registry/image-registry-74648bb95f-zz4vg"
Jul 27 17:29:39.532400 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:39.532398    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjfl9\" (UniqueName: \"kubernetes.io/projected/ab4044fe-ea94-4f18-9569-571d8fcd95df-kube-api-access-sjfl9\") pod \"klusterlet-addon-workmgr-6c547d977-nwxxl\" (UID: \"ab4044fe-ea94-4f18-9569-571d8fcd95df\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c547d977-nwxxl"
Jul 27 17:29:39.532653 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:39.532427    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ab4044fe-ea94-4f18-9569-571d8fcd95df-klusterlet-config\") pod \"klusterlet-addon-workmgr-6c547d977-nwxxl\" (UID: \"ab4044fe-ea94-4f18-9569-571d8fcd95df\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c547d977-nwxxl"
Jul 27 17:29:39.532653 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:39.532490    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0e5f0900-1fec-4aa8-9372-45cb418b7bdd-installation-pull-secrets\") pod \"image-registry-74648bb95f-zz4vg\" (UID: \"0e5f0900-1fec-4aa8-9372-45cb418b7bdd\") " pod="openshift-image-registry/image-registry-74648bb95f-zz4vg"
Jul 27 17:29:39.532653 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:39.532551    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ab4044fe-ea94-4f18-9569-571d8fcd95df-tmp\") pod \"klusterlet-addon-workmgr-6c547d977-nwxxl\" (UID: \"ab4044fe-ea94-4f18-9569-571d8fcd95df\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c547d977-nwxxl"
Jul 27 17:29:39.532653 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:39.532599    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqpzx\" (UniqueName: \"kubernetes.io/projected/0e5f0900-1fec-4aa8-9372-45cb418b7bdd-kube-api-access-vqpzx\") pod \"image-registry-74648bb95f-zz4vg\" (UID: \"0e5f0900-1fec-4aa8-9372-45cb418b7bdd\") " pod="openshift-image-registry/image-registry-74648bb95f-zz4vg"
Jul 27 17:29:39.532653 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:39.532629    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/da10c088-ba88-4668-8e2b-eda1f18d6203-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-9xsht\" (UID: \"da10c088-ba88-4668-8e2b-eda1f18d6203\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9xsht"
Jul 27 17:29:39.532924 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:39.532708    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0e5f0900-1fec-4aa8-9372-45cb418b7bdd-registry-tls\") pod \"image-registry-74648bb95f-zz4vg\" (UID: \"0e5f0900-1fec-4aa8-9372-45cb418b7bdd\") " pod="openshift-image-registry/image-registry-74648bb95f-zz4vg"
Jul 27 17:29:39.532924 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:39.532735    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0e5f0900-1fec-4aa8-9372-45cb418b7bdd-trusted-ca\") pod \"image-registry-74648bb95f-zz4vg\" (UID: \"0e5f0900-1fec-4aa8-9372-45cb418b7bdd\") " pod="openshift-image-registry/image-registry-74648bb95f-zz4vg"
Jul 27 17:29:39.532924 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:39.532776    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0e5f0900-1fec-4aa8-9372-45cb418b7bdd-image-registry-private-configuration\") pod \"image-registry-74648bb95f-zz4vg\" (UID: \"0e5f0900-1fec-4aa8-9372-45cb418b7bdd\") " pod="openshift-image-registry/image-registry-74648bb95f-zz4vg"
Jul 27 17:29:39.532924 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:39.532805    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0e5f0900-1fec-4aa8-9372-45cb418b7bdd-bound-sa-token\") pod \"image-registry-74648bb95f-zz4vg\" (UID: \"0e5f0900-1fec-4aa8-9372-45cb418b7bdd\") " pod="openshift-image-registry/image-registry-74648bb95f-zz4vg"
Jul 27 17:29:39.532924 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:39.532834    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0e5f0900-1fec-4aa8-9372-45cb418b7bdd-ca-trust-extracted\") pod \"image-registry-74648bb95f-zz4vg\" (UID: \"0e5f0900-1fec-4aa8-9372-45cb418b7bdd\") " pod="openshift-image-registry/image-registry-74648bb95f-zz4vg"
Jul 27 17:29:39.533161 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:39.533006    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ab4044fe-ea94-4f18-9569-571d8fcd95df-tmp\") pod \"klusterlet-addon-workmgr-6c547d977-nwxxl\" (UID: \"ab4044fe-ea94-4f18-9569-571d8fcd95df\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c547d977-nwxxl"
Jul 27 17:29:39.535351 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:39.535327    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ab4044fe-ea94-4f18-9569-571d8fcd95df-klusterlet-config\") pod \"klusterlet-addon-workmgr-6c547d977-nwxxl\" (UID: \"ab4044fe-ea94-4f18-9569-571d8fcd95df\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c547d977-nwxxl"
Jul 27 17:29:39.535464 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:39.535424    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/da10c088-ba88-4668-8e2b-eda1f18d6203-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-9xsht\" (UID: \"da10c088-ba88-4668-8e2b-eda1f18d6203\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9xsht"
Jul 27 17:29:39.543893 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:39.543833    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjfl9\" (UniqueName: \"kubernetes.io/projected/ab4044fe-ea94-4f18-9569-571d8fcd95df-kube-api-access-sjfl9\") pod \"klusterlet-addon-workmgr-6c547d977-nwxxl\" (UID: \"ab4044fe-ea94-4f18-9569-571d8fcd95df\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c547d977-nwxxl"
Jul 27 17:29:39.610010 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:39.609972    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:39.615842 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:39.615811    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c547d977-nwxxl"
Jul 27 17:29:39.633910 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:39.633873    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0e5f0900-1fec-4aa8-9372-45cb418b7bdd-registry-tls\") pod \"image-registry-74648bb95f-zz4vg\" (UID: \"0e5f0900-1fec-4aa8-9372-45cb418b7bdd\") " pod="openshift-image-registry/image-registry-74648bb95f-zz4vg"
Jul 27 17:29:39.634076 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:39.633919    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0e5f0900-1fec-4aa8-9372-45cb418b7bdd-trusted-ca\") pod \"image-registry-74648bb95f-zz4vg\" (UID: \"0e5f0900-1fec-4aa8-9372-45cb418b7bdd\") " pod="openshift-image-registry/image-registry-74648bb95f-zz4vg"
Jul 27 17:29:39.634076 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:39.633954    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0e5f0900-1fec-4aa8-9372-45cb418b7bdd-image-registry-private-configuration\") pod \"image-registry-74648bb95f-zz4vg\" (UID: \"0e5f0900-1fec-4aa8-9372-45cb418b7bdd\") " pod="openshift-image-registry/image-registry-74648bb95f-zz4vg"
Jul 27 17:29:39.634076 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:39.633980    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0e5f0900-1fec-4aa8-9372-45cb418b7bdd-bound-sa-token\") pod \"image-registry-74648bb95f-zz4vg\" (UID: \"0e5f0900-1fec-4aa8-9372-45cb418b7bdd\") " pod="openshift-image-registry/image-registry-74648bb95f-zz4vg"
Jul 27 17:29:39.634275 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:39.634150    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0e5f0900-1fec-4aa8-9372-45cb418b7bdd-ca-trust-extracted\") pod \"image-registry-74648bb95f-zz4vg\" (UID: \"0e5f0900-1fec-4aa8-9372-45cb418b7bdd\") " pod="openshift-image-registry/image-registry-74648bb95f-zz4vg"
Jul 27 17:29:39.634275 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:39.634200    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0e5f0900-1fec-4aa8-9372-45cb418b7bdd-registry-certificates\") pod \"image-registry-74648bb95f-zz4vg\" (UID: \"0e5f0900-1fec-4aa8-9372-45cb418b7bdd\") " pod="openshift-image-registry/image-registry-74648bb95f-zz4vg"
Jul 27 17:29:39.634388 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:39.634304    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0e5f0900-1fec-4aa8-9372-45cb418b7bdd-installation-pull-secrets\") pod \"image-registry-74648bb95f-zz4vg\" (UID: \"0e5f0900-1fec-4aa8-9372-45cb418b7bdd\") " pod="openshift-image-registry/image-registry-74648bb95f-zz4vg"
Jul 27 17:29:39.634388 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:39.634375    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqpzx\" (UniqueName: \"kubernetes.io/projected/0e5f0900-1fec-4aa8-9372-45cb418b7bdd-kube-api-access-vqpzx\") pod \"image-registry-74648bb95f-zz4vg\" (UID: \"0e5f0900-1fec-4aa8-9372-45cb418b7bdd\") " pod="openshift-image-registry/image-registry-74648bb95f-zz4vg"
Jul 27 17:29:39.634669 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:39.634646    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0e5f0900-1fec-4aa8-9372-45cb418b7bdd-ca-trust-extracted\") pod \"image-registry-74648bb95f-zz4vg\" (UID: \"0e5f0900-1fec-4aa8-9372-45cb418b7bdd\") " pod="openshift-image-registry/image-registry-74648bb95f-zz4vg"
Jul 27 17:29:39.635054 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:39.635031    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0e5f0900-1fec-4aa8-9372-45cb418b7bdd-registry-certificates\") pod \"image-registry-74648bb95f-zz4vg\" (UID: \"0e5f0900-1fec-4aa8-9372-45cb418b7bdd\") " pod="openshift-image-registry/image-registry-74648bb95f-zz4vg"
Jul 27 17:29:39.635161 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:39.635076    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0e5f0900-1fec-4aa8-9372-45cb418b7bdd-trusted-ca\") pod \"image-registry-74648bb95f-zz4vg\" (UID: \"0e5f0900-1fec-4aa8-9372-45cb418b7bdd\") " pod="openshift-image-registry/image-registry-74648bb95f-zz4vg"
Jul 27 17:29:39.637339 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:39.637317    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0e5f0900-1fec-4aa8-9372-45cb418b7bdd-registry-tls\") pod \"image-registry-74648bb95f-zz4vg\" (UID: \"0e5f0900-1fec-4aa8-9372-45cb418b7bdd\") " pod="openshift-image-registry/image-registry-74648bb95f-zz4vg"
Jul 27 17:29:39.637339 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:39.637328    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0e5f0900-1fec-4aa8-9372-45cb418b7bdd-image-registry-private-configuration\") pod \"image-registry-74648bb95f-zz4vg\" (UID: \"0e5f0900-1fec-4aa8-9372-45cb418b7bdd\") " pod="openshift-image-registry/image-registry-74648bb95f-zz4vg"
Jul 27 17:29:39.637527 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:39.637494    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0e5f0900-1fec-4aa8-9372-45cb418b7bdd-installation-pull-secrets\") pod \"image-registry-74648bb95f-zz4vg\" (UID: \"0e5f0900-1fec-4aa8-9372-45cb418b7bdd\") " pod="openshift-image-registry/image-registry-74648bb95f-zz4vg"
Jul 27 17:29:39.642461 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:39.642427    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9xsht"
Jul 27 17:29:39.645302 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:39.645278    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0e5f0900-1fec-4aa8-9372-45cb418b7bdd-bound-sa-token\") pod \"image-registry-74648bb95f-zz4vg\" (UID: \"0e5f0900-1fec-4aa8-9372-45cb418b7bdd\") " pod="openshift-image-registry/image-registry-74648bb95f-zz4vg"
Jul 27 17:29:39.645580 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:39.645501    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqpzx\" (UniqueName: \"kubernetes.io/projected/0e5f0900-1fec-4aa8-9372-45cb418b7bdd-kube-api-access-vqpzx\") pod \"image-registry-74648bb95f-zz4vg\" (UID: \"0e5f0900-1fec-4aa8-9372-45cb418b7bdd\") " pod="openshift-image-registry/image-registry-74648bb95f-zz4vg"
Jul 27 17:29:39.680454 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:39.680418    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74648bb95f-zz4vg"
Jul 27 17:29:40.746946 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:40.746915    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-74648bb95f-zz4vg"]
Jul 27 17:29:40.752174 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:40.752152    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9xsht"]
Jul 27 17:29:40.755828 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:40.755803    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c547d977-nwxxl"]
Jul 27 17:29:40.756723 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:29:40.756703    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda10c088_ba88_4668_8e2b_eda1f18d6203.slice/crio-af8a1ae3d25786c06280dc6ca94dc1c45713932ac2b182059ff27ad9ed766e4e WatchSource:0}: Error finding container af8a1ae3d25786c06280dc6ca94dc1c45713932ac2b182059ff27ad9ed766e4e: Status 404 returned error can't find the container with id af8a1ae3d25786c06280dc6ca94dc1c45713932ac2b182059ff27ad9ed766e4e
Jul 27 17:29:40.759056 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:29:40.759029    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab4044fe_ea94_4f18_9569_571d8fcd95df.slice/crio-3e6052c9326a5703bad09e5eca74ffd9c07d5453f5fc8744b2c27c8d3d0fc3b1 WatchSource:0}: Error finding container 3e6052c9326a5703bad09e5eca74ffd9c07d5453f5fc8744b2c27c8d3d0fc3b1: Status 404 returned error can't find the container with id 3e6052c9326a5703bad09e5eca74ffd9c07d5453f5fc8744b2c27c8d3d0fc3b1
Jul 27 17:29:40.885461 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:40.885348    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74648bb95f-zz4vg" event={"ID":"0e5f0900-1fec-4aa8-9372-45cb418b7bdd","Type":"ContainerStarted","Data":"f12c8cff73e6c4071cba1c05b023765c1089b45d5c316097902fee3fcf7337fb"}
Jul 27 17:29:40.885461 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:40.885412    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74648bb95f-zz4vg" event={"ID":"0e5f0900-1fec-4aa8-9372-45cb418b7bdd","Type":"ContainerStarted","Data":"b4bb89a13b712ea6cef4f965269bb06ef6d5be8e26b62217e9e0ebe818621f14"}
Jul 27 17:29:40.885461 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:40.885437    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-74648bb95f-zz4vg"
Jul 27 17:29:40.887160 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:40.887132    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8ctdw" event={"ID":"fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4","Type":"ContainerStarted","Data":"18d0c4a5e6dcf677f25d8081281ae6ce0eb39ad79c8c80ef264227eb51cd0f8e"}
Jul 27 17:29:40.888223 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:40.888196    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c547d977-nwxxl" event={"ID":"ab4044fe-ea94-4f18-9569-571d8fcd95df","Type":"ContainerStarted","Data":"3e6052c9326a5703bad09e5eca74ffd9c07d5453f5fc8744b2c27c8d3d0fc3b1"}
Jul 27 17:29:40.889367 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:40.889347    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9xsht" event={"ID":"da10c088-ba88-4668-8e2b-eda1f18d6203","Type":"ContainerStarted","Data":"af8a1ae3d25786c06280dc6ca94dc1c45713932ac2b182059ff27ad9ed766e4e"}
Jul 27 17:29:40.906689 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:40.906639    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-74648bb95f-zz4vg" podStartSLOduration=1.906624967 podStartE2EDuration="1.906624967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:29:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:40.905698183 +0000 UTC m=+123.049100246" watchObservedRunningTime="2026-07-27 17:29:40.906624967 +0000 UTC m=+123.050026995"
Jul 27 17:29:45.854879 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:45.854835    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:45.927518 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:45.927482    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:46.916873 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:46.916834    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8ctdw" event={"ID":"fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4","Type":"ContainerStarted","Data":"4773ccdbac68fa90a39aac7d87f4c7c5e9c2d816f7daf7cd890473dc5cb8a050"}
Jul 27 17:29:46.918319 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:46.918292    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c547d977-nwxxl" event={"ID":"ab4044fe-ea94-4f18-9569-571d8fcd95df","Type":"ContainerStarted","Data":"bd06a8d38a5bcf52475294bf190222b8785acad4824827ebd58fb2cf1e335751"}
Jul 27 17:29:46.918498 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:46.918480    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c547d977-nwxxl"
Jul 27 17:29:46.919791 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:46.919764    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9xsht" event={"ID":"da10c088-ba88-4668-8e2b-eda1f18d6203","Type":"ContainerStarted","Data":"0ebeff1a0447bb22e658c4486422d39cd7ed10ec2348d4cfe34a29b70ecce5dc"}
Jul 27 17:29:46.919984 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:46.919966    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9xsht"
Jul 27 17:29:46.920497 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:46.920479    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c547d977-nwxxl"
Jul 27 17:29:46.924268 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:46.924236    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9xsht"
Jul 27 17:29:46.975793 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:46.975737    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8ctdw" podStartSLOduration=17.340858238 podStartE2EDuration="31.975720594s" totalImagesPullingTime="14.634862356s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:29:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:46.939634776 +0000 UTC m=+129.083036826" watchObservedRunningTime="2026-07-27 17:29:46.975720594 +0000 UTC m=+129.119122645"
Jul 27 17:29:46.976283 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:46.976239    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6c547d977-nwxxl" podStartSLOduration=2.475469132 podStartE2EDuration="7.976229347s" totalImagesPullingTime="5.500760215s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:29:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:46.975645622 +0000 UTC m=+129.119047672" watchObservedRunningTime="2026-07-27 17:29:46.976229347 +0000 UTC m=+129.119631397"
Jul 27 17:29:47.006838 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:47.006788    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9xsht" podStartSLOduration=2.52338614 podStartE2EDuration="8.006770926s" totalImagesPullingTime="5.483384786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:29:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:47.006312434 +0000 UTC m=+129.149714486" watchObservedRunningTime="2026-07-27 17:29:47.006770926 +0000 UTC m=+129.150172976"
Jul 27 17:29:48.020945 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:48.020916    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:48.102494 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:48.102458    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:48.106963 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:48.106939    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-sscsd"]
Jul 27 17:29:48.110109 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:48.110094    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-sscsd"
Jul 27 17:29:48.112752 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:48.112731    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fedc794-294b-44d7-ba67-55dbad330693-metrics-certs\") pod \"network-metrics-daemon-zxdxx\" (UID: \"2fedc794-294b-44d7-ba67-55dbad330693\") " pod="openshift-multus/network-metrics-daemon-zxdxx"
Jul 27 17:29:48.114647 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:48.114627    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 27 17:29:48.114743 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:48.114627    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 27 17:29:48.114743 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:48.114655    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-6nc66\""
Jul 27 17:29:48.114861 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:48.114656    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 27 17:29:48.115178 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:48.115153    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fedc794-294b-44d7-ba67-55dbad330693-metrics-certs\") pod \"network-metrics-daemon-zxdxx\" (UID: \"2fedc794-294b-44d7-ba67-55dbad330693\") " pod="openshift-multus/network-metrics-daemon-zxdxx"
Jul 27 17:29:48.119547 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:48.119520    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-sscsd"]
Jul 27 17:29:48.176236 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:48.176201    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-p44n4\""
Jul 27 17:29:48.183853 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:48.183835    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zxdxx"
Jul 27 17:29:48.213320 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:48.213289    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzqs6\" (UniqueName: \"kubernetes.io/projected/8cf5e605-7a6e-4cf6-af0c-5e7b19038568-kube-api-access-qzqs6\") pod \"prometheus-operator-84d69f794d-sscsd\" (UID: \"8cf5e605-7a6e-4cf6-af0c-5e7b19038568\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-sscsd"
Jul 27 17:29:48.213460 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:48.213328    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8cf5e605-7a6e-4cf6-af0c-5e7b19038568-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-sscsd\" (UID: \"8cf5e605-7a6e-4cf6-af0c-5e7b19038568\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-sscsd"
Jul 27 17:29:48.213460 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:48.213349    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8cf5e605-7a6e-4cf6-af0c-5e7b19038568-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-sscsd\" (UID: \"8cf5e605-7a6e-4cf6-af0c-5e7b19038568\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-sscsd"
Jul 27 17:29:48.213460 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:48.213375    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8cf5e605-7a6e-4cf6-af0c-5e7b19038568-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-sscsd\" (UID: \"8cf5e605-7a6e-4cf6-af0c-5e7b19038568\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-sscsd"
Jul 27 17:29:48.310366 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:48.310284    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zxdxx"]
Jul 27 17:29:48.313473 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:29:48.313450    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fedc794_294b_44d7_ba67_55dbad330693.slice/crio-a6cc58f265211548111d646ca9f68ee4eb1ae57017562c8b59925e767ded4ef3 WatchSource:0}: Error finding container a6cc58f265211548111d646ca9f68ee4eb1ae57017562c8b59925e767ded4ef3: Status 404 returned error can't find the container with id a6cc58f265211548111d646ca9f68ee4eb1ae57017562c8b59925e767ded4ef3
Jul 27 17:29:48.314437 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:48.314406    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8cf5e605-7a6e-4cf6-af0c-5e7b19038568-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-sscsd\" (UID: \"8cf5e605-7a6e-4cf6-af0c-5e7b19038568\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-sscsd"
Jul 27 17:29:48.314578 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:48.314458    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8cf5e605-7a6e-4cf6-af0c-5e7b19038568-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-sscsd\" (UID: \"8cf5e605-7a6e-4cf6-af0c-5e7b19038568\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-sscsd"
Jul 27 17:29:48.314578 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:48.314494    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8cf5e605-7a6e-4cf6-af0c-5e7b19038568-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-sscsd\" (UID: \"8cf5e605-7a6e-4cf6-af0c-5e7b19038568\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-sscsd"
Jul 27 17:29:48.314578 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:29:48.314553    2573 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 27 17:29:48.314749 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:29:48.314605    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cf5e605-7a6e-4cf6-af0c-5e7b19038568-prometheus-operator-tls podName:8cf5e605-7a6e-4cf6-af0c-5e7b19038568 nodeName:}" failed. No retries permitted until 2026-07-27 17:29:48.814590846 +0000 UTC m=+130.957992875 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/8cf5e605-7a6e-4cf6-af0c-5e7b19038568-prometheus-operator-tls") pod "prometheus-operator-84d69f794d-sscsd" (UID: "8cf5e605-7a6e-4cf6-af0c-5e7b19038568") : secret "prometheus-operator-tls" not found
Jul 27 17:29:48.314749 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:48.314652    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzqs6\" (UniqueName: \"kubernetes.io/projected/8cf5e605-7a6e-4cf6-af0c-5e7b19038568-kube-api-access-qzqs6\") pod \"prometheus-operator-84d69f794d-sscsd\" (UID: \"8cf5e605-7a6e-4cf6-af0c-5e7b19038568\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-sscsd"
Jul 27 17:29:48.315303 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:48.315278    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8cf5e605-7a6e-4cf6-af0c-5e7b19038568-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-sscsd\" (UID: \"8cf5e605-7a6e-4cf6-af0c-5e7b19038568\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-sscsd"
Jul 27 17:29:48.317084 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:48.317041    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8cf5e605-7a6e-4cf6-af0c-5e7b19038568-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-sscsd\" (UID: \"8cf5e605-7a6e-4cf6-af0c-5e7b19038568\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-sscsd"
Jul 27 17:29:48.339271 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:48.336026    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzqs6\" (UniqueName: \"kubernetes.io/projected/8cf5e605-7a6e-4cf6-af0c-5e7b19038568-kube-api-access-qzqs6\") pod \"prometheus-operator-84d69f794d-sscsd\" (UID: \"8cf5e605-7a6e-4cf6-af0c-5e7b19038568\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-sscsd"
Jul 27 17:29:48.819386 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:48.819341    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8cf5e605-7a6e-4cf6-af0c-5e7b19038568-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-sscsd\" (UID: \"8cf5e605-7a6e-4cf6-af0c-5e7b19038568\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-sscsd"
Jul 27 17:29:48.819576 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:29:48.819506    2573 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 27 17:29:48.819636 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:29:48.819576    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cf5e605-7a6e-4cf6-af0c-5e7b19038568-prometheus-operator-tls podName:8cf5e605-7a6e-4cf6-af0c-5e7b19038568 nodeName:}" failed. No retries permitted until 2026-07-27 17:29:49.819559339 +0000 UTC m=+131.962961372 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/8cf5e605-7a6e-4cf6-af0c-5e7b19038568-prometheus-operator-tls") pod "prometheus-operator-84d69f794d-sscsd" (UID: "8cf5e605-7a6e-4cf6-af0c-5e7b19038568") : secret "prometheus-operator-tls" not found
Jul 27 17:29:48.927156 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:48.927121    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zxdxx" event={"ID":"2fedc794-294b-44d7-ba67-55dbad330693","Type":"ContainerStarted","Data":"a6cc58f265211548111d646ca9f68ee4eb1ae57017562c8b59925e767ded4ef3"}
Jul 27 17:29:49.197976 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:49.197938    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:49.299308 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:49.299241    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:49.342088 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:49.342055    2573 patch_prober.go:28] interesting pod/image-registry-6db8876459-vdgh6 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 27 17:29:49.342257 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:49.342110    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6db8876459-vdgh6" podUID="6f2becbf-a41c-4215-9435-9f735c60822e" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 17:29:49.399376 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:49.399341    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:49.828536 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:49.828494    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8cf5e605-7a6e-4cf6-af0c-5e7b19038568-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-sscsd\" (UID: \"8cf5e605-7a6e-4cf6-af0c-5e7b19038568\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-sscsd"
Jul 27 17:29:49.831021 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:49.830993    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8cf5e605-7a6e-4cf6-af0c-5e7b19038568-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-sscsd\" (UID: \"8cf5e605-7a6e-4cf6-af0c-5e7b19038568\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-sscsd"
Jul 27 17:29:49.925468 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:49.925419    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-sscsd"
Jul 27 17:29:50.054973 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:50.054953    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-sscsd"]
Jul 27 17:29:50.057430 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:29:50.057388    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cf5e605_7a6e_4cf6_af0c_5e7b19038568.slice/crio-a891da077ef29b39c70452bd98ec94e492ab51f2c3e2b8fd00fa51c275313607 WatchSource:0}: Error finding container a891da077ef29b39c70452bd98ec94e492ab51f2c3e2b8fd00fa51c275313607: Status 404 returned error can't find the container with id a891da077ef29b39c70452bd98ec94e492ab51f2c3e2b8fd00fa51c275313607
Jul 27 17:29:50.935948 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:50.935905    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-sscsd" event={"ID":"8cf5e605-7a6e-4cf6-af0c-5e7b19038568","Type":"ContainerStarted","Data":"a891da077ef29b39c70452bd98ec94e492ab51f2c3e2b8fd00fa51c275313607"}
Jul 27 17:29:51.688126 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:51.688052    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:51.793517 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:51.793477    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:51.863139 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:51.863104    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:51.940701 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:51.940608    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-sscsd" event={"ID":"8cf5e605-7a6e-4cf6-af0c-5e7b19038568","Type":"ContainerStarted","Data":"561b129ad7b485672270252969cf4aa4d11ade5c6b07109236a8c2158d1de3d2"}
Jul 27 17:29:51.940701 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:51.940676    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-sscsd" event={"ID":"8cf5e605-7a6e-4cf6-af0c-5e7b19038568","Type":"ContainerStarted","Data":"a48247c90fd9a8ea7212b48ddf3a09140e16f70359b0f7a3b2660b3aedd01448"}
Jul 27 17:29:51.942366 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:51.942329    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zxdxx" event={"ID":"2fedc794-294b-44d7-ba67-55dbad330693","Type":"ContainerStarted","Data":"bb4369a4a8dbfc9061f1ef55561a6b669545fdbf56eada58a05a863d1a901266"}
Jul 27 17:29:51.942486 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:51.942383    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zxdxx" event={"ID":"2fedc794-294b-44d7-ba67-55dbad330693","Type":"ContainerStarted","Data":"ee13888c398d0faafa157a862ebe0073b3c58fd228657ba21e975bbce1b1d29c"}
Jul 27 17:29:51.969449 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:51.969398    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-sscsd" podStartSLOduration=2.623706866 podStartE2EDuration="3.969382307s" totalImagesPullingTime="1.345675441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:29:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:51.96724058 +0000 UTC m=+134.110642645" watchObservedRunningTime="2026-07-27 17:29:51.969382307 +0000 UTC m=+134.112784388"
Jul 27 17:29:51.992626 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:51.992576    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-zxdxx" podStartSLOduration=130.905255629 podStartE2EDuration="2m13.99256071s" totalImagesPullingTime="3.087305081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:27:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:51.991625241 +0000 UTC m=+134.135027305" watchObservedRunningTime="2026-07-27 17:29:51.99256071 +0000 UTC m=+134.135962813"
Jul 27 17:29:52.093103 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:52.093063    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:53.525818 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:53.525782    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-rxfs4"]
Jul 27 17:29:53.530437 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:53.530414    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-chc8q"]
Jul 27 17:29:53.530553 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:53.530534    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rxfs4"
Jul 27 17:29:53.533477 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:53.533455    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 27 17:29:53.533477 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:53.533465    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 27 17:29:53.533834 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:53.533821    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-gqtsf\""
Jul 27 17:29:53.534574 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:53.534555    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-chc8q"
Jul 27 17:29:53.537165 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:53.537148    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 27 17:29:53.537438 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:53.537419    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 27 17:29:53.537545 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:53.537489    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 27 17:29:53.538170 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:53.537786    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-lknvb\""
Jul 27 17:29:53.543080 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:53.543055    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-rxfs4"]
Jul 27 17:29:53.658561 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:53.658523    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61385c00-c295-4f64-a5c8-d5f0dd2ce6af-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-rxfs4\" (UID: \"61385c00-c295-4f64-a5c8-d5f0dd2ce6af\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rxfs4"
Jul 27 17:29:53.658561 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:53.658564    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61385c00-c295-4f64-a5c8-d5f0dd2ce6af-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-rxfs4\" (UID: \"61385c00-c295-4f64-a5c8-d5f0dd2ce6af\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rxfs4"
Jul 27 17:29:53.658808 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:53.658591    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4cabfc1-9cab-4eb9-8e32-7dd66c04d572-metrics-client-ca\") pod \"node-exporter-chc8q\" (UID: \"e4cabfc1-9cab-4eb9-8e32-7dd66c04d572\") " pod="openshift-monitoring/node-exporter-chc8q"
Jul 27 17:29:53.658808 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:53.658621    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e4cabfc1-9cab-4eb9-8e32-7dd66c04d572-node-exporter-textfile\") pod \"node-exporter-chc8q\" (UID: \"e4cabfc1-9cab-4eb9-8e32-7dd66c04d572\") " pod="openshift-monitoring/node-exporter-chc8q"
Jul 27 17:29:53.658808 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:53.658661    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/61385c00-c295-4f64-a5c8-d5f0dd2ce6af-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-rxfs4\" (UID: \"61385c00-c295-4f64-a5c8-d5f0dd2ce6af\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rxfs4"
Jul 27 17:29:53.658808 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:53.658686    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e4cabfc1-9cab-4eb9-8e32-7dd66c04d572-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-chc8q\" (UID: \"e4cabfc1-9cab-4eb9-8e32-7dd66c04d572\") " pod="openshift-monitoring/node-exporter-chc8q"
Jul 27 17:29:53.658808 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:53.658745    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e4cabfc1-9cab-4eb9-8e32-7dd66c04d572-node-exporter-tls\") pod \"node-exporter-chc8q\" (UID: \"e4cabfc1-9cab-4eb9-8e32-7dd66c04d572\") " pod="openshift-monitoring/node-exporter-chc8q"
Jul 27 17:29:53.658808 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:53.658779    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x9n6\" (UniqueName: \"kubernetes.io/projected/61385c00-c295-4f64-a5c8-d5f0dd2ce6af-kube-api-access-5x9n6\") pod \"openshift-state-metrics-b8f49f5c6-rxfs4\" (UID: \"61385c00-c295-4f64-a5c8-d5f0dd2ce6af\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rxfs4"
Jul 27 17:29:53.658808 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:53.658796    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e4cabfc1-9cab-4eb9-8e32-7dd66c04d572-node-exporter-accelerators-collector-config\") pod \"node-exporter-chc8q\" (UID: \"e4cabfc1-9cab-4eb9-8e32-7dd66c04d572\") " pod="openshift-monitoring/node-exporter-chc8q"
Jul 27 17:29:53.659049 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:53.658820    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58zl8\" (UniqueName: \"kubernetes.io/projected/e4cabfc1-9cab-4eb9-8e32-7dd66c04d572-kube-api-access-58zl8\") pod \"node-exporter-chc8q\" (UID: \"e4cabfc1-9cab-4eb9-8e32-7dd66c04d572\") " pod="openshift-monitoring/node-exporter-chc8q"
Jul 27 17:29:53.659049 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:53.658858    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e4cabfc1-9cab-4eb9-8e32-7dd66c04d572-node-exporter-wtmp\") pod \"node-exporter-chc8q\" (UID: \"e4cabfc1-9cab-4eb9-8e32-7dd66c04d572\") " pod="openshift-monitoring/node-exporter-chc8q"
Jul 27 17:29:53.659049 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:53.658881    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e4cabfc1-9cab-4eb9-8e32-7dd66c04d572-root\") pod \"node-exporter-chc8q\" (UID: \"e4cabfc1-9cab-4eb9-8e32-7dd66c04d572\") " pod="openshift-monitoring/node-exporter-chc8q"
Jul 27 17:29:53.659049 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:53.658907    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e4cabfc1-9cab-4eb9-8e32-7dd66c04d572-sys\") pod \"node-exporter-chc8q\" (UID: \"e4cabfc1-9cab-4eb9-8e32-7dd66c04d572\") " pod="openshift-monitoring/node-exporter-chc8q"
Jul 27 17:29:53.759855 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:53.759821    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/61385c00-c295-4f64-a5c8-d5f0dd2ce6af-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-rxfs4\" (UID: \"61385c00-c295-4f64-a5c8-d5f0dd2ce6af\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rxfs4"
Jul 27 17:29:53.759855 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:53.759864    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e4cabfc1-9cab-4eb9-8e32-7dd66c04d572-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-chc8q\" (UID: \"e4cabfc1-9cab-4eb9-8e32-7dd66c04d572\") " pod="openshift-monitoring/node-exporter-chc8q"
Jul 27 17:29:53.760120 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:53.759893    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e4cabfc1-9cab-4eb9-8e32-7dd66c04d572-node-exporter-tls\") pod \"node-exporter-chc8q\" (UID: \"e4cabfc1-9cab-4eb9-8e32-7dd66c04d572\") " pod="openshift-monitoring/node-exporter-chc8q"
Jul 27 17:29:53.760120 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:29:53.759981    2573 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 27 17:29:53.760120 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:53.760018    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x9n6\" (UniqueName: \"kubernetes.io/projected/61385c00-c295-4f64-a5c8-d5f0dd2ce6af-kube-api-access-5x9n6\") pod \"openshift-state-metrics-b8f49f5c6-rxfs4\" (UID: \"61385c00-c295-4f64-a5c8-d5f0dd2ce6af\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rxfs4"
Jul 27 17:29:53.760120 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:29:53.760044    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4cabfc1-9cab-4eb9-8e32-7dd66c04d572-node-exporter-tls podName:e4cabfc1-9cab-4eb9-8e32-7dd66c04d572 nodeName:}" failed. No retries permitted until 2026-07-27 17:29:54.260023505 +0000 UTC m=+136.403425549 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/e4cabfc1-9cab-4eb9-8e32-7dd66c04d572-node-exporter-tls") pod "node-exporter-chc8q" (UID: "e4cabfc1-9cab-4eb9-8e32-7dd66c04d572") : secret "node-exporter-tls" not found
Jul 27 17:29:53.760120 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:53.760071    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e4cabfc1-9cab-4eb9-8e32-7dd66c04d572-node-exporter-accelerators-collector-config\") pod \"node-exporter-chc8q\" (UID: \"e4cabfc1-9cab-4eb9-8e32-7dd66c04d572\") " pod="openshift-monitoring/node-exporter-chc8q"
Jul 27 17:29:53.760120 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:53.760112    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58zl8\" (UniqueName: \"kubernetes.io/projected/e4cabfc1-9cab-4eb9-8e32-7dd66c04d572-kube-api-access-58zl8\") pod \"node-exporter-chc8q\" (UID: \"e4cabfc1-9cab-4eb9-8e32-7dd66c04d572\") " pod="openshift-monitoring/node-exporter-chc8q"
Jul 27 17:29:53.760460 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:53.760138    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e4cabfc1-9cab-4eb9-8e32-7dd66c04d572-node-exporter-wtmp\") pod \"node-exporter-chc8q\" (UID: \"e4cabfc1-9cab-4eb9-8e32-7dd66c04d572\") " pod="openshift-monitoring/node-exporter-chc8q"
Jul 27 17:29:53.760460 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:53.760166    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e4cabfc1-9cab-4eb9-8e32-7dd66c04d572-root\") pod \"node-exporter-chc8q\" (UID: \"e4cabfc1-9cab-4eb9-8e32-7dd66c04d572\") " pod="openshift-monitoring/node-exporter-chc8q"
Jul 27 17:29:53.760460 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:53.760202    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e4cabfc1-9cab-4eb9-8e32-7dd66c04d572-sys\") pod \"node-exporter-chc8q\" (UID: \"e4cabfc1-9cab-4eb9-8e32-7dd66c04d572\") " pod="openshift-monitoring/node-exporter-chc8q"
Jul 27 17:29:53.760460 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:53.760312    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61385c00-c295-4f64-a5c8-d5f0dd2ce6af-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-rxfs4\" (UID: \"61385c00-c295-4f64-a5c8-d5f0dd2ce6af\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rxfs4"
Jul 27 17:29:53.760460 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:53.760318    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e4cabfc1-9cab-4eb9-8e32-7dd66c04d572-root\") pod \"node-exporter-chc8q\" (UID: \"e4cabfc1-9cab-4eb9-8e32-7dd66c04d572\") " pod="openshift-monitoring/node-exporter-chc8q"
Jul 27 17:29:53.760460 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:53.760338    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e4cabfc1-9cab-4eb9-8e32-7dd66c04d572-node-exporter-wtmp\") pod \"node-exporter-chc8q\" (UID: \"e4cabfc1-9cab-4eb9-8e32-7dd66c04d572\") " pod="openshift-monitoring/node-exporter-chc8q"
Jul 27 17:29:53.760460 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:53.760349    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61385c00-c295-4f64-a5c8-d5f0dd2ce6af-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-rxfs4\" (UID: \"61385c00-c295-4f64-a5c8-d5f0dd2ce6af\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rxfs4"
Jul 27 17:29:53.760460 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:53.760390    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e4cabfc1-9cab-4eb9-8e32-7dd66c04d572-sys\") pod \"node-exporter-chc8q\" (UID: \"e4cabfc1-9cab-4eb9-8e32-7dd66c04d572\") " pod="openshift-monitoring/node-exporter-chc8q"
Jul 27 17:29:53.760460 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:53.760429    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4cabfc1-9cab-4eb9-8e32-7dd66c04d572-metrics-client-ca\") pod \"node-exporter-chc8q\" (UID: \"e4cabfc1-9cab-4eb9-8e32-7dd66c04d572\") " pod="openshift-monitoring/node-exporter-chc8q"
Jul 27 17:29:53.760460 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:53.760464    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e4cabfc1-9cab-4eb9-8e32-7dd66c04d572-node-exporter-textfile\") pod \"node-exporter-chc8q\" (UID: \"e4cabfc1-9cab-4eb9-8e32-7dd66c04d572\") " pod="openshift-monitoring/node-exporter-chc8q"
Jul 27 17:29:53.760931 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:53.760739    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e4cabfc1-9cab-4eb9-8e32-7dd66c04d572-node-exporter-accelerators-collector-config\") pod \"node-exporter-chc8q\" (UID: \"e4cabfc1-9cab-4eb9-8e32-7dd66c04d572\") " pod="openshift-monitoring/node-exporter-chc8q"
Jul 27 17:29:53.760931 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:53.760813    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e4cabfc1-9cab-4eb9-8e32-7dd66c04d572-node-exporter-textfile\") pod \"node-exporter-chc8q\" (UID: \"e4cabfc1-9cab-4eb9-8e32-7dd66c04d572\") " pod="openshift-monitoring/node-exporter-chc8q"
Jul 27 17:29:53.761060 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:53.761038    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4cabfc1-9cab-4eb9-8e32-7dd66c04d572-metrics-client-ca\") pod \"node-exporter-chc8q\" (UID: \"e4cabfc1-9cab-4eb9-8e32-7dd66c04d572\") " pod="openshift-monitoring/node-exporter-chc8q"
Jul 27 17:29:53.761152 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:53.761131    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61385c00-c295-4f64-a5c8-d5f0dd2ce6af-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-rxfs4\" (UID: \"61385c00-c295-4f64-a5c8-d5f0dd2ce6af\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rxfs4"
Jul 27 17:29:53.762574 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:53.762547    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/61385c00-c295-4f64-a5c8-d5f0dd2ce6af-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-rxfs4\" (UID: \"61385c00-c295-4f64-a5c8-d5f0dd2ce6af\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rxfs4"
Jul 27 17:29:53.762822 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:53.762795    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e4cabfc1-9cab-4eb9-8e32-7dd66c04d572-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-chc8q\" (UID: \"e4cabfc1-9cab-4eb9-8e32-7dd66c04d572\") " pod="openshift-monitoring/node-exporter-chc8q"
Jul 27 17:29:53.763092 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:53.763072    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61385c00-c295-4f64-a5c8-d5f0dd2ce6af-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-rxfs4\" (UID: \"61385c00-c295-4f64-a5c8-d5f0dd2ce6af\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rxfs4"
Jul 27 17:29:53.773282 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:53.773239    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58zl8\" (UniqueName: \"kubernetes.io/projected/e4cabfc1-9cab-4eb9-8e32-7dd66c04d572-kube-api-access-58zl8\") pod \"node-exporter-chc8q\" (UID: \"e4cabfc1-9cab-4eb9-8e32-7dd66c04d572\") " pod="openshift-monitoring/node-exporter-chc8q"
Jul 27 17:29:53.773714 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:53.773699    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x9n6\" (UniqueName: \"kubernetes.io/projected/61385c00-c295-4f64-a5c8-d5f0dd2ce6af-kube-api-access-5x9n6\") pod \"openshift-state-metrics-b8f49f5c6-rxfs4\" (UID: \"61385c00-c295-4f64-a5c8-d5f0dd2ce6af\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rxfs4"
Jul 27 17:29:53.841442 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:53.841364    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rxfs4"
Jul 27 17:29:53.851060 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:53.851022    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:53.951468 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:53.951430    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:53.976031 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:53.975997    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-rxfs4"]
Jul 27 17:29:53.979489 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:29:53.979460    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61385c00_c295_4f64_a5c8_d5f0dd2ce6af.slice/crio-cac1d06179f76d0a30311c3787fe9a575f0d7f8f388138ba9fcf59a9cdc76856 WatchSource:0}: Error finding container cac1d06179f76d0a30311c3787fe9a575f0d7f8f388138ba9fcf59a9cdc76856: Status 404 returned error can't find the container with id cac1d06179f76d0a30311c3787fe9a575f0d7f8f388138ba9fcf59a9cdc76856
Jul 27 17:29:54.052934 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.052909    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:54.151413 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.151377    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:54.264517 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.264482    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e4cabfc1-9cab-4eb9-8e32-7dd66c04d572-node-exporter-tls\") pod \"node-exporter-chc8q\" (UID: \"e4cabfc1-9cab-4eb9-8e32-7dd66c04d572\") " pod="openshift-monitoring/node-exporter-chc8q"
Jul 27 17:29:54.266700 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.266676    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e4cabfc1-9cab-4eb9-8e32-7dd66c04d572-node-exporter-tls\") pod \"node-exporter-chc8q\" (UID: \"e4cabfc1-9cab-4eb9-8e32-7dd66c04d572\") " pod="openshift-monitoring/node-exporter-chc8q"
Jul 27 17:29:54.447212 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.447180    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-chc8q"
Jul 27 17:29:54.459199 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:29:54.459160    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4cabfc1_9cab_4eb9_8e32_7dd66c04d572.slice/crio-16af1191381c31d41c5dc824f74d1f50e01007bdb1cb5590780c121735a9aa0e WatchSource:0}: Error finding container 16af1191381c31d41c5dc824f74d1f50e01007bdb1cb5590780c121735a9aa0e: Status 404 returned error can't find the container with id 16af1191381c31d41c5dc824f74d1f50e01007bdb1cb5590780c121735a9aa0e
Jul 27 17:29:54.492326 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.492296    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:54.573568 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.573518    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:54.579115 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.579059    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 17:29:54.583341 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.583302    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:29:54.586672 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.586647    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 27 17:29:54.586808 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.586657    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 27 17:29:54.586863 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.586822    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 27 17:29:54.586916 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.586892    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 27 17:29:54.587538 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.587081    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 27 17:29:54.587538 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.587109    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-cpvdx\""
Jul 27 17:29:54.587538 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.587129    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 27 17:29:54.587538 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.587130    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 27 17:29:54.587538 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.587349    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 27 17:29:54.587538 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.587437    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 27 17:29:54.604347 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.603586    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 17:29:54.668151 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.668116    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a0874ece-9e32-43fd-b926-98c8a3d7b44c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:29:54.668363 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.668161    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a0874ece-9e32-43fd-b926-98c8a3d7b44c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:29:54.668363 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.668191    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a0874ece-9e32-43fd-b926-98c8a3d7b44c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:29:54.668363 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.668225    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0874ece-9e32-43fd-b926-98c8a3d7b44c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:29:54.668363 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.668274    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a0874ece-9e32-43fd-b926-98c8a3d7b44c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:29:54.668363 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.668313    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a0874ece-9e32-43fd-b926-98c8a3d7b44c-config-out\") pod \"alertmanager-main-0\" (UID: \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:29:54.668363 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.668348    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0874ece-9e32-43fd-b926-98c8a3d7b44c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:29:54.668671 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.668382    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a0874ece-9e32-43fd-b926-98c8a3d7b44c-config-volume\") pod \"alertmanager-main-0\" (UID: \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:29:54.668671 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.668405    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a0874ece-9e32-43fd-b926-98c8a3d7b44c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:29:54.668671 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.668452    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a0874ece-9e32-43fd-b926-98c8a3d7b44c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:29:54.668671 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.668469    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a0874ece-9e32-43fd-b926-98c8a3d7b44c-web-config\") pod \"alertmanager-main-0\" (UID: \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:29:54.668671 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.668494    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a0874ece-9e32-43fd-b926-98c8a3d7b44c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:29:54.668671 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.668532    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzjzb\" (UniqueName: \"kubernetes.io/projected/a0874ece-9e32-43fd-b926-98c8a3d7b44c-kube-api-access-kzjzb\") pod \"alertmanager-main-0\" (UID: \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:29:54.769670 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.769585    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a0874ece-9e32-43fd-b926-98c8a3d7b44c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:29:54.769670 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.769633    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a0874ece-9e32-43fd-b926-98c8a3d7b44c-web-config\") pod \"alertmanager-main-0\" (UID: \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:29:54.769670 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.769667    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a0874ece-9e32-43fd-b926-98c8a3d7b44c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:29:54.771463 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.769696    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzjzb\" (UniqueName: \"kubernetes.io/projected/a0874ece-9e32-43fd-b926-98c8a3d7b44c-kube-api-access-kzjzb\") pod \"alertmanager-main-0\" (UID: \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:29:54.771463 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.769736    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a0874ece-9e32-43fd-b926-98c8a3d7b44c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:29:54.771463 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.769769    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a0874ece-9e32-43fd-b926-98c8a3d7b44c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:29:54.771463 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.769796    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a0874ece-9e32-43fd-b926-98c8a3d7b44c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:29:54.771463 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.769823    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0874ece-9e32-43fd-b926-98c8a3d7b44c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:29:54.771463 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.769863    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a0874ece-9e32-43fd-b926-98c8a3d7b44c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:29:54.771463 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.769905    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a0874ece-9e32-43fd-b926-98c8a3d7b44c-config-out\") pod \"alertmanager-main-0\" (UID: \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:29:54.771463 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.769928    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0874ece-9e32-43fd-b926-98c8a3d7b44c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:29:54.771463 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.769962    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a0874ece-9e32-43fd-b926-98c8a3d7b44c-config-volume\") pod \"alertmanager-main-0\" (UID: \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:29:54.771463 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.769993    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a0874ece-9e32-43fd-b926-98c8a3d7b44c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:29:54.771463 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.770847    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a0874ece-9e32-43fd-b926-98c8a3d7b44c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:29:54.772019 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.771926    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0874ece-9e32-43fd-b926-98c8a3d7b44c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:29:54.772863 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.772803    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a0874ece-9e32-43fd-b926-98c8a3d7b44c-config-out\") pod \"alertmanager-main-0\" (UID: \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:29:54.773344 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.773318    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a0874ece-9e32-43fd-b926-98c8a3d7b44c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:29:54.774793 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.774747    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0874ece-9e32-43fd-b926-98c8a3d7b44c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:29:54.775593 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.775552    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a0874ece-9e32-43fd-b926-98c8a3d7b44c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:29:54.775715 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.775693    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a0874ece-9e32-43fd-b926-98c8a3d7b44c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:29:54.775835 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.775814    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a0874ece-9e32-43fd-b926-98c8a3d7b44c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:29:54.775965 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.775949    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a0874ece-9e32-43fd-b926-98c8a3d7b44c-config-volume\") pod \"alertmanager-main-0\" (UID: \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:29:54.776106 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.776085    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a0874ece-9e32-43fd-b926-98c8a3d7b44c-web-config\") pod \"alertmanager-main-0\" (UID: \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:29:54.776185 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.776167    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a0874ece-9e32-43fd-b926-98c8a3d7b44c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:29:54.778291 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.777590    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a0874ece-9e32-43fd-b926-98c8a3d7b44c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:29:54.788808 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.788718    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzjzb\" (UniqueName: \"kubernetes.io/projected/a0874ece-9e32-43fd-b926-98c8a3d7b44c-kube-api-access-kzjzb\") pod \"alertmanager-main-0\" (UID: \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:29:54.833792 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.833758    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:54.902559 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.902526    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:29:54.953580 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.953536    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-chc8q" event={"ID":"e4cabfc1-9cab-4eb9-8e32-7dd66c04d572","Type":"ContainerStarted","Data":"16af1191381c31d41c5dc824f74d1f50e01007bdb1cb5590780c121735a9aa0e"}
Jul 27 17:29:54.955543 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.955516    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rxfs4" event={"ID":"61385c00-c295-4f64-a5c8-d5f0dd2ce6af","Type":"ContainerStarted","Data":"192a7c6dd1eb64223c3977b9a2efa2b9382a84ddd5a7a95e5b4635ea4380005e"}
Jul 27 17:29:54.955676 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.955572    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rxfs4" event={"ID":"61385c00-c295-4f64-a5c8-d5f0dd2ce6af","Type":"ContainerStarted","Data":"0e13c1e602a58c5c5a6cfe3b052806816851b1fa4b9952626d8f4fbef5e2ab93"}
Jul 27 17:29:54.955676 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:54.955601    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rxfs4" event={"ID":"61385c00-c295-4f64-a5c8-d5f0dd2ce6af","Type":"ContainerStarted","Data":"cac1d06179f76d0a30311c3787fe9a575f0d7f8f388138ba9fcf59a9cdc76856"}
Jul 27 17:29:55.154912 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:55.154723    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 17:29:55.157190 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:29:55.157152    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0874ece_9e32_43fd_b926_98c8a3d7b44c.slice/crio-cf0510b1d73b2ce356c940a6398141d33f2630139ab330bdcfa4fe80ab698483 WatchSource:0}: Error finding container cf0510b1d73b2ce356c940a6398141d33f2630139ab330bdcfa4fe80ab698483: Status 404 returned error can't find the container with id cf0510b1d73b2ce356c940a6398141d33f2630139ab330bdcfa4fe80ab698483
Jul 27 17:29:55.961376 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:55.961339    2573 generic.go:356] "Generic (PLEG): container finished" podID="e4cabfc1-9cab-4eb9-8e32-7dd66c04d572" containerID="fa8876ab031504ad0685858a4a443d7bc80a0f491c550d308ff5e55fcdbabe97" exitCode=0
Jul 27 17:29:55.961811 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:55.961426    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-chc8q" event={"ID":"e4cabfc1-9cab-4eb9-8e32-7dd66c04d572","Type":"ContainerDied","Data":"fa8876ab031504ad0685858a4a443d7bc80a0f491c550d308ff5e55fcdbabe97"}
Jul 27 17:29:55.964047 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:55.964021    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rxfs4" event={"ID":"61385c00-c295-4f64-a5c8-d5f0dd2ce6af","Type":"ContainerStarted","Data":"5a683d4de133e7b525e1b7793e821deec5276cfdfd7281d97136bb020f590132"}
Jul 27 17:29:55.965348 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:55.965322    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0874ece-9e32-43fd-b926-98c8a3d7b44c","Type":"ContainerStarted","Data":"cf0510b1d73b2ce356c940a6398141d33f2630139ab330bdcfa4fe80ab698483"}
Jul 27 17:29:56.006832 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:56.006774    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rxfs4" podStartSLOduration=2.055726038 podStartE2EDuration="3.006751651s" totalImagesPullingTime="951.025613ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:29:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:56.005764158 +0000 UTC m=+138.149166210" watchObservedRunningTime="2026-07-27 17:29:56.006751651 +0000 UTC m=+138.150153706"
Jul 27 17:29:56.206244 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:56.206203    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:56.971634 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:56.971601    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-chc8q" event={"ID":"e4cabfc1-9cab-4eb9-8e32-7dd66c04d572","Type":"ContainerStarted","Data":"871e6192d922eae13316f2acbc34aca63b467c0888f142d62f67f74cc50ba473"}
Jul 27 17:29:56.972008 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:56.971661    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-chc8q" event={"ID":"e4cabfc1-9cab-4eb9-8e32-7dd66c04d572","Type":"ContainerStarted","Data":"aefd83edc2c377ad2b4b10eb802ffbd1c87cf7b64359ccaff9772f2b5f2de941"}
Jul 27 17:29:56.973062 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:56.973039    2573 generic.go:356] "Generic (PLEG): container finished" podID="a0874ece-9e32-43fd-b926-98c8a3d7b44c" containerID="112d8c4407d93d8ee91c65179d949f8310702be16c0266429b9630f39097fcee" exitCode=0
Jul 27 17:29:56.973178 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:56.973109    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0874ece-9e32-43fd-b926-98c8a3d7b44c","Type":"ContainerDied","Data":"112d8c4407d93d8ee91c65179d949f8310702be16c0266429b9630f39097fcee"}
Jul 27 17:29:56.999608 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:56.999561    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-chc8q" podStartSLOduration=2.825677247 podStartE2EDuration="3.999546093s" totalImagesPullingTime="1.104985855s" totalInitContainerRuntime="68.882991ms" isStatefulPod=true podCreationTimestamp="2026-07-27 17:29:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:56.997414882 +0000 UTC m=+139.140816934" watchObservedRunningTime="2026-07-27 17:29:56.999546093 +0000 UTC m=+139.142948143"
Jul 27 17:29:57.958320 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:57.958276    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:58.061677 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:58.061644    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:58.162116 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:58.162056    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:58.270270 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:58.270118    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:58.286347 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:58.286314    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-pwwlv"]
Jul 27 17:29:58.296293 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:58.296241    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-pwwlv"
Jul 27 17:29:58.300614 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:58.300550    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 27 17:29:58.300801 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:58.300615    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-gs4xv\""
Jul 27 17:29:58.302604 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:58.302546    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-pwwlv"]
Jul 27 17:29:58.344651 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:58.344608    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:29:58.400613 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:58.400576    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d89a1662-c5db-4f1a-8466-2006b605667b-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-pwwlv\" (UID: \"d89a1662-c5db-4f1a-8466-2006b605667b\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-pwwlv"
Jul 27 17:29:58.400795 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:58.400698    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zg8l\" (UniqueName: \"kubernetes.io/projected/d89a1662-c5db-4f1a-8466-2006b605667b-kube-api-access-7zg8l\") pod \"monitoring-plugin-68b7fc9574-pwwlv\" (UID: \"d89a1662-c5db-4f1a-8466-2006b605667b\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-pwwlv"
Jul 27 17:29:58.501693 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:58.501652    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zg8l\" (UniqueName: \"kubernetes.io/projected/d89a1662-c5db-4f1a-8466-2006b605667b-kube-api-access-7zg8l\") pod \"monitoring-plugin-68b7fc9574-pwwlv\" (UID: \"d89a1662-c5db-4f1a-8466-2006b605667b\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-pwwlv"
Jul 27 17:29:58.501892 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:58.501714    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d89a1662-c5db-4f1a-8466-2006b605667b-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-pwwlv\" (UID: \"d89a1662-c5db-4f1a-8466-2006b605667b\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-pwwlv"
Jul 27 17:29:58.501968 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:29:58.501893    2573 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 27 17:29:58.501968 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:29:58.501964    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d89a1662-c5db-4f1a-8466-2006b605667b-monitoring-plugin-cert podName:d89a1662-c5db-4f1a-8466-2006b605667b nodeName:}" failed. No retries permitted until 2026-07-27 17:29:59.00194116 +0000 UTC m=+141.145343202 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/d89a1662-c5db-4f1a-8466-2006b605667b-monitoring-plugin-cert") pod "monitoring-plugin-68b7fc9574-pwwlv" (UID: "d89a1662-c5db-4f1a-8466-2006b605667b") : secret "monitoring-plugin-cert" not found
Jul 27 17:29:58.515285 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:58.515240    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zg8l\" (UniqueName: \"kubernetes.io/projected/d89a1662-c5db-4f1a-8466-2006b605667b-kube-api-access-7zg8l\") pod \"monitoring-plugin-68b7fc9574-pwwlv\" (UID: \"d89a1662-c5db-4f1a-8466-2006b605667b\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-pwwlv"
Jul 27 17:29:58.982423 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:58.982395    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0874ece-9e32-43fd-b926-98c8a3d7b44c","Type":"ContainerStarted","Data":"e7b3c3f6eb76a1550f5bd8badd477b872f565226dde98b845f3bc632692ce0c5"}
Jul 27 17:29:59.005565 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:59.005539    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d89a1662-c5db-4f1a-8466-2006b605667b-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-pwwlv\" (UID: \"d89a1662-c5db-4f1a-8466-2006b605667b\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-pwwlv"
Jul 27 17:29:59.007816 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:59.007796    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d89a1662-c5db-4f1a-8466-2006b605667b-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-pwwlv\" (UID: \"d89a1662-c5db-4f1a-8466-2006b605667b\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-pwwlv"
Jul 27 17:29:59.208036 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:59.208012    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-pwwlv"
Jul 27 17:29:59.340830 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:59.340794    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-pwwlv"]
Jul 27 17:29:59.340968 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:59.340839    2573 patch_prober.go:28] interesting pod/image-registry-6db8876459-vdgh6 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 27 17:29:59.340968 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:59.340879    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6db8876459-vdgh6" podUID="6f2becbf-a41c-4215-9435-9f735c60822e" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 17:29:59.343745 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:29:59.343724    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd89a1662_c5db_4f1a_8466_2006b605667b.slice/crio-9b99e808a7ad8084483ad0bccce90705b9f91004c623dfe0477253bb11873148 WatchSource:0}: Error finding container 9b99e808a7ad8084483ad0bccce90705b9f91004c623dfe0477253bb11873148: Status 404 returned error can't find the container with id 9b99e808a7ad8084483ad0bccce90705b9f91004c623dfe0477253bb11873148
Jul 27 17:29:59.688048 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:59.688017    2573 patch_prober.go:28] interesting pod/image-registry-74648bb95f-zz4vg container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 27 17:29:59.688237 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:59.688075    2573 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-74648bb95f-zz4vg" podUID="0e5f0900-1fec-4aa8-9372-45cb418b7bdd" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 17:29:59.990109 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:59.990070    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0874ece-9e32-43fd-b926-98c8a3d7b44c","Type":"ContainerStarted","Data":"b33e182091f1d2f0dadf726285b173356e2b6f3b4783ed9016fa02ca5987fd19"}
Jul 27 17:29:59.990316 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:59.990148    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0874ece-9e32-43fd-b926-98c8a3d7b44c","Type":"ContainerStarted","Data":"ffef14148efe02ae50befbc26b89bcb4b0ae615ad54a48621ddb1d6c880d05bf"}
Jul 27 17:29:59.990316 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:59.990188    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0874ece-9e32-43fd-b926-98c8a3d7b44c","Type":"ContainerStarted","Data":"be3d640dce34f19bc0d947c76c71272dd7849b55fc09e438548bf2f7768353d0"}
Jul 27 17:29:59.990316 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:59.990222    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0874ece-9e32-43fd-b926-98c8a3d7b44c","Type":"ContainerStarted","Data":"d18a3bf271c93c98660afea784f45b2b8941b98cddd408f9efddb4bf9c212d78"}
Jul 27 17:29:59.990316 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:59.990274    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0874ece-9e32-43fd-b926-98c8a3d7b44c","Type":"ContainerStarted","Data":"14e579c9523eba2d8aa3310df56d6d1096909a52c61416b6ef1499e49c827a03"}
Jul 27 17:29:59.991339 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:29:59.991306    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-pwwlv" event={"ID":"d89a1662-c5db-4f1a-8466-2006b605667b","Type":"ContainerStarted","Data":"9b99e808a7ad8084483ad0bccce90705b9f91004c623dfe0477253bb11873148"}
Jul 27 17:30:00.026067 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:00.026003    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.368392118 podStartE2EDuration="6.0259812s" totalImagesPullingTime="3.64204916s" totalInitContainerRuntime="15.539922ms" isStatefulPod=false podCreationTimestamp="2026-07-27 17:29:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:30:00.022976895 +0000 UTC m=+142.166378970" watchObservedRunningTime="2026-07-27 17:30:00.0259812 +0000 UTC m=+142.169383251"
Jul 27 17:30:00.782517 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:00.782482    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:30:00.879385 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:00.879360    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:30:00.996816 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:00.996783    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-pwwlv" event={"ID":"d89a1662-c5db-4f1a-8466-2006b605667b","Type":"ContainerStarted","Data":"63f6871838042e548323d39e5fc1792d34142f28cc627d33700bd6b02a94c5f8"}
Jul 27 17:30:00.997135 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:00.997118    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-pwwlv"
Jul 27 17:30:01.001666 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:01.001638    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-pwwlv"
Jul 27 17:30:01.014966 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:01.014918    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-pwwlv" podStartSLOduration=1.523362774 podStartE2EDuration="3.01490217s" totalImagesPullingTime="1.491539396s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:29:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:30:01.014265559 +0000 UTC m=+143.157667610" watchObservedRunningTime="2026-07-27 17:30:01.01490217 +0000 UTC m=+143.158304220"
Jul 27 17:30:01.900233 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:01.900194    2573 patch_prober.go:28] interesting pod/image-registry-74648bb95f-zz4vg container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 27 17:30:01.900634 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:01.900270    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-74648bb95f-zz4vg" podUID="0e5f0900-1fec-4aa8-9372-45cb418b7bdd" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 17:30:04.367162 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:04.367113    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6db8876459-vdgh6" podUID="6f2becbf-a41c-4215-9435-9f735c60822e" containerName="registry" containerID="cri-o://12119fa4a86bbac817678b8c8914b14f52a77a8593ebe191c1213155c79dee62" gracePeriod=30
Jul 27 17:30:04.630592 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:04.630557    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6db8876459-vdgh6"
Jul 27 17:30:04.652523 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:04.652462    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6f2becbf-a41c-4215-9435-9f735c60822e-ca-trust-extracted\") pod \"6f2becbf-a41c-4215-9435-9f735c60822e\" (UID: \"6f2becbf-a41c-4215-9435-9f735c60822e\") "
Jul 27 17:30:04.652523 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:04.652508    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zptp5\" (UniqueName: \"kubernetes.io/projected/6f2becbf-a41c-4215-9435-9f735c60822e-kube-api-access-zptp5\") pod \"6f2becbf-a41c-4215-9435-9f735c60822e\" (UID: \"6f2becbf-a41c-4215-9435-9f735c60822e\") "
Jul 27 17:30:04.652789 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:04.652533    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6f2becbf-a41c-4215-9435-9f735c60822e-registry-certificates\") pod \"6f2becbf-a41c-4215-9435-9f735c60822e\" (UID: \"6f2becbf-a41c-4215-9435-9f735c60822e\") "
Jul 27 17:30:04.652789 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:04.652560    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6f2becbf-a41c-4215-9435-9f735c60822e-trusted-ca\") pod \"6f2becbf-a41c-4215-9435-9f735c60822e\" (UID: \"6f2becbf-a41c-4215-9435-9f735c60822e\") "
Jul 27 17:30:04.652789 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:04.652589    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6f2becbf-a41c-4215-9435-9f735c60822e-installation-pull-secrets\") pod \"6f2becbf-a41c-4215-9435-9f735c60822e\" (UID: \"6f2becbf-a41c-4215-9435-9f735c60822e\") "
Jul 27 17:30:04.652789 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:04.652620    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6f2becbf-a41c-4215-9435-9f735c60822e-image-registry-private-configuration\") pod \"6f2becbf-a41c-4215-9435-9f735c60822e\" (UID: \"6f2becbf-a41c-4215-9435-9f735c60822e\") "
Jul 27 17:30:04.652789 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:04.652679    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6f2becbf-a41c-4215-9435-9f735c60822e-bound-sa-token\") pod \"6f2becbf-a41c-4215-9435-9f735c60822e\" (UID: \"6f2becbf-a41c-4215-9435-9f735c60822e\") "
Jul 27 17:30:04.652789 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:04.652711    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6f2becbf-a41c-4215-9435-9f735c60822e-registry-tls\") pod \"6f2becbf-a41c-4215-9435-9f735c60822e\" (UID: \"6f2becbf-a41c-4215-9435-9f735c60822e\") "
Jul 27 17:30:04.653076 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:04.653037    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f2becbf-a41c-4215-9435-9f735c60822e-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "6f2becbf-a41c-4215-9435-9f735c60822e" (UID: "6f2becbf-a41c-4215-9435-9f735c60822e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:30:04.653126 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:04.653069    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f2becbf-a41c-4215-9435-9f735c60822e-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "6f2becbf-a41c-4215-9435-9f735c60822e" (UID: "6f2becbf-a41c-4215-9435-9f735c60822e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:30:04.655721 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:04.655640    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f2becbf-a41c-4215-9435-9f735c60822e-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "6f2becbf-a41c-4215-9435-9f735c60822e" (UID: "6f2becbf-a41c-4215-9435-9f735c60822e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:30:04.655721 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:04.655659    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f2becbf-a41c-4215-9435-9f735c60822e-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "6f2becbf-a41c-4215-9435-9f735c60822e" (UID: "6f2becbf-a41c-4215-9435-9f735c60822e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:30:04.656049 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:04.656020    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f2becbf-a41c-4215-9435-9f735c60822e-kube-api-access-zptp5" (OuterVolumeSpecName: "kube-api-access-zptp5") pod "6f2becbf-a41c-4215-9435-9f735c60822e" (UID: "6f2becbf-a41c-4215-9435-9f735c60822e"). InnerVolumeSpecName "kube-api-access-zptp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:30:04.656270 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:04.656024    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f2becbf-a41c-4215-9435-9f735c60822e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "6f2becbf-a41c-4215-9435-9f735c60822e" (UID: "6f2becbf-a41c-4215-9435-9f735c60822e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:30:04.656270 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:04.656050    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f2becbf-a41c-4215-9435-9f735c60822e-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "6f2becbf-a41c-4215-9435-9f735c60822e" (UID: "6f2becbf-a41c-4215-9435-9f735c60822e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:30:04.668061 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:04.668020    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f2becbf-a41c-4215-9435-9f735c60822e-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "6f2becbf-a41c-4215-9435-9f735c60822e" (UID: "6f2becbf-a41c-4215-9435-9f735c60822e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:30:04.754458 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:04.754412    2573 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6f2becbf-a41c-4215-9435-9f735c60822e-ca-trust-extracted\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:30:04.754458 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:04.754456    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zptp5\" (UniqueName: \"kubernetes.io/projected/6f2becbf-a41c-4215-9435-9f735c60822e-kube-api-access-zptp5\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:30:04.754664 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:04.754473    2573 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6f2becbf-a41c-4215-9435-9f735c60822e-registry-certificates\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:30:04.754664 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:04.754489    2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6f2becbf-a41c-4215-9435-9f735c60822e-trusted-ca\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:30:04.754664 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:04.754503    2573 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6f2becbf-a41c-4215-9435-9f735c60822e-installation-pull-secrets\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:30:04.754664 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:04.754518    2573 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6f2becbf-a41c-4215-9435-9f735c60822e-image-registry-private-configuration\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:30:04.754664 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:04.754532    2573 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6f2becbf-a41c-4215-9435-9f735c60822e-bound-sa-token\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:30:04.754664 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:04.754545    2573 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6f2becbf-a41c-4215-9435-9f735c60822e-registry-tls\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:30:05.012765 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:05.012728    2573 generic.go:356] "Generic (PLEG): container finished" podID="6f2becbf-a41c-4215-9435-9f735c60822e" containerID="12119fa4a86bbac817678b8c8914b14f52a77a8593ebe191c1213155c79dee62" exitCode=0
Jul 27 17:30:05.012936 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:05.012780    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6db8876459-vdgh6"
Jul 27 17:30:05.012936 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:05.012828    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6db8876459-vdgh6" event={"ID":"6f2becbf-a41c-4215-9435-9f735c60822e","Type":"ContainerDied","Data":"12119fa4a86bbac817678b8c8914b14f52a77a8593ebe191c1213155c79dee62"}
Jul 27 17:30:05.012936 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:05.012883    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6db8876459-vdgh6" event={"ID":"6f2becbf-a41c-4215-9435-9f735c60822e","Type":"ContainerDied","Data":"633b71016060ff13344a7000dacb6a1558b5d85ce105c990c18b19eb558466fe"}
Jul 27 17:30:05.012936 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:05.012910    2573 scope.go:117] "RemoveContainer" containerID="12119fa4a86bbac817678b8c8914b14f52a77a8593ebe191c1213155c79dee62"
Jul 27 17:30:05.021735 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:05.021716    2573 scope.go:117] "RemoveContainer" containerID="12119fa4a86bbac817678b8c8914b14f52a77a8593ebe191c1213155c79dee62"
Jul 27 17:30:05.021992 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:30:05.021973    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12119fa4a86bbac817678b8c8914b14f52a77a8593ebe191c1213155c79dee62\": container with ID starting with 12119fa4a86bbac817678b8c8914b14f52a77a8593ebe191c1213155c79dee62 not found: ID does not exist" containerID="12119fa4a86bbac817678b8c8914b14f52a77a8593ebe191c1213155c79dee62"
Jul 27 17:30:05.022038 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:05.022004    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12119fa4a86bbac817678b8c8914b14f52a77a8593ebe191c1213155c79dee62"} err="failed to get container status \"12119fa4a86bbac817678b8c8914b14f52a77a8593ebe191c1213155c79dee62\": rpc error: code = NotFound desc = could not find container \"12119fa4a86bbac817678b8c8914b14f52a77a8593ebe191c1213155c79dee62\": container with ID starting with 12119fa4a86bbac817678b8c8914b14f52a77a8593ebe191c1213155c79dee62 not found: ID does not exist"
Jul 27 17:30:05.039666 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:05.039637    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6db8876459-vdgh6"]
Jul 27 17:30:05.041823 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:05.041800    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6db8876459-vdgh6"]
Jul 27 17:30:05.577911 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:05.577882    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:30:05.638550 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:05.638517    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:30:05.982412 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:05.982382    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-59d7d5f8d8-vxsmf"]
Jul 27 17:30:05.982694 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:05.982682    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f2becbf-a41c-4215-9435-9f735c60822e" containerName="registry"
Jul 27 17:30:05.982741 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:05.982695    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f2becbf-a41c-4215-9435-9f735c60822e" containerName="registry"
Jul 27 17:30:05.982773 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:05.982759    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f2becbf-a41c-4215-9435-9f735c60822e" containerName="registry"
Jul 27 17:30:05.986227 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:05.986207    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59d7d5f8d8-vxsmf"
Jul 27 17:30:05.989420 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:05.989396    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 27 17:30:05.989538 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:05.989515    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-d8nfx\""
Jul 27 17:30:05.991160 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:05.991102    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 27 17:30:05.991240 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:05.991174    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 27 17:30:05.991240 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:05.991109    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 27 17:30:05.991240 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:05.991216    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 27 17:30:05.991568 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:05.991551    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 27 17:30:05.991697 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:05.991632    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 27 17:30:05.996398 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:05.996373    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 27 17:30:05.997945 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:05.997927    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59d7d5f8d8-vxsmf"]
Jul 27 17:30:06.066935 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:06.066905    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-894dr\" (UniqueName: \"kubernetes.io/projected/ce271c8e-e982-4e3e-8588-26b79ccc6488-kube-api-access-894dr\") pod \"console-59d7d5f8d8-vxsmf\" (UID: \"ce271c8e-e982-4e3e-8588-26b79ccc6488\") " pod="openshift-console/console-59d7d5f8d8-vxsmf"
Jul 27 17:30:06.066935 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:06.066938    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ce271c8e-e982-4e3e-8588-26b79ccc6488-service-ca\") pod \"console-59d7d5f8d8-vxsmf\" (UID: \"ce271c8e-e982-4e3e-8588-26b79ccc6488\") " pod="openshift-console/console-59d7d5f8d8-vxsmf"
Jul 27 17:30:06.067132 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:06.066973    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce271c8e-e982-4e3e-8588-26b79ccc6488-console-serving-cert\") pod \"console-59d7d5f8d8-vxsmf\" (UID: \"ce271c8e-e982-4e3e-8588-26b79ccc6488\") " pod="openshift-console/console-59d7d5f8d8-vxsmf"
Jul 27 17:30:06.067132 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:06.066994    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ce271c8e-e982-4e3e-8588-26b79ccc6488-oauth-serving-cert\") pod \"console-59d7d5f8d8-vxsmf\" (UID: \"ce271c8e-e982-4e3e-8588-26b79ccc6488\") " pod="openshift-console/console-59d7d5f8d8-vxsmf"
Jul 27 17:30:06.067132 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:06.067039    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ce271c8e-e982-4e3e-8588-26b79ccc6488-console-config\") pod \"console-59d7d5f8d8-vxsmf\" (UID: \"ce271c8e-e982-4e3e-8588-26b79ccc6488\") " pod="openshift-console/console-59d7d5f8d8-vxsmf"
Jul 27 17:30:06.067132 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:06.067060    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ce271c8e-e982-4e3e-8588-26b79ccc6488-console-oauth-config\") pod \"console-59d7d5f8d8-vxsmf\" (UID: \"ce271c8e-e982-4e3e-8588-26b79ccc6488\") " pod="openshift-console/console-59d7d5f8d8-vxsmf"
Jul 27 17:30:06.067132 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:06.067079    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce271c8e-e982-4e3e-8588-26b79ccc6488-trusted-ca-bundle\") pod \"console-59d7d5f8d8-vxsmf\" (UID: \"ce271c8e-e982-4e3e-8588-26b79ccc6488\") " pod="openshift-console/console-59d7d5f8d8-vxsmf"
Jul 27 17:30:06.168296 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:06.168221    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce271c8e-e982-4e3e-8588-26b79ccc6488-console-serving-cert\") pod \"console-59d7d5f8d8-vxsmf\" (UID: \"ce271c8e-e982-4e3e-8588-26b79ccc6488\") " pod="openshift-console/console-59d7d5f8d8-vxsmf"
Jul 27 17:30:06.168434 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:06.168315    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ce271c8e-e982-4e3e-8588-26b79ccc6488-oauth-serving-cert\") pod \"console-59d7d5f8d8-vxsmf\" (UID: \"ce271c8e-e982-4e3e-8588-26b79ccc6488\") " pod="openshift-console/console-59d7d5f8d8-vxsmf"
Jul 27 17:30:06.168434 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:06.168350    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ce271c8e-e982-4e3e-8588-26b79ccc6488-console-config\") pod \"console-59d7d5f8d8-vxsmf\" (UID: \"ce271c8e-e982-4e3e-8588-26b79ccc6488\") " pod="openshift-console/console-59d7d5f8d8-vxsmf"
Jul 27 17:30:06.168434 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:06.168385    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ce271c8e-e982-4e3e-8588-26b79ccc6488-console-oauth-config\") pod \"console-59d7d5f8d8-vxsmf\" (UID: \"ce271c8e-e982-4e3e-8588-26b79ccc6488\") " pod="openshift-console/console-59d7d5f8d8-vxsmf"
Jul 27 17:30:06.168434 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:06.168421    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce271c8e-e982-4e3e-8588-26b79ccc6488-trusted-ca-bundle\") pod \"console-59d7d5f8d8-vxsmf\" (UID: \"ce271c8e-e982-4e3e-8588-26b79ccc6488\") " pod="openshift-console/console-59d7d5f8d8-vxsmf"
Jul 27 17:30:06.168594 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:06.168465    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-894dr\" (UniqueName: \"kubernetes.io/projected/ce271c8e-e982-4e3e-8588-26b79ccc6488-kube-api-access-894dr\") pod \"console-59d7d5f8d8-vxsmf\" (UID: \"ce271c8e-e982-4e3e-8588-26b79ccc6488\") " pod="openshift-console/console-59d7d5f8d8-vxsmf"
Jul 27 17:30:06.168594 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:06.168487    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ce271c8e-e982-4e3e-8588-26b79ccc6488-service-ca\") pod \"console-59d7d5f8d8-vxsmf\" (UID: \"ce271c8e-e982-4e3e-8588-26b79ccc6488\") " pod="openshift-console/console-59d7d5f8d8-vxsmf"
Jul 27 17:30:06.169214 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:06.169185    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ce271c8e-e982-4e3e-8588-26b79ccc6488-oauth-serving-cert\") pod \"console-59d7d5f8d8-vxsmf\" (UID: \"ce271c8e-e982-4e3e-8588-26b79ccc6488\") " pod="openshift-console/console-59d7d5f8d8-vxsmf"
Jul 27 17:30:06.169387 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:06.169283    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ce271c8e-e982-4e3e-8588-26b79ccc6488-console-config\") pod \"console-59d7d5f8d8-vxsmf\" (UID: \"ce271c8e-e982-4e3e-8588-26b79ccc6488\") " pod="openshift-console/console-59d7d5f8d8-vxsmf"
Jul 27 17:30:06.169387 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:06.169316    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ce271c8e-e982-4e3e-8588-26b79ccc6488-service-ca\") pod \"console-59d7d5f8d8-vxsmf\" (UID: \"ce271c8e-e982-4e3e-8588-26b79ccc6488\") " pod="openshift-console/console-59d7d5f8d8-vxsmf"
Jul 27 17:30:06.169387 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:06.169336    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce271c8e-e982-4e3e-8588-26b79ccc6488-trusted-ca-bundle\") pod \"console-59d7d5f8d8-vxsmf\" (UID: \"ce271c8e-e982-4e3e-8588-26b79ccc6488\") " pod="openshift-console/console-59d7d5f8d8-vxsmf"
Jul 27 17:30:06.170797 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:06.170776    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce271c8e-e982-4e3e-8588-26b79ccc6488-console-serving-cert\") pod \"console-59d7d5f8d8-vxsmf\" (UID: \"ce271c8e-e982-4e3e-8588-26b79ccc6488\") " pod="openshift-console/console-59d7d5f8d8-vxsmf"
Jul 27 17:30:06.170895 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:06.170807    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ce271c8e-e982-4e3e-8588-26b79ccc6488-console-oauth-config\") pod \"console-59d7d5f8d8-vxsmf\" (UID: \"ce271c8e-e982-4e3e-8588-26b79ccc6488\") " pod="openshift-console/console-59d7d5f8d8-vxsmf"
Jul 27 17:30:06.177453 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:06.177431    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-894dr\" (UniqueName: \"kubernetes.io/projected/ce271c8e-e982-4e3e-8588-26b79ccc6488-kube-api-access-894dr\") pod \"console-59d7d5f8d8-vxsmf\" (UID: \"ce271c8e-e982-4e3e-8588-26b79ccc6488\") " pod="openshift-console/console-59d7d5f8d8-vxsmf"
Jul 27 17:30:06.295941 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:06.295858    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59d7d5f8d8-vxsmf"
Jul 27 17:30:06.313190 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:06.313113    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:30:06.429934 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:06.429902    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59d7d5f8d8-vxsmf"]
Jul 27 17:30:06.432621 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:30:06.432594    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce271c8e_e982_4e3e_8588_26b79ccc6488.slice/crio-d4d4b0a2548bc955dfd4b52715aaad71f1615f2bc82a83ffb3a1a580ed745c25 WatchSource:0}: Error finding container d4d4b0a2548bc955dfd4b52715aaad71f1615f2bc82a83ffb3a1a580ed745c25: Status 404 returned error can't find the container with id d4d4b0a2548bc955dfd4b52715aaad71f1615f2bc82a83ffb3a1a580ed745c25
Jul 27 17:30:06.468813 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:06.468776    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f2becbf-a41c-4215-9435-9f735c60822e" path="/var/lib/kubelet/pods/6f2becbf-a41c-4215-9435-9f735c60822e/volumes"
Jul 27 17:30:06.900452 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:06.900408    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:30:06.977132 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:06.977101    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:30:07.023301 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:07.023239    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59d7d5f8d8-vxsmf" event={"ID":"ce271c8e-e982-4e3e-8588-26b79ccc6488","Type":"ContainerStarted","Data":"d4d4b0a2548bc955dfd4b52715aaad71f1615f2bc82a83ffb3a1a580ed745c25"}
Jul 27 17:30:09.685892 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:09.685853    2573 patch_prober.go:28] interesting pod/image-registry-74648bb95f-zz4vg container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 27 17:30:09.686410 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:09.685916    2573 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-74648bb95f-zz4vg" podUID="0e5f0900-1fec-4aa8-9372-45cb418b7bdd" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 17:30:10.362902 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:10.362849    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:30:10.421003 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:10.420963    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:30:11.900463 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:11.900431    2573 patch_prober.go:28] interesting pod/image-registry-74648bb95f-zz4vg container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 27 17:30:11.900865 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:11.900487    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-74648bb95f-zz4vg" podUID="0e5f0900-1fec-4aa8-9372-45cb418b7bdd" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 17:30:12.047974 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:12.047934    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59d7d5f8d8-vxsmf" event={"ID":"ce271c8e-e982-4e3e-8588-26b79ccc6488","Type":"ContainerStarted","Data":"362811b07c4dc85154d11f4fea315745e2e161dc3d1e80429717bc62758c39dc"}
Jul 27 17:30:12.067768 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:12.067721    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-59d7d5f8d8-vxsmf" podStartSLOduration=1.62995735 podStartE2EDuration="7.067705726s" totalImagesPullingTime="5.437748376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:30:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:30:12.067120346 +0000 UTC m=+154.210522398" watchObservedRunningTime="2026-07-27 17:30:12.067705726 +0000 UTC m=+154.211107776"
Jul 27 17:30:13.353796 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:13.353759    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:30:13.414298 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:13.414239    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:30:16.296111 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:16.296071    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-59d7d5f8d8-vxsmf"
Jul 27 17:30:16.296564 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:16.296130    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-59d7d5f8d8-vxsmf"
Jul 27 17:30:16.297598 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:16.297577    2573 patch_prober.go:28] interesting pod/console-59d7d5f8d8-vxsmf container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.133.0.20:8443/health\": dial tcp 10.133.0.20:8443: connect: connection refused" start-of-body=
Jul 27 17:30:16.297654 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:16.297620    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-59d7d5f8d8-vxsmf" podUID="ce271c8e-e982-4e3e-8588-26b79ccc6488" containerName="console" probeResult="failure" output="Get \"https://10.133.0.20:8443/health\": dial tcp 10.133.0.20:8443: connect: connection refused"
Jul 27 17:30:17.162533 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:30:17.162486    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-php8z" podUID="b4f69c89-1bf8-483d-8854-324c9a551e60"
Jul 27 17:30:18.069882 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:18.069855    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-php8z"
Jul 27 17:30:19.684555 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:19.684514    2573 patch_prober.go:28] interesting pod/image-registry-74648bb95f-zz4vg container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 27 17:30:19.685053 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:19.684570    2573 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-74648bb95f-zz4vg" podUID="0e5f0900-1fec-4aa8-9372-45cb418b7bdd" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 17:30:19.685053 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:19.684615    2573 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-image-registry/image-registry-74648bb95f-zz4vg"
Jul 27 17:30:19.685208 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:19.685185    2573 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="registry" containerStatusID={"Type":"cri-o","ID":"f12c8cff73e6c4071cba1c05b023765c1089b45d5c316097902fee3fcf7337fb"} pod="openshift-image-registry/image-registry-74648bb95f-zz4vg" containerMessage="Container registry failed liveness probe, will be restarted"
Jul 27 17:30:19.688798 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:19.688775    2573 patch_prober.go:28] interesting pod/image-registry-74648bb95f-zz4vg container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 27 17:30:19.688922 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:19.688814    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-74648bb95f-zz4vg" podUID="0e5f0900-1fec-4aa8-9372-45cb418b7bdd" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 17:30:22.115027 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:22.114987    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4f69c89-1bf8-483d-8854-324c9a551e60-metrics-tls\") pod \"dns-default-php8z\" (UID: \"b4f69c89-1bf8-483d-8854-324c9a551e60\") " pod="openshift-dns/dns-default-php8z"
Jul 27 17:30:22.117316 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:22.117295    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4f69c89-1bf8-483d-8854-324c9a551e60-metrics-tls\") pod \"dns-default-php8z\" (UID: \"b4f69c89-1bf8-483d-8854-324c9a551e60\") " pod="openshift-dns/dns-default-php8z"
Jul 27 17:30:22.274275 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:22.274217    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-swd8n\""
Jul 27 17:30:22.281978 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:22.281956    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-php8z"
Jul 27 17:30:22.437881 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:22.437857    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-php8z"]
Jul 27 17:30:22.440404 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:30:22.440379    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4f69c89_1bf8_483d_8854_324c9a551e60.slice/crio-f0461a2e21acf014aab47f91c2abcbdba49a79ec552e026d301a3866d3524b64 WatchSource:0}: Error finding container f0461a2e21acf014aab47f91c2abcbdba49a79ec552e026d301a3866d3524b64: Status 404 returned error can't find the container with id f0461a2e21acf014aab47f91c2abcbdba49a79ec552e026d301a3866d3524b64
Jul 27 17:30:22.994229 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:22.994190    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:30:23.048696 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:23.048645    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:30:23.089779 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:23.089744    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-php8z" event={"ID":"b4f69c89-1bf8-483d-8854-324c9a551e60","Type":"ContainerStarted","Data":"f0461a2e21acf014aab47f91c2abcbdba49a79ec552e026d301a3866d3524b64"}
Jul 27 17:30:24.095450 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:24.095415    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-php8z" event={"ID":"b4f69c89-1bf8-483d-8854-324c9a551e60","Type":"ContainerStarted","Data":"9f925f02faab5510fec2649bc2176614bdae1e4e1608aef4aaeaeebd27c6f39b"}
Jul 27 17:30:24.095839 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:24.095472    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-php8z" event={"ID":"b4f69c89-1bf8-483d-8854-324c9a551e60","Type":"ContainerStarted","Data":"bea779812f1e68acd3ef580e5582cbf5789b95de19388a4b1065a38806029bff"}
Jul 27 17:30:24.095839 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:24.095494    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-php8z"
Jul 27 17:30:24.115767 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:24.115658    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-php8z" podStartSLOduration=128.701769724 podStartE2EDuration="2m10.115641709s" totalImagesPullingTime="1.413871985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:28:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:30:24.114120764 +0000 UTC m=+166.257522817" watchObservedRunningTime="2026-07-27 17:30:24.115641709 +0000 UTC m=+166.259043760"
Jul 27 17:30:25.106815 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:25.106786    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:30:25.172915 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:25.172875    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:30:26.296652 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:26.296616    2573 patch_prober.go:28] interesting pod/console-59d7d5f8d8-vxsmf container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.133.0.20:8443/health\": dial tcp 10.133.0.20:8443: connect: connection refused" start-of-body=
Jul 27 17:30:26.297044 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:26.296672    2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-59d7d5f8d8-vxsmf" podUID="ce271c8e-e982-4e3e-8588-26b79ccc6488" containerName="console" probeResult="failure" output="Get \"https://10.133.0.20:8443/health\": dial tcp 10.133.0.20:8443: connect: connection refused"
Jul 27 17:30:29.689073 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:29.688991    2573 patch_prober.go:28] interesting pod/image-registry-74648bb95f-zz4vg container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 27 17:30:29.689073 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:29.689055    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-74648bb95f-zz4vg" podUID="0e5f0900-1fec-4aa8-9372-45cb418b7bdd" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 17:30:32.486350 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:32.486317    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:30:32.591305 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:32.591267    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:30:32.689082 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:32.689040    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:30:32.787647 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:32.787545    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:30:34.101532 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:34.101502    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-php8z"
Jul 27 17:30:35.904029 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:35.903998    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:30:35.969825 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:35.969794    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:30:36.301219 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:36.301139    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-59d7d5f8d8-vxsmf"
Jul 27 17:30:36.305053 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:36.305020    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-59d7d5f8d8-vxsmf"
Jul 27 17:30:36.515042 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:36.515002    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:30:36.615283 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:36.615180    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:30:36.915776 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:36.915741    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:30:37.016108 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:37.016065    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:30:37.210841 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:37.210750    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a0874ece-9e32-43fd-b926-98c8a3d7b44c/init-config-reloader/0.log"
Jul 27 17:30:37.411096 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:37.411067    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a0874ece-9e32-43fd-b926-98c8a3d7b44c/alertmanager/0.log"
Jul 27 17:30:37.611124 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:37.611036    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a0874ece-9e32-43fd-b926-98c8a3d7b44c/config-reloader/0.log"
Jul 27 17:30:37.810810 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:37.810783    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a0874ece-9e32-43fd-b926-98c8a3d7b44c/kube-rbac-proxy-web/0.log"
Jul 27 17:30:38.010092 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:38.010067    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a0874ece-9e32-43fd-b926-98c8a3d7b44c/kube-rbac-proxy/0.log"
Jul 27 17:30:38.210855 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:38.210827    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a0874ece-9e32-43fd-b926-98c8a3d7b44c/kube-rbac-proxy-metric/0.log"
Jul 27 17:30:38.411373 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:38.411348    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a0874ece-9e32-43fd-b926-98c8a3d7b44c/prom-label-proxy/0.log"
Jul 27 17:30:38.612795 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:38.612766    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-vx9kx_7ab3e588-fe39-4747-8d1f-9b1ce44ffed4/cluster-monitoring-operator/0.log"
Jul 27 17:30:39.610605 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:39.610563    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-pwwlv_d89a1662-c5db-4f1a-8466-2006b605667b/monitoring-plugin/0.log"
Jul 27 17:30:39.689699 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:39.689665    2573 patch_prober.go:28] interesting pod/image-registry-74648bb95f-zz4vg container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 27 17:30:39.689861 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:39.689714    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-74648bb95f-zz4vg" podUID="0e5f0900-1fec-4aa8-9372-45cb418b7bdd" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 17:30:40.410224 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:40.410189    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-chc8q_e4cabfc1-9cab-4eb9-8e32-7dd66c04d572/init-textfile/0.log"
Jul 27 17:30:40.617202 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:40.617174    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-chc8q_e4cabfc1-9cab-4eb9-8e32-7dd66c04d572/node-exporter/0.log"
Jul 27 17:30:40.813499 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:40.813422    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-chc8q_e4cabfc1-9cab-4eb9-8e32-7dd66c04d572/kube-rbac-proxy/0.log"
Jul 27 17:30:41.008181 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:41.008149    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:30:41.110046 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:41.109962    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:30:41.611306 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:41.611274    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-rxfs4_61385c00-c295-4f64-a5c8-d5f0dd2ce6af/kube-rbac-proxy-main/0.log"
Jul 27 17:30:41.810807 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:41.810770    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-rxfs4_61385c00-c295-4f64-a5c8-d5f0dd2ce6af/kube-rbac-proxy-self/0.log"
Jul 27 17:30:42.011032 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:42.011002    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-rxfs4_61385c00-c295-4f64-a5c8-d5f0dd2ce6af/openshift-state-metrics/0.log"
Jul 27 17:30:42.587993 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:42.587963    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:30:43.613370 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:43.613336    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-sscsd_8cf5e605-7a6e-4cf6-af0c-5e7b19038568/prometheus-operator/0.log"
Jul 27 17:30:43.822883 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:43.822852    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-sscsd_8cf5e605-7a6e-4cf6-af0c-5e7b19038568/kube-rbac-proxy/0.log"
Jul 27 17:30:44.010759 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:44.010732    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-9xsht_da10c088-ba88-4668-8e2b-eda1f18d6203/prometheus-operator-admission-webhook/0.log"
Jul 27 17:30:44.703626 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:44.703581    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-74648bb95f-zz4vg" podUID="0e5f0900-1fec-4aa8-9372-45cb418b7bdd" containerName="registry" containerID="cri-o://f12c8cff73e6c4071cba1c05b023765c1089b45d5c316097902fee3fcf7337fb" gracePeriod=30
Jul 27 17:30:44.905441 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:44.905402    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:30:45.005508 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:45.005427    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:30:46.182941 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:46.182906    2573 generic.go:356] "Generic (PLEG): container finished" podID="0e5f0900-1fec-4aa8-9372-45cb418b7bdd" containerID="f12c8cff73e6c4071cba1c05b023765c1089b45d5c316097902fee3fcf7337fb" exitCode=0
Jul 27 17:30:46.183323 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:46.182983    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74648bb95f-zz4vg" event={"ID":"0e5f0900-1fec-4aa8-9372-45cb418b7bdd","Type":"ContainerDied","Data":"f12c8cff73e6c4071cba1c05b023765c1089b45d5c316097902fee3fcf7337fb"}
Jul 27 17:30:46.183323 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:46.183030    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74648bb95f-zz4vg" event={"ID":"0e5f0900-1fec-4aa8-9372-45cb418b7bdd","Type":"ContainerStarted","Data":"39e0087d37f06d7b6f7e31ae0c6bc17a149c190c80fbe4b83b57a02a18920873"}
Jul 27 17:30:46.183323 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:46.183187    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-74648bb95f-zz4vg"
Jul 27 17:30:46.611806 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:46.611723    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-59d7d5f8d8-vxsmf_ce271c8e-e982-4e3e-8588-26b79ccc6488/console/0.log"
Jul 27 17:30:47.629678 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:47.629649    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-php8z_b4f69c89-1bf8-483d-8854-324c9a551e60/dns/0.log"
Jul 27 17:30:47.811684 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:47.811652    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-php8z_b4f69c89-1bf8-483d-8854-324c9a551e60/kube-rbac-proxy/0.log"
Jul 27 17:30:49.011084 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:49.011062    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-95qjs_e19a218b-06c1-4547-9f14-00d201dc6030/dns-node-resolver/0.log"
Jul 27 17:30:49.611133 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:49.611098    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-74648bb95f-zz4vg_0e5f0900-1fec-4aa8-9372-45cb418b7bdd/registry/1.log"
Jul 27 17:30:49.994702 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:49.994671    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:30:50.051256 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:50.051229    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:30:50.212621 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:50.212596    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zb877_2802ee72-4be9-423a-a9bd-102c54f96d06/node-ca/0.log"
Jul 27 17:30:55.103596 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:55.103564    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:30:55.205045 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:30:55.205007    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:31:07.192190 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:07.192158    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-74648bb95f-zz4vg"
Jul 27 17:31:07.333902 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:07.333862    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:31:07.392807 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:07.392777    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:31:07.451615 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:07.451517    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:31:07.511591 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:07.511556    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:31:09.577934 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:09.577898    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:31:09.630641 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:09.630594    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:31:10.101048 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:10.101007    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:31:10.159450 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:10.159414    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:31:13.820534 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:13.820496    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 17:31:13.820968 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:13.820765    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a0874ece-9e32-43fd-b926-98c8a3d7b44c" containerName="alertmanager" containerID="cri-o://e7b3c3f6eb76a1550f5bd8badd477b872f565226dde98b845f3bc632692ce0c5" gracePeriod=120
Jul 27 17:31:13.820968 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:13.820857    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a0874ece-9e32-43fd-b926-98c8a3d7b44c" containerName="kube-rbac-proxy-web" containerID="cri-o://d18a3bf271c93c98660afea784f45b2b8941b98cddd408f9efddb4bf9c212d78" gracePeriod=120
Jul 27 17:31:13.820968 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:13.820870    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a0874ece-9e32-43fd-b926-98c8a3d7b44c" containerName="config-reloader" containerID="cri-o://14e579c9523eba2d8aa3310df56d6d1096909a52c61416b6ef1499e49c827a03" gracePeriod=120
Jul 27 17:31:13.820968 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:13.820926    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a0874ece-9e32-43fd-b926-98c8a3d7b44c" containerName="prom-label-proxy" containerID="cri-o://b33e182091f1d2f0dadf726285b173356e2b6f3b4783ed9016fa02ca5987fd19" gracePeriod=120
Jul 27 17:31:13.821183 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:13.820952    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a0874ece-9e32-43fd-b926-98c8a3d7b44c" containerName="kube-rbac-proxy" containerID="cri-o://be3d640dce34f19bc0d947c76c71272dd7849b55fc09e438548bf2f7768353d0" gracePeriod=120
Jul 27 17:31:13.821183 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:13.820846    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a0874ece-9e32-43fd-b926-98c8a3d7b44c" containerName="kube-rbac-proxy-metric" containerID="cri-o://ffef14148efe02ae50befbc26b89bcb4b0ae615ad54a48621ddb1d6c880d05bf" gracePeriod=120
Jul 27 17:31:14.300139 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:14.300106    2573 generic.go:356] "Generic (PLEG): container finished" podID="a0874ece-9e32-43fd-b926-98c8a3d7b44c" containerID="b33e182091f1d2f0dadf726285b173356e2b6f3b4783ed9016fa02ca5987fd19" exitCode=0
Jul 27 17:31:14.300139 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:14.300131    2573 generic.go:356] "Generic (PLEG): container finished" podID="a0874ece-9e32-43fd-b926-98c8a3d7b44c" containerID="be3d640dce34f19bc0d947c76c71272dd7849b55fc09e438548bf2f7768353d0" exitCode=0
Jul 27 17:31:14.300139 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:14.300137    2573 generic.go:356] "Generic (PLEG): container finished" podID="a0874ece-9e32-43fd-b926-98c8a3d7b44c" containerID="14e579c9523eba2d8aa3310df56d6d1096909a52c61416b6ef1499e49c827a03" exitCode=0
Jul 27 17:31:14.300139 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:14.300144    2573 generic.go:356] "Generic (PLEG): container finished" podID="a0874ece-9e32-43fd-b926-98c8a3d7b44c" containerID="e7b3c3f6eb76a1550f5bd8badd477b872f565226dde98b845f3bc632692ce0c5" exitCode=0
Jul 27 17:31:14.300477 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:14.300184    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0874ece-9e32-43fd-b926-98c8a3d7b44c","Type":"ContainerDied","Data":"b33e182091f1d2f0dadf726285b173356e2b6f3b4783ed9016fa02ca5987fd19"}
Jul 27 17:31:14.300477 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:14.300282    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0874ece-9e32-43fd-b926-98c8a3d7b44c","Type":"ContainerDied","Data":"be3d640dce34f19bc0d947c76c71272dd7849b55fc09e438548bf2f7768353d0"}
Jul 27 17:31:14.300477 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:14.300324    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0874ece-9e32-43fd-b926-98c8a3d7b44c","Type":"ContainerDied","Data":"14e579c9523eba2d8aa3310df56d6d1096909a52c61416b6ef1499e49c827a03"}
Jul 27 17:31:14.300477 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:14.300361    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0874ece-9e32-43fd-b926-98c8a3d7b44c","Type":"ContainerDied","Data":"e7b3c3f6eb76a1550f5bd8badd477b872f565226dde98b845f3bc632692ce0c5"}
Jul 27 17:31:15.079680 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.079644    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:31:15.153095 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.153062    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0874ece-9e32-43fd-b926-98c8a3d7b44c-metrics-client-ca\") pod \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\" (UID: \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\") "
Jul 27 17:31:15.153095 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.153101    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a0874ece-9e32-43fd-b926-98c8a3d7b44c-alertmanager-main-db\") pod \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\" (UID: \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\") "
Jul 27 17:31:15.153383 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.153121    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a0874ece-9e32-43fd-b926-98c8a3d7b44c-config-out\") pod \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\" (UID: \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\") "
Jul 27 17:31:15.153383 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.153148    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a0874ece-9e32-43fd-b926-98c8a3d7b44c-cluster-tls-config\") pod \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\" (UID: \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\") "
Jul 27 17:31:15.153383 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.153174    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a0874ece-9e32-43fd-b926-98c8a3d7b44c-tls-assets\") pod \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\" (UID: \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\") "
Jul 27 17:31:15.153383 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.153197    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a0874ece-9e32-43fd-b926-98c8a3d7b44c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\" (UID: \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\") "
Jul 27 17:31:15.153383 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.153268    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a0874ece-9e32-43fd-b926-98c8a3d7b44c-secret-alertmanager-kube-rbac-proxy\") pod \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\" (UID: \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\") "
Jul 27 17:31:15.153383 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.153313    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a0874ece-9e32-43fd-b926-98c8a3d7b44c-config-volume\") pod \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\" (UID: \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\") "
Jul 27 17:31:15.153383 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.153342    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a0874ece-9e32-43fd-b926-98c8a3d7b44c-secret-alertmanager-main-tls\") pod \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\" (UID: \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\") "
Jul 27 17:31:15.153787 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.153385    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a0874ece-9e32-43fd-b926-98c8a3d7b44c-secret-alertmanager-kube-rbac-proxy-web\") pod \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\" (UID: \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\") "
Jul 27 17:31:15.153787 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.153412    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0874ece-9e32-43fd-b926-98c8a3d7b44c-alertmanager-trusted-ca-bundle\") pod \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\" (UID: \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\") "
Jul 27 17:31:15.153787 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.153437    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kzjzb\" (UniqueName: \"kubernetes.io/projected/a0874ece-9e32-43fd-b926-98c8a3d7b44c-kube-api-access-kzjzb\") pod \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\" (UID: \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\") "
Jul 27 17:31:15.153787 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.153464    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a0874ece-9e32-43fd-b926-98c8a3d7b44c-web-config\") pod \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\" (UID: \"a0874ece-9e32-43fd-b926-98c8a3d7b44c\") "
Jul 27 17:31:15.153787 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.153507    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0874ece-9e32-43fd-b926-98c8a3d7b44c-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "a0874ece-9e32-43fd-b926-98c8a3d7b44c" (UID: "a0874ece-9e32-43fd-b926-98c8a3d7b44c"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:31:15.153787 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.153517    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0874ece-9e32-43fd-b926-98c8a3d7b44c-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "a0874ece-9e32-43fd-b926-98c8a3d7b44c" (UID: "a0874ece-9e32-43fd-b926-98c8a3d7b44c"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:31:15.154092 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.153850    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0874ece-9e32-43fd-b926-98c8a3d7b44c-metrics-client-ca\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:31:15.154092 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.153873    2573 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a0874ece-9e32-43fd-b926-98c8a3d7b44c-alertmanager-main-db\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:31:15.154092 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.153928    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0874ece-9e32-43fd-b926-98c8a3d7b44c-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "a0874ece-9e32-43fd-b926-98c8a3d7b44c" (UID: "a0874ece-9e32-43fd-b926-98c8a3d7b44c"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:31:15.156609 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.156568    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0874ece-9e32-43fd-b926-98c8a3d7b44c-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "a0874ece-9e32-43fd-b926-98c8a3d7b44c" (UID: "a0874ece-9e32-43fd-b926-98c8a3d7b44c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:31:15.156609 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.156591    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0874ece-9e32-43fd-b926-98c8a3d7b44c-config-out" (OuterVolumeSpecName: "config-out") pod "a0874ece-9e32-43fd-b926-98c8a3d7b44c" (UID: "a0874ece-9e32-43fd-b926-98c8a3d7b44c"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:31:15.157135 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.157026    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0874ece-9e32-43fd-b926-98c8a3d7b44c-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "a0874ece-9e32-43fd-b926-98c8a3d7b44c" (UID: "a0874ece-9e32-43fd-b926-98c8a3d7b44c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:31:15.157135 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.157101    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0874ece-9e32-43fd-b926-98c8a3d7b44c-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "a0874ece-9e32-43fd-b926-98c8a3d7b44c" (UID: "a0874ece-9e32-43fd-b926-98c8a3d7b44c"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:31:15.157447 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.157420    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0874ece-9e32-43fd-b926-98c8a3d7b44c-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "a0874ece-9e32-43fd-b926-98c8a3d7b44c" (UID: "a0874ece-9e32-43fd-b926-98c8a3d7b44c"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:31:15.157525 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.157440    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0874ece-9e32-43fd-b926-98c8a3d7b44c-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "a0874ece-9e32-43fd-b926-98c8a3d7b44c" (UID: "a0874ece-9e32-43fd-b926-98c8a3d7b44c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:31:15.157910 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.157886    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0874ece-9e32-43fd-b926-98c8a3d7b44c-kube-api-access-kzjzb" (OuterVolumeSpecName: "kube-api-access-kzjzb") pod "a0874ece-9e32-43fd-b926-98c8a3d7b44c" (UID: "a0874ece-9e32-43fd-b926-98c8a3d7b44c"). InnerVolumeSpecName "kube-api-access-kzjzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:31:15.159123 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.159099    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0874ece-9e32-43fd-b926-98c8a3d7b44c-config-volume" (OuterVolumeSpecName: "config-volume") pod "a0874ece-9e32-43fd-b926-98c8a3d7b44c" (UID: "a0874ece-9e32-43fd-b926-98c8a3d7b44c"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:31:15.161870 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.161798    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0874ece-9e32-43fd-b926-98c8a3d7b44c-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "a0874ece-9e32-43fd-b926-98c8a3d7b44c" (UID: "a0874ece-9e32-43fd-b926-98c8a3d7b44c"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:31:15.168347 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.168320    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0874ece-9e32-43fd-b926-98c8a3d7b44c-web-config" (OuterVolumeSpecName: "web-config") pod "a0874ece-9e32-43fd-b926-98c8a3d7b44c" (UID: "a0874ece-9e32-43fd-b926-98c8a3d7b44c"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:31:15.255034 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.255002    2573 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a0874ece-9e32-43fd-b926-98c8a3d7b44c-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:31:15.255034 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.255030    2573 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a0874ece-9e32-43fd-b926-98c8a3d7b44c-config-volume\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:31:15.255034 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.255042    2573 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a0874ece-9e32-43fd-b926-98c8a3d7b44c-secret-alertmanager-main-tls\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:31:15.255209 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.255051    2573 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a0874ece-9e32-43fd-b926-98c8a3d7b44c-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:31:15.255209 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.255062    2573 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0874ece-9e32-43fd-b926-98c8a3d7b44c-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:31:15.255209 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.255070    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzjzb\" (UniqueName: \"kubernetes.io/projected/a0874ece-9e32-43fd-b926-98c8a3d7b44c-kube-api-access-kzjzb\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:31:15.255209 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.255101    2573 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a0874ece-9e32-43fd-b926-98c8a3d7b44c-web-config\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:31:15.255209 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.255110    2573 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a0874ece-9e32-43fd-b926-98c8a3d7b44c-config-out\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:31:15.255209 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.255119    2573 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a0874ece-9e32-43fd-b926-98c8a3d7b44c-cluster-tls-config\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:31:15.255209 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.255128    2573 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a0874ece-9e32-43fd-b926-98c8a3d7b44c-tls-assets\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:31:15.255209 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.255138    2573 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a0874ece-9e32-43fd-b926-98c8a3d7b44c-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:31:15.307420 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.307390    2573 generic.go:356] "Generic (PLEG): container finished" podID="a0874ece-9e32-43fd-b926-98c8a3d7b44c" containerID="ffef14148efe02ae50befbc26b89bcb4b0ae615ad54a48621ddb1d6c880d05bf" exitCode=0
Jul 27 17:31:15.307420 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.307414    2573 generic.go:356] "Generic (PLEG): container finished" podID="a0874ece-9e32-43fd-b926-98c8a3d7b44c" containerID="d18a3bf271c93c98660afea784f45b2b8941b98cddd408f9efddb4bf9c212d78" exitCode=0
Jul 27 17:31:15.307575 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.307433    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0874ece-9e32-43fd-b926-98c8a3d7b44c","Type":"ContainerDied","Data":"ffef14148efe02ae50befbc26b89bcb4b0ae615ad54a48621ddb1d6c880d05bf"}
Jul 27 17:31:15.307575 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.307464    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:31:15.307575 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.307514    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0874ece-9e32-43fd-b926-98c8a3d7b44c","Type":"ContainerDied","Data":"d18a3bf271c93c98660afea784f45b2b8941b98cddd408f9efddb4bf9c212d78"}
Jul 27 17:31:15.307575 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.307546    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a0874ece-9e32-43fd-b926-98c8a3d7b44c","Type":"ContainerDied","Data":"cf0510b1d73b2ce356c940a6398141d33f2630139ab330bdcfa4fe80ab698483"}
Jul 27 17:31:15.307733 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.307588    2573 scope.go:117] "RemoveContainer" containerID="b33e182091f1d2f0dadf726285b173356e2b6f3b4783ed9016fa02ca5987fd19"
Jul 27 17:31:15.315912 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.315887    2573 scope.go:117] "RemoveContainer" containerID="ffef14148efe02ae50befbc26b89bcb4b0ae615ad54a48621ddb1d6c880d05bf"
Jul 27 17:31:15.323002 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.322985    2573 scope.go:117] "RemoveContainer" containerID="be3d640dce34f19bc0d947c76c71272dd7849b55fc09e438548bf2f7768353d0"
Jul 27 17:31:15.329952 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.329933    2573 scope.go:117] "RemoveContainer" containerID="d18a3bf271c93c98660afea784f45b2b8941b98cddd408f9efddb4bf9c212d78"
Jul 27 17:31:15.332201 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.332177    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 17:31:15.337096 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.337071    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 17:31:15.338381 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.338365    2573 scope.go:117] "RemoveContainer" containerID="14e579c9523eba2d8aa3310df56d6d1096909a52c61416b6ef1499e49c827a03"
Jul 27 17:31:15.345308 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.345285    2573 scope.go:117] "RemoveContainer" containerID="e7b3c3f6eb76a1550f5bd8badd477b872f565226dde98b845f3bc632692ce0c5"
Jul 27 17:31:15.352508 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.352488    2573 scope.go:117] "RemoveContainer" containerID="112d8c4407d93d8ee91c65179d949f8310702be16c0266429b9630f39097fcee"
Jul 27 17:31:15.359546 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.359529    2573 scope.go:117] "RemoveContainer" containerID="b33e182091f1d2f0dadf726285b173356e2b6f3b4783ed9016fa02ca5987fd19"
Jul 27 17:31:15.359839 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:31:15.359808    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b33e182091f1d2f0dadf726285b173356e2b6f3b4783ed9016fa02ca5987fd19\": container with ID starting with b33e182091f1d2f0dadf726285b173356e2b6f3b4783ed9016fa02ca5987fd19 not found: ID does not exist" containerID="b33e182091f1d2f0dadf726285b173356e2b6f3b4783ed9016fa02ca5987fd19"
Jul 27 17:31:15.359917 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.359853    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b33e182091f1d2f0dadf726285b173356e2b6f3b4783ed9016fa02ca5987fd19"} err="failed to get container status \"b33e182091f1d2f0dadf726285b173356e2b6f3b4783ed9016fa02ca5987fd19\": rpc error: code = NotFound desc = could not find container \"b33e182091f1d2f0dadf726285b173356e2b6f3b4783ed9016fa02ca5987fd19\": container with ID starting with b33e182091f1d2f0dadf726285b173356e2b6f3b4783ed9016fa02ca5987fd19 not found: ID does not exist"
Jul 27 17:31:15.359917 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.359880    2573 scope.go:117] "RemoveContainer" containerID="ffef14148efe02ae50befbc26b89bcb4b0ae615ad54a48621ddb1d6c880d05bf"
Jul 27 17:31:15.360146 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:31:15.360126    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffef14148efe02ae50befbc26b89bcb4b0ae615ad54a48621ddb1d6c880d05bf\": container with ID starting with ffef14148efe02ae50befbc26b89bcb4b0ae615ad54a48621ddb1d6c880d05bf not found: ID does not exist" containerID="ffef14148efe02ae50befbc26b89bcb4b0ae615ad54a48621ddb1d6c880d05bf"
Jul 27 17:31:15.360194 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.360152    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffef14148efe02ae50befbc26b89bcb4b0ae615ad54a48621ddb1d6c880d05bf"} err="failed to get container status \"ffef14148efe02ae50befbc26b89bcb4b0ae615ad54a48621ddb1d6c880d05bf\": rpc error: code = NotFound desc = could not find container \"ffef14148efe02ae50befbc26b89bcb4b0ae615ad54a48621ddb1d6c880d05bf\": container with ID starting with ffef14148efe02ae50befbc26b89bcb4b0ae615ad54a48621ddb1d6c880d05bf not found: ID does not exist"
Jul 27 17:31:15.360194 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.360169    2573 scope.go:117] "RemoveContainer" containerID="be3d640dce34f19bc0d947c76c71272dd7849b55fc09e438548bf2f7768353d0"
Jul 27 17:31:15.360418 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:31:15.360401    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be3d640dce34f19bc0d947c76c71272dd7849b55fc09e438548bf2f7768353d0\": container with ID starting with be3d640dce34f19bc0d947c76c71272dd7849b55fc09e438548bf2f7768353d0 not found: ID does not exist" containerID="be3d640dce34f19bc0d947c76c71272dd7849b55fc09e438548bf2f7768353d0"
Jul 27 17:31:15.360489 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.360427    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be3d640dce34f19bc0d947c76c71272dd7849b55fc09e438548bf2f7768353d0"} err="failed to get container status \"be3d640dce34f19bc0d947c76c71272dd7849b55fc09e438548bf2f7768353d0\": rpc error: code = NotFound desc = could not find container \"be3d640dce34f19bc0d947c76c71272dd7849b55fc09e438548bf2f7768353d0\": container with ID starting with be3d640dce34f19bc0d947c76c71272dd7849b55fc09e438548bf2f7768353d0 not found: ID does not exist"
Jul 27 17:31:15.360489 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.360448    2573 scope.go:117] "RemoveContainer" containerID="d18a3bf271c93c98660afea784f45b2b8941b98cddd408f9efddb4bf9c212d78"
Jul 27 17:31:15.360743 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:31:15.360726    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d18a3bf271c93c98660afea784f45b2b8941b98cddd408f9efddb4bf9c212d78\": container with ID starting with d18a3bf271c93c98660afea784f45b2b8941b98cddd408f9efddb4bf9c212d78 not found: ID does not exist" containerID="d18a3bf271c93c98660afea784f45b2b8941b98cddd408f9efddb4bf9c212d78"
Jul 27 17:31:15.360781 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.360750    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d18a3bf271c93c98660afea784f45b2b8941b98cddd408f9efddb4bf9c212d78"} err="failed to get container status \"d18a3bf271c93c98660afea784f45b2b8941b98cddd408f9efddb4bf9c212d78\": rpc error: code = NotFound desc = could not find container \"d18a3bf271c93c98660afea784f45b2b8941b98cddd408f9efddb4bf9c212d78\": container with ID starting with d18a3bf271c93c98660afea784f45b2b8941b98cddd408f9efddb4bf9c212d78 not found: ID does not exist"
Jul 27 17:31:15.360781 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.360765    2573 scope.go:117] "RemoveContainer" containerID="14e579c9523eba2d8aa3310df56d6d1096909a52c61416b6ef1499e49c827a03"
Jul 27 17:31:15.360990 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:31:15.360974    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14e579c9523eba2d8aa3310df56d6d1096909a52c61416b6ef1499e49c827a03\": container with ID starting with 14e579c9523eba2d8aa3310df56d6d1096909a52c61416b6ef1499e49c827a03 not found: ID does not exist" containerID="14e579c9523eba2d8aa3310df56d6d1096909a52c61416b6ef1499e49c827a03"
Jul 27 17:31:15.361032 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.360995    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14e579c9523eba2d8aa3310df56d6d1096909a52c61416b6ef1499e49c827a03"} err="failed to get container status \"14e579c9523eba2d8aa3310df56d6d1096909a52c61416b6ef1499e49c827a03\": rpc error: code = NotFound desc = could not find container \"14e579c9523eba2d8aa3310df56d6d1096909a52c61416b6ef1499e49c827a03\": container with ID starting with 14e579c9523eba2d8aa3310df56d6d1096909a52c61416b6ef1499e49c827a03 not found: ID does not exist"
Jul 27 17:31:15.361032 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.361014    2573 scope.go:117] "RemoveContainer" containerID="e7b3c3f6eb76a1550f5bd8badd477b872f565226dde98b845f3bc632692ce0c5"
Jul 27 17:31:15.361232 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:31:15.361217    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7b3c3f6eb76a1550f5bd8badd477b872f565226dde98b845f3bc632692ce0c5\": container with ID starting with e7b3c3f6eb76a1550f5bd8badd477b872f565226dde98b845f3bc632692ce0c5 not found: ID does not exist" containerID="e7b3c3f6eb76a1550f5bd8badd477b872f565226dde98b845f3bc632692ce0c5"
Jul 27 17:31:15.361329 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.361239    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7b3c3f6eb76a1550f5bd8badd477b872f565226dde98b845f3bc632692ce0c5"} err="failed to get container status \"e7b3c3f6eb76a1550f5bd8badd477b872f565226dde98b845f3bc632692ce0c5\": rpc error: code = NotFound desc = could not find container \"e7b3c3f6eb76a1550f5bd8badd477b872f565226dde98b845f3bc632692ce0c5\": container with ID starting with e7b3c3f6eb76a1550f5bd8badd477b872f565226dde98b845f3bc632692ce0c5 not found: ID does not exist"
Jul 27 17:31:15.361329 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.361280    2573 scope.go:117] "RemoveContainer" containerID="112d8c4407d93d8ee91c65179d949f8310702be16c0266429b9630f39097fcee"
Jul 27 17:31:15.361489 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:31:15.361474    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"112d8c4407d93d8ee91c65179d949f8310702be16c0266429b9630f39097fcee\": container with ID starting with 112d8c4407d93d8ee91c65179d949f8310702be16c0266429b9630f39097fcee not found: ID does not exist" containerID="112d8c4407d93d8ee91c65179d949f8310702be16c0266429b9630f39097fcee"
Jul 27 17:31:15.361529 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.361497    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"112d8c4407d93d8ee91c65179d949f8310702be16c0266429b9630f39097fcee"} err="failed to get container status \"112d8c4407d93d8ee91c65179d949f8310702be16c0266429b9630f39097fcee\": rpc error: code = NotFound desc = could not find container \"112d8c4407d93d8ee91c65179d949f8310702be16c0266429b9630f39097fcee\": container with ID starting with 112d8c4407d93d8ee91c65179d949f8310702be16c0266429b9630f39097fcee not found: ID does not exist"
Jul 27 17:31:15.361529 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.361514    2573 scope.go:117] "RemoveContainer" containerID="b33e182091f1d2f0dadf726285b173356e2b6f3b4783ed9016fa02ca5987fd19"
Jul 27 17:31:15.361702 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.361685    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b33e182091f1d2f0dadf726285b173356e2b6f3b4783ed9016fa02ca5987fd19"} err="failed to get container status \"b33e182091f1d2f0dadf726285b173356e2b6f3b4783ed9016fa02ca5987fd19\": rpc error: code = NotFound desc = could not find container \"b33e182091f1d2f0dadf726285b173356e2b6f3b4783ed9016fa02ca5987fd19\": container with ID starting with b33e182091f1d2f0dadf726285b173356e2b6f3b4783ed9016fa02ca5987fd19 not found: ID does not exist"
Jul 27 17:31:15.361771 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.361704    2573 scope.go:117] "RemoveContainer" containerID="ffef14148efe02ae50befbc26b89bcb4b0ae615ad54a48621ddb1d6c880d05bf"
Jul 27 17:31:15.361909 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.361891    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffef14148efe02ae50befbc26b89bcb4b0ae615ad54a48621ddb1d6c880d05bf"} err="failed to get container status \"ffef14148efe02ae50befbc26b89bcb4b0ae615ad54a48621ddb1d6c880d05bf\": rpc error: code = NotFound desc = could not find container \"ffef14148efe02ae50befbc26b89bcb4b0ae615ad54a48621ddb1d6c880d05bf\": container with ID starting with ffef14148efe02ae50befbc26b89bcb4b0ae615ad54a48621ddb1d6c880d05bf not found: ID does not exist"
Jul 27 17:31:15.361965 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.361909    2573 scope.go:117] "RemoveContainer" containerID="be3d640dce34f19bc0d947c76c71272dd7849b55fc09e438548bf2f7768353d0"
Jul 27 17:31:15.362095 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.362080    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be3d640dce34f19bc0d947c76c71272dd7849b55fc09e438548bf2f7768353d0"} err="failed to get container status \"be3d640dce34f19bc0d947c76c71272dd7849b55fc09e438548bf2f7768353d0\": rpc error: code = NotFound desc = could not find container \"be3d640dce34f19bc0d947c76c71272dd7849b55fc09e438548bf2f7768353d0\": container with ID starting with be3d640dce34f19bc0d947c76c71272dd7849b55fc09e438548bf2f7768353d0 not found: ID does not exist"
Jul 27 17:31:15.362095 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.362094    2573 scope.go:117] "RemoveContainer" containerID="d18a3bf271c93c98660afea784f45b2b8941b98cddd408f9efddb4bf9c212d78"
Jul 27 17:31:15.362321 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.362303    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d18a3bf271c93c98660afea784f45b2b8941b98cddd408f9efddb4bf9c212d78"} err="failed to get container status \"d18a3bf271c93c98660afea784f45b2b8941b98cddd408f9efddb4bf9c212d78\": rpc error: code = NotFound desc = could not find container \"d18a3bf271c93c98660afea784f45b2b8941b98cddd408f9efddb4bf9c212d78\": container with ID starting with d18a3bf271c93c98660afea784f45b2b8941b98cddd408f9efddb4bf9c212d78 not found: ID does not exist"
Jul 27 17:31:15.362386 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.362322    2573 scope.go:117] "RemoveContainer" containerID="14e579c9523eba2d8aa3310df56d6d1096909a52c61416b6ef1499e49c827a03"
Jul 27 17:31:15.362544 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.362527    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14e579c9523eba2d8aa3310df56d6d1096909a52c61416b6ef1499e49c827a03"} err="failed to get container status \"14e579c9523eba2d8aa3310df56d6d1096909a52c61416b6ef1499e49c827a03\": rpc error: code = NotFound desc = could not find container \"14e579c9523eba2d8aa3310df56d6d1096909a52c61416b6ef1499e49c827a03\": container with ID starting with 14e579c9523eba2d8aa3310df56d6d1096909a52c61416b6ef1499e49c827a03 not found: ID does not exist"
Jul 27 17:31:15.362593 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.362543    2573 scope.go:117] "RemoveContainer" containerID="e7b3c3f6eb76a1550f5bd8badd477b872f565226dde98b845f3bc632692ce0c5"
Jul 27 17:31:15.362717 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.362702    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7b3c3f6eb76a1550f5bd8badd477b872f565226dde98b845f3bc632692ce0c5"} err="failed to get container status \"e7b3c3f6eb76a1550f5bd8badd477b872f565226dde98b845f3bc632692ce0c5\": rpc error: code = NotFound desc = could not find container \"e7b3c3f6eb76a1550f5bd8badd477b872f565226dde98b845f3bc632692ce0c5\": container with ID starting with e7b3c3f6eb76a1550f5bd8badd477b872f565226dde98b845f3bc632692ce0c5 not found: ID does not exist"
Jul 27 17:31:15.362717 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.362716    2573 scope.go:117] "RemoveContainer" containerID="112d8c4407d93d8ee91c65179d949f8310702be16c0266429b9630f39097fcee"
Jul 27 17:31:15.362893 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.362880    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"112d8c4407d93d8ee91c65179d949f8310702be16c0266429b9630f39097fcee"} err="failed to get container status \"112d8c4407d93d8ee91c65179d949f8310702be16c0266429b9630f39097fcee\": rpc error: code = NotFound desc = could not find container \"112d8c4407d93d8ee91c65179d949f8310702be16c0266429b9630f39097fcee\": container with ID starting with 112d8c4407d93d8ee91c65179d949f8310702be16c0266429b9630f39097fcee not found: ID does not exist"
Jul 27 17:31:15.365196 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.365176    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 17:31:15.365514 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.365500    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0874ece-9e32-43fd-b926-98c8a3d7b44c" containerName="kube-rbac-proxy-metric"
Jul 27 17:31:15.365568 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.365515    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0874ece-9e32-43fd-b926-98c8a3d7b44c" containerName="kube-rbac-proxy-metric"
Jul 27 17:31:15.365568 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.365527    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0874ece-9e32-43fd-b926-98c8a3d7b44c" containerName="config-reloader"
Jul 27 17:31:15.365568 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.365532    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0874ece-9e32-43fd-b926-98c8a3d7b44c" containerName="config-reloader"
Jul 27 17:31:15.365568 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.365538    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0874ece-9e32-43fd-b926-98c8a3d7b44c" containerName="kube-rbac-proxy"
Jul 27 17:31:15.365568 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.365544    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0874ece-9e32-43fd-b926-98c8a3d7b44c" containerName="kube-rbac-proxy"
Jul 27 17:31:15.365568 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.365551    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0874ece-9e32-43fd-b926-98c8a3d7b44c" containerName="kube-rbac-proxy-web"
Jul 27 17:31:15.365568 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.365557    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0874ece-9e32-43fd-b926-98c8a3d7b44c" containerName="kube-rbac-proxy-web"
Jul 27 17:31:15.365568 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.365564    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0874ece-9e32-43fd-b926-98c8a3d7b44c" containerName="alertmanager"
Jul 27 17:31:15.365568 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.365569    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0874ece-9e32-43fd-b926-98c8a3d7b44c" containerName="alertmanager"
Jul 27 17:31:15.365837 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.365581    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0874ece-9e32-43fd-b926-98c8a3d7b44c" containerName="prom-label-proxy"
Jul 27 17:31:15.365837 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.365586    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0874ece-9e32-43fd-b926-98c8a3d7b44c" containerName="prom-label-proxy"
Jul 27 17:31:15.365837 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.365598    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0874ece-9e32-43fd-b926-98c8a3d7b44c" containerName="init-config-reloader"
Jul 27 17:31:15.365837 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.365603    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0874ece-9e32-43fd-b926-98c8a3d7b44c" containerName="init-config-reloader"
Jul 27 17:31:15.365837 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.365644    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0874ece-9e32-43fd-b926-98c8a3d7b44c" containerName="kube-rbac-proxy-metric"
Jul 27 17:31:15.365837 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.365651    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0874ece-9e32-43fd-b926-98c8a3d7b44c" containerName="kube-rbac-proxy-web"
Jul 27 17:31:15.365837 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.365658    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0874ece-9e32-43fd-b926-98c8a3d7b44c" containerName="alertmanager"
Jul 27 17:31:15.365837 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.365664    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0874ece-9e32-43fd-b926-98c8a3d7b44c" containerName="kube-rbac-proxy"
Jul 27 17:31:15.365837 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.365670    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0874ece-9e32-43fd-b926-98c8a3d7b44c" containerName="config-reloader"
Jul 27 17:31:15.365837 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.365675    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0874ece-9e32-43fd-b926-98c8a3d7b44c" containerName="prom-label-proxy"
Jul 27 17:31:15.369317 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.369301    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:31:15.372211 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.372190    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 27 17:31:15.372375 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.372274    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 27 17:31:15.372375 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.372298    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-cpvdx\""
Jul 27 17:31:15.372525 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.372498    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 27 17:31:15.372576 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.372524    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 27 17:31:15.372701 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.372685    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 27 17:31:15.372750 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.372742    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 27 17:31:15.372811 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.372770    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 27 17:31:15.372984 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.372970    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 27 17:31:15.378135 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.378118    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 27 17:31:15.382920 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.382900    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 17:31:15.456903 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.456808    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/00a563aa-d5f9-4c94-86b9-9c013e1473b0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"00a563aa-d5f9-4c94-86b9-9c013e1473b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:31:15.456903 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.456845    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/00a563aa-d5f9-4c94-86b9-9c013e1473b0-config-volume\") pod \"alertmanager-main-0\" (UID: \"00a563aa-d5f9-4c94-86b9-9c013e1473b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:31:15.456903 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.456869    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/00a563aa-d5f9-4c94-86b9-9c013e1473b0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"00a563aa-d5f9-4c94-86b9-9c013e1473b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:31:15.457182 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.456975    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/00a563aa-d5f9-4c94-86b9-9c013e1473b0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"00a563aa-d5f9-4c94-86b9-9c013e1473b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:31:15.457182 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.457014    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/00a563aa-d5f9-4c94-86b9-9c013e1473b0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"00a563aa-d5f9-4c94-86b9-9c013e1473b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:31:15.457182 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.457066    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/00a563aa-d5f9-4c94-86b9-9c013e1473b0-web-config\") pod \"alertmanager-main-0\" (UID: \"00a563aa-d5f9-4c94-86b9-9c013e1473b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:31:15.457182 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.457093    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/00a563aa-d5f9-4c94-86b9-9c013e1473b0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"00a563aa-d5f9-4c94-86b9-9c013e1473b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:31:15.457182 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.457147    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6mv8\" (UniqueName: \"kubernetes.io/projected/00a563aa-d5f9-4c94-86b9-9c013e1473b0-kube-api-access-z6mv8\") pod \"alertmanager-main-0\" (UID: \"00a563aa-d5f9-4c94-86b9-9c013e1473b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:31:15.457429 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.457194    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/00a563aa-d5f9-4c94-86b9-9c013e1473b0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"00a563aa-d5f9-4c94-86b9-9c013e1473b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:31:15.457429 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.457213    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00a563aa-d5f9-4c94-86b9-9c013e1473b0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"00a563aa-d5f9-4c94-86b9-9c013e1473b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:31:15.457429 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.457237    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/00a563aa-d5f9-4c94-86b9-9c013e1473b0-config-out\") pod \"alertmanager-main-0\" (UID: \"00a563aa-d5f9-4c94-86b9-9c013e1473b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:31:15.457429 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.457271    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/00a563aa-d5f9-4c94-86b9-9c013e1473b0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"00a563aa-d5f9-4c94-86b9-9c013e1473b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:31:15.457429 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.457329    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/00a563aa-d5f9-4c94-86b9-9c013e1473b0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"00a563aa-d5f9-4c94-86b9-9c013e1473b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:31:15.558307 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.558233    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/00a563aa-d5f9-4c94-86b9-9c013e1473b0-config-out\") pod \"alertmanager-main-0\" (UID: \"00a563aa-d5f9-4c94-86b9-9c013e1473b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:31:15.558307 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.558307    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/00a563aa-d5f9-4c94-86b9-9c013e1473b0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"00a563aa-d5f9-4c94-86b9-9c013e1473b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:31:15.558576 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.558337    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/00a563aa-d5f9-4c94-86b9-9c013e1473b0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"00a563aa-d5f9-4c94-86b9-9c013e1473b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:31:15.558576 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.558382    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/00a563aa-d5f9-4c94-86b9-9c013e1473b0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"00a563aa-d5f9-4c94-86b9-9c013e1473b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:31:15.558576 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.558405    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/00a563aa-d5f9-4c94-86b9-9c013e1473b0-config-volume\") pod \"alertmanager-main-0\" (UID: \"00a563aa-d5f9-4c94-86b9-9c013e1473b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:31:15.558576 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.558435    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/00a563aa-d5f9-4c94-86b9-9c013e1473b0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"00a563aa-d5f9-4c94-86b9-9c013e1473b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:31:15.558576 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.558475    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/00a563aa-d5f9-4c94-86b9-9c013e1473b0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"00a563aa-d5f9-4c94-86b9-9c013e1473b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:31:15.558576 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.558507    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/00a563aa-d5f9-4c94-86b9-9c013e1473b0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"00a563aa-d5f9-4c94-86b9-9c013e1473b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:31:15.558576 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.558538    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/00a563aa-d5f9-4c94-86b9-9c013e1473b0-web-config\") pod \"alertmanager-main-0\" (UID: \"00a563aa-d5f9-4c94-86b9-9c013e1473b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:31:15.558576 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.558562    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/00a563aa-d5f9-4c94-86b9-9c013e1473b0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"00a563aa-d5f9-4c94-86b9-9c013e1473b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:31:15.558963 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.558589    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6mv8\" (UniqueName: \"kubernetes.io/projected/00a563aa-d5f9-4c94-86b9-9c013e1473b0-kube-api-access-z6mv8\") pod \"alertmanager-main-0\" (UID: \"00a563aa-d5f9-4c94-86b9-9c013e1473b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:31:15.558963 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.558632    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/00a563aa-d5f9-4c94-86b9-9c013e1473b0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"00a563aa-d5f9-4c94-86b9-9c013e1473b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:31:15.558963 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.558656    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00a563aa-d5f9-4c94-86b9-9c013e1473b0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"00a563aa-d5f9-4c94-86b9-9c013e1473b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:31:15.559891 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.559858    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/00a563aa-d5f9-4c94-86b9-9c013e1473b0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"00a563aa-d5f9-4c94-86b9-9c013e1473b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:31:15.561314 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.561282    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/00a563aa-d5f9-4c94-86b9-9c013e1473b0-config-out\") pod \"alertmanager-main-0\" (UID: \"00a563aa-d5f9-4c94-86b9-9c013e1473b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:31:15.561820 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.561459    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/00a563aa-d5f9-4c94-86b9-9c013e1473b0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"00a563aa-d5f9-4c94-86b9-9c013e1473b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:31:15.561820 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.561468    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/00a563aa-d5f9-4c94-86b9-9c013e1473b0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"00a563aa-d5f9-4c94-86b9-9c013e1473b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:31:15.561820 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.561599    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/00a563aa-d5f9-4c94-86b9-9c013e1473b0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"00a563aa-d5f9-4c94-86b9-9c013e1473b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:31:15.561820 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.561604    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/00a563aa-d5f9-4c94-86b9-9c013e1473b0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"00a563aa-d5f9-4c94-86b9-9c013e1473b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:31:15.561820 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.561680    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/00a563aa-d5f9-4c94-86b9-9c013e1473b0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"00a563aa-d5f9-4c94-86b9-9c013e1473b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:31:15.561820 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.561802    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/00a563aa-d5f9-4c94-86b9-9c013e1473b0-config-volume\") pod \"alertmanager-main-0\" (UID: \"00a563aa-d5f9-4c94-86b9-9c013e1473b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:31:15.562111 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.561801    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/00a563aa-d5f9-4c94-86b9-9c013e1473b0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"00a563aa-d5f9-4c94-86b9-9c013e1473b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:31:15.562111 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.562097    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/00a563aa-d5f9-4c94-86b9-9c013e1473b0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"00a563aa-d5f9-4c94-86b9-9c013e1473b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:31:15.562348 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.562331    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/00a563aa-d5f9-4c94-86b9-9c013e1473b0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"00a563aa-d5f9-4c94-86b9-9c013e1473b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:31:15.563330 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.563313    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/00a563aa-d5f9-4c94-86b9-9c013e1473b0-web-config\") pod \"alertmanager-main-0\" (UID: \"00a563aa-d5f9-4c94-86b9-9c013e1473b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:31:15.570124 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.570103    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6mv8\" (UniqueName: \"kubernetes.io/projected/00a563aa-d5f9-4c94-86b9-9c013e1473b0-kube-api-access-z6mv8\") pod \"alertmanager-main-0\" (UID: \"00a563aa-d5f9-4c94-86b9-9c013e1473b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:31:15.679069 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.679027    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 27 17:31:15.820704 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:15.820671    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 17:31:15.823680 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:31:15.823648    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00a563aa_d5f9_4c94_86b9_9c013e1473b0.slice/crio-2d53b607b2f57b3fc97f920e2925266b00e08619f3f7a2dce2f691fc5df4f723 WatchSource:0}: Error finding container 2d53b607b2f57b3fc97f920e2925266b00e08619f3f7a2dce2f691fc5df4f723: Status 404 returned error can't find the container with id 2d53b607b2f57b3fc97f920e2925266b00e08619f3f7a2dce2f691fc5df4f723
Jul 27 17:31:15.943133 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:31:15.943099    2573 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00a563aa_d5f9_4c94_86b9_9c013e1473b0.slice/crio-46b48c38ed2c7b0ed9da3c18d1bee69e65f025ca8b8b1d2812bc1745326e0b29.scope\": RecentStats: unable to find data in memory cache]"
Jul 27 17:31:15.943312 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:31:15.943099    2573 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00a563aa_d5f9_4c94_86b9_9c013e1473b0.slice/crio-conmon-46b48c38ed2c7b0ed9da3c18d1bee69e65f025ca8b8b1d2812bc1745326e0b29.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00a563aa_d5f9_4c94_86b9_9c013e1473b0.slice/crio-46b48c38ed2c7b0ed9da3c18d1bee69e65f025ca8b8b1d2812bc1745326e0b29.scope\": RecentStats: unable to find data in memory cache]"
Jul 27 17:31:16.312819 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:16.312782    2573 generic.go:356] "Generic (PLEG): container finished" podID="00a563aa-d5f9-4c94-86b9-9c013e1473b0" containerID="46b48c38ed2c7b0ed9da3c18d1bee69e65f025ca8b8b1d2812bc1745326e0b29" exitCode=0
Jul 27 17:31:16.313197 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:16.312841    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"00a563aa-d5f9-4c94-86b9-9c013e1473b0","Type":"ContainerDied","Data":"46b48c38ed2c7b0ed9da3c18d1bee69e65f025ca8b8b1d2812bc1745326e0b29"}
Jul 27 17:31:16.313197 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:16.312889    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"00a563aa-d5f9-4c94-86b9-9c013e1473b0","Type":"ContainerStarted","Data":"2d53b607b2f57b3fc97f920e2925266b00e08619f3f7a2dce2f691fc5df4f723"}
Jul 27 17:31:16.469733 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:16.469701    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0874ece-9e32-43fd-b926-98c8a3d7b44c" path="/var/lib/kubelet/pods/a0874ece-9e32-43fd-b926-98c8a3d7b44c/volumes"
Jul 27 17:31:17.320058 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:17.320020    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"00a563aa-d5f9-4c94-86b9-9c013e1473b0","Type":"ContainerStarted","Data":"625196b58e1e7ce531e3ffe4462fdc35cf92757101d495fea949ade182a8b486"}
Jul 27 17:31:17.320420 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:17.320102    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"00a563aa-d5f9-4c94-86b9-9c013e1473b0","Type":"ContainerStarted","Data":"7f2399d2ab595d89c0acf1b3000fc81600edcd07a79b18f6b961363c88493137"}
Jul 27 17:31:17.320420 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:17.320143    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"00a563aa-d5f9-4c94-86b9-9c013e1473b0","Type":"ContainerStarted","Data":"d431ee4af344c3260bec428fa70b98d24e08d8da12ef458cb8a66cac17cd5b42"}
Jul 27 17:31:17.320420 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:17.320185    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"00a563aa-d5f9-4c94-86b9-9c013e1473b0","Type":"ContainerStarted","Data":"148191114eab6e6d3f6e82f4d85760175b06e58ef4cb8cbffd2f0356585b6b9d"}
Jul 27 17:31:17.320420 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:17.320220    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"00a563aa-d5f9-4c94-86b9-9c013e1473b0","Type":"ContainerStarted","Data":"6be0f3a6402f9eb422ca863b2dc844772a775e6be75b4ff803cc52e7e5d71361"}
Jul 27 17:31:17.320420 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:17.320272    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"00a563aa-d5f9-4c94-86b9-9c013e1473b0","Type":"ContainerStarted","Data":"d2db3f852d5f4223cb249c7d4baa4d8b74cfa622351b89e7df2a75de04c639ec"}
Jul 27 17:31:17.350150 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:17.350095    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.333051986 podStartE2EDuration="2.350074672s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.022686ms" isStatefulPod=false podCreationTimestamp="2026-07-27 17:31:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:31:17.347584225 +0000 UTC m=+219.490986300" watchObservedRunningTime="2026-07-27 17:31:17.350074672 +0000 UTC m=+219.493476722"
Jul 27 17:31:18.207666 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:18.207633    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:31:18.276962 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:18.276928    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:31:22.798034 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:22.797999    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-77ddc7856-pzqmk"]
Jul 27 17:31:22.800224 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:22.800205    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77ddc7856-pzqmk"
Jul 27 17:31:22.811150 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:22.811123    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77ddc7856-pzqmk"]
Jul 27 17:31:22.863243 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:22.863205    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:31:22.917926 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:22.917887    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66a0649f-1102-4dd9-9b70-49d34f92c512-console-oauth-config\") pod \"console-77ddc7856-pzqmk\" (UID: \"66a0649f-1102-4dd9-9b70-49d34f92c512\") " pod="openshift-console/console-77ddc7856-pzqmk"
Jul 27 17:31:22.917926 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:22.917927    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66a0649f-1102-4dd9-9b70-49d34f92c512-service-ca\") pod \"console-77ddc7856-pzqmk\" (UID: \"66a0649f-1102-4dd9-9b70-49d34f92c512\") " pod="openshift-console/console-77ddc7856-pzqmk"
Jul 27 17:31:22.918168 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:22.917943    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r59w\" (UniqueName: \"kubernetes.io/projected/66a0649f-1102-4dd9-9b70-49d34f92c512-kube-api-access-9r59w\") pod \"console-77ddc7856-pzqmk\" (UID: \"66a0649f-1102-4dd9-9b70-49d34f92c512\") " pod="openshift-console/console-77ddc7856-pzqmk"
Jul 27 17:31:22.918168 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:22.918036    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66a0649f-1102-4dd9-9b70-49d34f92c512-oauth-serving-cert\") pod \"console-77ddc7856-pzqmk\" (UID: \"66a0649f-1102-4dd9-9b70-49d34f92c512\") " pod="openshift-console/console-77ddc7856-pzqmk"
Jul 27 17:31:22.918168 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:22.918079    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66a0649f-1102-4dd9-9b70-49d34f92c512-console-serving-cert\") pod \"console-77ddc7856-pzqmk\" (UID: \"66a0649f-1102-4dd9-9b70-49d34f92c512\") " pod="openshift-console/console-77ddc7856-pzqmk"
Jul 27 17:31:22.918168 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:22.918106    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66a0649f-1102-4dd9-9b70-49d34f92c512-trusted-ca-bundle\") pod \"console-77ddc7856-pzqmk\" (UID: \"66a0649f-1102-4dd9-9b70-49d34f92c512\") " pod="openshift-console/console-77ddc7856-pzqmk"
Jul 27 17:31:22.918168 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:22.918157    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66a0649f-1102-4dd9-9b70-49d34f92c512-console-config\") pod \"console-77ddc7856-pzqmk\" (UID: \"66a0649f-1102-4dd9-9b70-49d34f92c512\") " pod="openshift-console/console-77ddc7856-pzqmk"
Jul 27 17:31:22.920561 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:22.920532    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:31:22.999681 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:22.999650    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:31:23.018731 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:23.018706    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66a0649f-1102-4dd9-9b70-49d34f92c512-console-oauth-config\") pod \"console-77ddc7856-pzqmk\" (UID: \"66a0649f-1102-4dd9-9b70-49d34f92c512\") " pod="openshift-console/console-77ddc7856-pzqmk"
Jul 27 17:31:23.018812 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:23.018737    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66a0649f-1102-4dd9-9b70-49d34f92c512-service-ca\") pod \"console-77ddc7856-pzqmk\" (UID: \"66a0649f-1102-4dd9-9b70-49d34f92c512\") " pod="openshift-console/console-77ddc7856-pzqmk"
Jul 27 17:31:23.018812 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:23.018756    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r59w\" (UniqueName: \"kubernetes.io/projected/66a0649f-1102-4dd9-9b70-49d34f92c512-kube-api-access-9r59w\") pod \"console-77ddc7856-pzqmk\" (UID: \"66a0649f-1102-4dd9-9b70-49d34f92c512\") " pod="openshift-console/console-77ddc7856-pzqmk"
Jul 27 17:31:23.018812 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:23.018781    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66a0649f-1102-4dd9-9b70-49d34f92c512-oauth-serving-cert\") pod \"console-77ddc7856-pzqmk\" (UID: \"66a0649f-1102-4dd9-9b70-49d34f92c512\") " pod="openshift-console/console-77ddc7856-pzqmk"
Jul 27 17:31:23.018812 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:23.018805    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66a0649f-1102-4dd9-9b70-49d34f92c512-console-serving-cert\") pod \"console-77ddc7856-pzqmk\" (UID: \"66a0649f-1102-4dd9-9b70-49d34f92c512\") " pod="openshift-console/console-77ddc7856-pzqmk"
Jul 27 17:31:23.019010 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:23.018827    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66a0649f-1102-4dd9-9b70-49d34f92c512-trusted-ca-bundle\") pod \"console-77ddc7856-pzqmk\" (UID: \"66a0649f-1102-4dd9-9b70-49d34f92c512\") " pod="openshift-console/console-77ddc7856-pzqmk"
Jul 27 17:31:23.019010 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:23.018868    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66a0649f-1102-4dd9-9b70-49d34f92c512-console-config\") pod \"console-77ddc7856-pzqmk\" (UID: \"66a0649f-1102-4dd9-9b70-49d34f92c512\") " pod="openshift-console/console-77ddc7856-pzqmk"
Jul 27 17:31:23.019673 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:23.019496    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66a0649f-1102-4dd9-9b70-49d34f92c512-service-ca\") pod \"console-77ddc7856-pzqmk\" (UID: \"66a0649f-1102-4dd9-9b70-49d34f92c512\") " pod="openshift-console/console-77ddc7856-pzqmk"
Jul 27 17:31:23.019673 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:23.019567    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66a0649f-1102-4dd9-9b70-49d34f92c512-console-config\") pod \"console-77ddc7856-pzqmk\" (UID: \"66a0649f-1102-4dd9-9b70-49d34f92c512\") " pod="openshift-console/console-77ddc7856-pzqmk"
Jul 27 17:31:23.019673 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:23.019625    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66a0649f-1102-4dd9-9b70-49d34f92c512-oauth-serving-cert\") pod \"console-77ddc7856-pzqmk\" (UID: \"66a0649f-1102-4dd9-9b70-49d34f92c512\") " pod="openshift-console/console-77ddc7856-pzqmk"
Jul 27 17:31:23.019903 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:23.019787    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66a0649f-1102-4dd9-9b70-49d34f92c512-trusted-ca-bundle\") pod \"console-77ddc7856-pzqmk\" (UID: \"66a0649f-1102-4dd9-9b70-49d34f92c512\") " pod="openshift-console/console-77ddc7856-pzqmk"
Jul 27 17:31:23.021182 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:23.021160    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66a0649f-1102-4dd9-9b70-49d34f92c512-console-oauth-config\") pod \"console-77ddc7856-pzqmk\" (UID: \"66a0649f-1102-4dd9-9b70-49d34f92c512\") " pod="openshift-console/console-77ddc7856-pzqmk"
Jul 27 17:31:23.021355 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:23.021338    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66a0649f-1102-4dd9-9b70-49d34f92c512-console-serving-cert\") pod \"console-77ddc7856-pzqmk\" (UID: \"66a0649f-1102-4dd9-9b70-49d34f92c512\") " pod="openshift-console/console-77ddc7856-pzqmk"
Jul 27 17:31:23.027353 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:23.027328    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r59w\" (UniqueName: \"kubernetes.io/projected/66a0649f-1102-4dd9-9b70-49d34f92c512-kube-api-access-9r59w\") pod \"console-77ddc7856-pzqmk\" (UID: \"66a0649f-1102-4dd9-9b70-49d34f92c512\") " pod="openshift-console/console-77ddc7856-pzqmk"
Jul 27 17:31:23.058753 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:23.058671    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:31:23.110742 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:23.110708    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77ddc7856-pzqmk"
Jul 27 17:31:23.245148 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:23.245072    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77ddc7856-pzqmk"]
Jul 27 17:31:23.247424 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:31:23.247392    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66a0649f_1102_4dd9_9b70_49d34f92c512.slice/crio-3366a002287d1011e0eb874997df99fb30a0cc341da55794bb0f1b6e41b67eba WatchSource:0}: Error finding container 3366a002287d1011e0eb874997df99fb30a0cc341da55794bb0f1b6e41b67eba: Status 404 returned error can't find the container with id 3366a002287d1011e0eb874997df99fb30a0cc341da55794bb0f1b6e41b67eba
Jul 27 17:31:23.345582 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:23.345217    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77ddc7856-pzqmk" event={"ID":"66a0649f-1102-4dd9-9b70-49d34f92c512","Type":"ContainerStarted","Data":"bce03b917fb06840178e708dd738ad226f476f8828f9dd6ae8e598b5c4edb532"}
Jul 27 17:31:23.345582 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:23.345304    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77ddc7856-pzqmk" event={"ID":"66a0649f-1102-4dd9-9b70-49d34f92c512","Type":"ContainerStarted","Data":"3366a002287d1011e0eb874997df99fb30a0cc341da55794bb0f1b6e41b67eba"}
Jul 27 17:31:23.380425 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:23.376081    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-77ddc7856-pzqmk" podStartSLOduration=1.3760574540000001 podStartE2EDuration="1.376057454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:31:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:31:23.375286113 +0000 UTC m=+225.518688164" watchObservedRunningTime="2026-07-27 17:31:23.376057454 +0000 UTC m=+225.519459507"
Jul 27 17:31:23.597742 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:23.597660    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:31:23.655434 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:23.655400    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:31:25.101326 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:25.101294    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:31:25.202432 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:25.202370    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:31:33.111516 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:33.111475    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-77ddc7856-pzqmk"
Jul 27 17:31:33.111968 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:33.111534    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-77ddc7856-pzqmk"
Jul 27 17:31:33.116674 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:33.116650    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-77ddc7856-pzqmk"
Jul 27 17:31:33.389601 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:33.389574    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-77ddc7856-pzqmk"
Jul 27 17:31:33.444374 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:33.444342    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59d7d5f8d8-vxsmf"]
Jul 27 17:31:33.562803 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:33.562760    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:31:33.662319 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:33.662218    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:31:33.716402 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:33.716368    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:31:53.609861 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:53.609821    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:31:53.711351 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:53.711321    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:31:58.463377 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:58.463335    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-59d7d5f8d8-vxsmf" podUID="ce271c8e-e982-4e3e-8588-26b79ccc6488" containerName="console" containerID="cri-o://362811b07c4dc85154d11f4fea315745e2e161dc3d1e80429717bc62758c39dc" gracePeriod=15
Jul 27 17:31:58.716991 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:58.716923    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59d7d5f8d8-vxsmf"
Jul 27 17:31:58.822178 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:58.822135    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ce271c8e-e982-4e3e-8588-26b79ccc6488-console-config\") pod \"ce271c8e-e982-4e3e-8588-26b79ccc6488\" (UID: \"ce271c8e-e982-4e3e-8588-26b79ccc6488\") "
Jul 27 17:31:58.822369 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:58.822208    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ce271c8e-e982-4e3e-8588-26b79ccc6488-oauth-serving-cert\") pod \"ce271c8e-e982-4e3e-8588-26b79ccc6488\" (UID: \"ce271c8e-e982-4e3e-8588-26b79ccc6488\") "
Jul 27 17:31:58.822369 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:58.822238    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-894dr\" (UniqueName: \"kubernetes.io/projected/ce271c8e-e982-4e3e-8588-26b79ccc6488-kube-api-access-894dr\") pod \"ce271c8e-e982-4e3e-8588-26b79ccc6488\" (UID: \"ce271c8e-e982-4e3e-8588-26b79ccc6488\") "
Jul 27 17:31:58.822369 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:58.822287    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce271c8e-e982-4e3e-8588-26b79ccc6488-console-serving-cert\") pod \"ce271c8e-e982-4e3e-8588-26b79ccc6488\" (UID: \"ce271c8e-e982-4e3e-8588-26b79ccc6488\") "
Jul 27 17:31:58.822369 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:58.822338    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ce271c8e-e982-4e3e-8588-26b79ccc6488-service-ca\") pod \"ce271c8e-e982-4e3e-8588-26b79ccc6488\" (UID: \"ce271c8e-e982-4e3e-8588-26b79ccc6488\") "
Jul 27 17:31:58.822369 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:58.822365    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce271c8e-e982-4e3e-8588-26b79ccc6488-trusted-ca-bundle\") pod \"ce271c8e-e982-4e3e-8588-26b79ccc6488\" (UID: \"ce271c8e-e982-4e3e-8588-26b79ccc6488\") "
Jul 27 17:31:58.822634 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:58.822404    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ce271c8e-e982-4e3e-8588-26b79ccc6488-console-oauth-config\") pod \"ce271c8e-e982-4e3e-8588-26b79ccc6488\" (UID: \"ce271c8e-e982-4e3e-8588-26b79ccc6488\") "
Jul 27 17:31:58.822725 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:58.822694    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce271c8e-e982-4e3e-8588-26b79ccc6488-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ce271c8e-e982-4e3e-8588-26b79ccc6488" (UID: "ce271c8e-e982-4e3e-8588-26b79ccc6488"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:31:58.822788 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:58.822720    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce271c8e-e982-4e3e-8588-26b79ccc6488-console-config" (OuterVolumeSpecName: "console-config") pod "ce271c8e-e982-4e3e-8588-26b79ccc6488" (UID: "ce271c8e-e982-4e3e-8588-26b79ccc6488"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:31:58.822847 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:58.822797    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce271c8e-e982-4e3e-8588-26b79ccc6488-service-ca" (OuterVolumeSpecName: "service-ca") pod "ce271c8e-e982-4e3e-8588-26b79ccc6488" (UID: "ce271c8e-e982-4e3e-8588-26b79ccc6488"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:31:58.822902 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:58.822841    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce271c8e-e982-4e3e-8588-26b79ccc6488-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ce271c8e-e982-4e3e-8588-26b79ccc6488" (UID: "ce271c8e-e982-4e3e-8588-26b79ccc6488"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:31:58.824670 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:58.824639    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce271c8e-e982-4e3e-8588-26b79ccc6488-kube-api-access-894dr" (OuterVolumeSpecName: "kube-api-access-894dr") pod "ce271c8e-e982-4e3e-8588-26b79ccc6488" (UID: "ce271c8e-e982-4e3e-8588-26b79ccc6488"). InnerVolumeSpecName "kube-api-access-894dr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:31:58.824775 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:58.824682    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce271c8e-e982-4e3e-8588-26b79ccc6488-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ce271c8e-e982-4e3e-8588-26b79ccc6488" (UID: "ce271c8e-e982-4e3e-8588-26b79ccc6488"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:31:58.824775 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:58.824698    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce271c8e-e982-4e3e-8588-26b79ccc6488-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ce271c8e-e982-4e3e-8588-26b79ccc6488" (UID: "ce271c8e-e982-4e3e-8588-26b79ccc6488"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:31:58.923502 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:58.923468    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-894dr\" (UniqueName: \"kubernetes.io/projected/ce271c8e-e982-4e3e-8588-26b79ccc6488-kube-api-access-894dr\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:31:58.923502 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:58.923495    2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce271c8e-e982-4e3e-8588-26b79ccc6488-console-serving-cert\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:31:58.923502 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:58.923505    2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ce271c8e-e982-4e3e-8588-26b79ccc6488-service-ca\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:31:58.923502 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:58.923513    2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce271c8e-e982-4e3e-8588-26b79ccc6488-trusted-ca-bundle\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:31:58.923755 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:58.923521    2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ce271c8e-e982-4e3e-8588-26b79ccc6488-console-oauth-config\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:31:58.923755 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:58.923529    2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ce271c8e-e982-4e3e-8588-26b79ccc6488-console-config\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:31:58.923755 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:58.923537    2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ce271c8e-e982-4e3e-8588-26b79ccc6488-oauth-serving-cert\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:31:59.484677 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:59.484639    2573 generic.go:356] "Generic (PLEG): container finished" podID="ce271c8e-e982-4e3e-8588-26b79ccc6488" containerID="362811b07c4dc85154d11f4fea315745e2e161dc3d1e80429717bc62758c39dc" exitCode=0
Jul 27 17:31:59.485088 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:59.484697    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59d7d5f8d8-vxsmf"
Jul 27 17:31:59.485088 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:59.484717    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59d7d5f8d8-vxsmf" event={"ID":"ce271c8e-e982-4e3e-8588-26b79ccc6488","Type":"ContainerDied","Data":"362811b07c4dc85154d11f4fea315745e2e161dc3d1e80429717bc62758c39dc"}
Jul 27 17:31:59.485088 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:59.484776    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59d7d5f8d8-vxsmf" event={"ID":"ce271c8e-e982-4e3e-8588-26b79ccc6488","Type":"ContainerDied","Data":"d4d4b0a2548bc955dfd4b52715aaad71f1615f2bc82a83ffb3a1a580ed745c25"}
Jul 27 17:31:59.485088 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:59.484797    2573 scope.go:117] "RemoveContainer" containerID="362811b07c4dc85154d11f4fea315745e2e161dc3d1e80429717bc62758c39dc"
Jul 27 17:31:59.493299 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:59.493278    2573 scope.go:117] "RemoveContainer" containerID="362811b07c4dc85154d11f4fea315745e2e161dc3d1e80429717bc62758c39dc"
Jul 27 17:31:59.493533 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:31:59.493511    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"362811b07c4dc85154d11f4fea315745e2e161dc3d1e80429717bc62758c39dc\": container with ID starting with 362811b07c4dc85154d11f4fea315745e2e161dc3d1e80429717bc62758c39dc not found: ID does not exist" containerID="362811b07c4dc85154d11f4fea315745e2e161dc3d1e80429717bc62758c39dc"
Jul 27 17:31:59.493603 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:59.493546    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"362811b07c4dc85154d11f4fea315745e2e161dc3d1e80429717bc62758c39dc"} err="failed to get container status \"362811b07c4dc85154d11f4fea315745e2e161dc3d1e80429717bc62758c39dc\": rpc error: code = NotFound desc = could not find container \"362811b07c4dc85154d11f4fea315745e2e161dc3d1e80429717bc62758c39dc\": container with ID starting with 362811b07c4dc85154d11f4fea315745e2e161dc3d1e80429717bc62758c39dc not found: ID does not exist"
Jul 27 17:31:59.508406 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:59.508379    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59d7d5f8d8-vxsmf"]
Jul 27 17:31:59.511858 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:31:59.511834    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-59d7d5f8d8-vxsmf"]
Jul 27 17:32:00.468469 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:00.468439    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce271c8e-e982-4e3e-8588-26b79ccc6488" path="/var/lib/kubelet/pods/ce271c8e-e982-4e3e-8588-26b79ccc6488/volumes"
Jul 27 17:32:06.190272 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:06.190229    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:32:06.249765 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:06.249735    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:32:14.838804 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:14.838762    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:32:14.940826 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:14.940784    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:32:15.187060 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:15.187020    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:32:15.307517 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:15.307485    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:32:15.391875 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:15.391841    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:32:37.382365 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:37.382327    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:32:37.479012 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:37.478978    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:32:38.249831 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:38.249800    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vc2tz_87dac797-f8d2-4d36-999f-f8ce855dbeff/ovn-acl-logging/0.log"
Jul 27 17:32:38.253649 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:38.253621    2573 kubelet.go:1648] "Image garbage collection succeeded"
Jul 27 17:32:38.256840 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:38.256822    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vc2tz_87dac797-f8d2-4d36-999f-f8ce855dbeff/ovn-acl-logging/0.log"
Jul 27 17:32:38.876415 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:38.876381    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-jfxd6"]
Jul 27 17:32:38.878572 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:38.876697    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce271c8e-e982-4e3e-8588-26b79ccc6488" containerName="console"
Jul 27 17:32:38.878572 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:38.876712    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce271c8e-e982-4e3e-8588-26b79ccc6488" containerName="console"
Jul 27 17:32:38.878572 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:38.876767    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce271c8e-e982-4e3e-8588-26b79ccc6488" containerName="console"
Jul 27 17:32:38.879585 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:38.879570    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-jfxd6"
Jul 27 17:32:38.885065 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:38.885036    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-kkzll\""
Jul 27 17:32:38.885206 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:38.885040    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 27 17:32:38.885206 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:38.885040    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 27 17:32:38.885324 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:38.885219    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 27 17:32:38.904275 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:38.903147    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-jfxd6"]
Jul 27 17:32:39.062356 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:39.062315    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/eb2b15e7-dae4-4e54-a06f-83d5ffdb081a-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-jfxd6\" (UID: \"eb2b15e7-dae4-4e54-a06f-83d5ffdb081a\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-jfxd6"
Jul 27 17:32:39.062552 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:39.062440    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rfr6\" (UniqueName: \"kubernetes.io/projected/eb2b15e7-dae4-4e54-a06f-83d5ffdb081a-kube-api-access-9rfr6\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-jfxd6\" (UID: \"eb2b15e7-dae4-4e54-a06f-83d5ffdb081a\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-jfxd6"
Jul 27 17:32:39.163079 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:39.163048    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rfr6\" (UniqueName: \"kubernetes.io/projected/eb2b15e7-dae4-4e54-a06f-83d5ffdb081a-kube-api-access-9rfr6\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-jfxd6\" (UID: \"eb2b15e7-dae4-4e54-a06f-83d5ffdb081a\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-jfxd6"
Jul 27 17:32:39.163238 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:39.163089    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/eb2b15e7-dae4-4e54-a06f-83d5ffdb081a-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-jfxd6\" (UID: \"eb2b15e7-dae4-4e54-a06f-83d5ffdb081a\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-jfxd6"
Jul 27 17:32:39.165488 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:39.165465    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/eb2b15e7-dae4-4e54-a06f-83d5ffdb081a-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-jfxd6\" (UID: \"eb2b15e7-dae4-4e54-a06f-83d5ffdb081a\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-jfxd6"
Jul 27 17:32:39.172718 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:39.172694    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rfr6\" (UniqueName: \"kubernetes.io/projected/eb2b15e7-dae4-4e54-a06f-83d5ffdb081a-kube-api-access-9rfr6\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-jfxd6\" (UID: \"eb2b15e7-dae4-4e54-a06f-83d5ffdb081a\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-jfxd6"
Jul 27 17:32:39.189635 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:39.189609    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-jfxd6"
Jul 27 17:32:39.331353 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:39.331321    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-jfxd6"]
Jul 27 17:32:39.335077 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:32:39.335048    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb2b15e7_dae4_4e54_a06f_83d5ffdb081a.slice/crio-d5b4ab27851966e832ad88ecca84c6bb84b4bcfc6437545c28431ba14e98ace7 WatchSource:0}: Error finding container d5b4ab27851966e832ad88ecca84c6bb84b4bcfc6437545c28431ba14e98ace7: Status 404 returned error can't find the container with id d5b4ab27851966e832ad88ecca84c6bb84b4bcfc6437545c28431ba14e98ace7
Jul 27 17:32:39.469242 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:39.469140    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:32:39.642944 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:39.642907    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-jfxd6" event={"ID":"eb2b15e7-dae4-4e54-a06f-83d5ffdb081a","Type":"ContainerStarted","Data":"d5b4ab27851966e832ad88ecca84c6bb84b4bcfc6437545c28431ba14e98ace7"}
Jul 27 17:32:43.448069 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:43.448034    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:32:44.950314 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:44.950260    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:32:48.683393 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:48.683297    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-jfxd6" event={"ID":"eb2b15e7-dae4-4e54-a06f-83d5ffdb081a","Type":"ContainerStarted","Data":"f30d1ed9e1ed96964dc8410384fca0afaccbc3aacc6b0acd8518f7cdce20d3ef"}
Jul 27 17:32:48.683772 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:48.683397    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-jfxd6"
Jul 27 17:32:48.709204 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:48.709154    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-jfxd6" podStartSLOduration=1.641177244 podStartE2EDuration="10.709137622s" totalImagesPullingTime="9.067960378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:32:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:32:48.708087632 +0000 UTC m=+310.851489714" watchObservedRunningTime="2026-07-27 17:32:48.709137622 +0000 UTC m=+310.852539673"
Jul 27 17:32:48.806095 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:48.806052    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:32:49.088605 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:49.088525    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-77f648d694-qv6qv"]
Jul 27 17:32:49.090619 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:49.090587    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:32:49.091884 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:49.091867    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-qv6qv"
Jul 27 17:32:49.094692 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:49.094671    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 27 17:32:49.094810 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:49.094675    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 27 17:32:49.094959 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:49.094944    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-w92c5\""
Jul 27 17:32:49.100724 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:49.100701    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-qv6qv"]
Jul 27 17:32:49.133379 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:49.133342    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f064dfdb-f134-4da8-969e-c497a8cdd3a4-certificates\") pod \"keda-operator-77f648d694-qv6qv\" (UID: \"f064dfdb-f134-4da8-969e-c497a8cdd3a4\") " pod="openshift-keda/keda-operator-77f648d694-qv6qv"
Jul 27 17:32:49.133555 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:49.133401    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl7pj\" (UniqueName: \"kubernetes.io/projected/f064dfdb-f134-4da8-969e-c497a8cdd3a4-kube-api-access-sl7pj\") pod \"keda-operator-77f648d694-qv6qv\" (UID: \"f064dfdb-f134-4da8-969e-c497a8cdd3a4\") " pod="openshift-keda/keda-operator-77f648d694-qv6qv"
Jul 27 17:32:49.133555 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:49.133427    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/f064dfdb-f134-4da8-969e-c497a8cdd3a4-cabundle0\") pod \"keda-operator-77f648d694-qv6qv\" (UID: \"f064dfdb-f134-4da8-969e-c497a8cdd3a4\") " pod="openshift-keda/keda-operator-77f648d694-qv6qv"
Jul 27 17:32:49.234952 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:49.234918    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl7pj\" (UniqueName: \"kubernetes.io/projected/f064dfdb-f134-4da8-969e-c497a8cdd3a4-kube-api-access-sl7pj\") pod \"keda-operator-77f648d694-qv6qv\" (UID: \"f064dfdb-f134-4da8-969e-c497a8cdd3a4\") " pod="openshift-keda/keda-operator-77f648d694-qv6qv"
Jul 27 17:32:49.235220 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:49.235196    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/f064dfdb-f134-4da8-969e-c497a8cdd3a4-cabundle0\") pod \"keda-operator-77f648d694-qv6qv\" (UID: \"f064dfdb-f134-4da8-969e-c497a8cdd3a4\") " pod="openshift-keda/keda-operator-77f648d694-qv6qv"
Jul 27 17:32:49.235405 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:49.235280    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f064dfdb-f134-4da8-969e-c497a8cdd3a4-certificates\") pod \"keda-operator-77f648d694-qv6qv\" (UID: \"f064dfdb-f134-4da8-969e-c497a8cdd3a4\") " pod="openshift-keda/keda-operator-77f648d694-qv6qv"
Jul 27 17:32:49.235476 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:32:49.235443    2573 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Jul 27 17:32:49.235476 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:32:49.235469    2573 secret.go:281] references non-existent secret key: ca.crt
Jul 27 17:32:49.235577 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:32:49.235482    2573 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 27 17:32:49.235577 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:32:49.235499    2573 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-qv6qv: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 27 17:32:49.235577 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:32:49.235566    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f064dfdb-f134-4da8-969e-c497a8cdd3a4-certificates podName:f064dfdb-f134-4da8-969e-c497a8cdd3a4 nodeName:}" failed. No retries permitted until 2026-07-27 17:32:49.735545522 +0000 UTC m=+311.878947552 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/f064dfdb-f134-4da8-969e-c497a8cdd3a4-certificates") pod "keda-operator-77f648d694-qv6qv" (UID: "f064dfdb-f134-4da8-969e-c497a8cdd3a4") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 27 17:32:49.235889 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:49.235871    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/f064dfdb-f134-4da8-969e-c497a8cdd3a4-cabundle0\") pod \"keda-operator-77f648d694-qv6qv\" (UID: \"f064dfdb-f134-4da8-969e-c497a8cdd3a4\") " pod="openshift-keda/keda-operator-77f648d694-qv6qv"
Jul 27 17:32:49.245901 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:49.245875    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl7pj\" (UniqueName: \"kubernetes.io/projected/f064dfdb-f134-4da8-969e-c497a8cdd3a4-kube-api-access-sl7pj\") pod \"keda-operator-77f648d694-qv6qv\" (UID: \"f064dfdb-f134-4da8-969e-c497a8cdd3a4\") " pod="openshift-keda/keda-operator-77f648d694-qv6qv"
Jul 27 17:32:49.740099 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:49.740066    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f064dfdb-f134-4da8-969e-c497a8cdd3a4-certificates\") pod \"keda-operator-77f648d694-qv6qv\" (UID: \"f064dfdb-f134-4da8-969e-c497a8cdd3a4\") " pod="openshift-keda/keda-operator-77f648d694-qv6qv"
Jul 27 17:32:49.740506 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:32:49.740190    2573 secret.go:281] references non-existent secret key: ca.crt
Jul 27 17:32:49.740506 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:32:49.740213    2573 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 27 17:32:49.740506 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:32:49.740222    2573 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-qv6qv: references non-existent secret key: ca.crt
Jul 27 17:32:49.740506 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:32:49.740289    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f064dfdb-f134-4da8-969e-c497a8cdd3a4-certificates podName:f064dfdb-f134-4da8-969e-c497a8cdd3a4 nodeName:}" failed. No retries permitted until 2026-07-27 17:32:50.740274927 +0000 UTC m=+312.883676959 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/f064dfdb-f134-4da8-969e-c497a8cdd3a4-certificates") pod "keda-operator-77f648d694-qv6qv" (UID: "f064dfdb-f134-4da8-969e-c497a8cdd3a4") : references non-existent secret key: ca.crt
Jul 27 17:32:50.749321 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:50.749277    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f064dfdb-f134-4da8-969e-c497a8cdd3a4-certificates\") pod \"keda-operator-77f648d694-qv6qv\" (UID: \"f064dfdb-f134-4da8-969e-c497a8cdd3a4\") " pod="openshift-keda/keda-operator-77f648d694-qv6qv"
Jul 27 17:32:50.751830 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:50.751803    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f064dfdb-f134-4da8-969e-c497a8cdd3a4-certificates\") pod \"keda-operator-77f648d694-qv6qv\" (UID: \"f064dfdb-f134-4da8-969e-c497a8cdd3a4\") " pod="openshift-keda/keda-operator-77f648d694-qv6qv"
Jul 27 17:32:50.835618 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:50.835572    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:32:50.903194 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:50.903162    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-qv6qv"
Jul 27 17:32:50.903647 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:50.903619    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:32:51.035494 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:51.035453    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-qv6qv"]
Jul 27 17:32:51.037896 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:32:51.037868    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf064dfdb_f134_4da8_969e_c497a8cdd3a4.slice/crio-7ce0c0a1c07973bfb49c28b937fe3918d4c51097dc678065e30c23c5e8765d3b WatchSource:0}: Error finding container 7ce0c0a1c07973bfb49c28b937fe3918d4c51097dc678065e30c23c5e8765d3b: Status 404 returned error can't find the container with id 7ce0c0a1c07973bfb49c28b937fe3918d4c51097dc678065e30c23c5e8765d3b
Jul 27 17:32:51.039445 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:51.039425    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 17:32:51.696360 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:51.696322    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-qv6qv" event={"ID":"f064dfdb-f134-4da8-969e-c497a8cdd3a4","Type":"ContainerStarted","Data":"7ce0c0a1c07973bfb49c28b937fe3918d4c51097dc678065e30c23c5e8765d3b"}
Jul 27 17:32:54.711668 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:54.711629    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-qv6qv" event={"ID":"f064dfdb-f134-4da8-969e-c497a8cdd3a4","Type":"ContainerStarted","Data":"ea00fadb1c4f695fda632d409823a2a1fdf20f8e86edc6be0c2e61545a36d1fc"}
Jul 27 17:32:54.712120 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:54.711762    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-77f648d694-qv6qv"
Jul 27 17:32:54.731040 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:32:54.730990    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-77f648d694-qv6qv" podStartSLOduration=2.53747919 podStartE2EDuration="5.730974242s" totalImagesPullingTime="3.193495052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:32:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:32:54.729557487 +0000 UTC m=+316.872959538" watchObservedRunningTime="2026-07-27 17:32:54.730974242 +0000 UTC m=+316.874376292"
Jul 27 17:33:09.690243 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:33:09.690213    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-jfxd6"
Jul 27 17:33:15.718670 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:33:15.718639    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-77f648d694-qv6qv"
Jul 27 17:34:06.606328 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:34:06.606200    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:34:06.663072 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:34:06.663036    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-84fc959947-j9x2v"]
Jul 27 17:34:06.665462 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:34:06.665439    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-84fc959947-j9x2v"
Jul 27 17:34:06.668318 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:34:06.668294    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 27 17:34:06.668473 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:34:06.668294    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 27 17:34:06.668473 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:34:06.668369    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 27 17:34:06.669659 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:34:06.669638    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-xm76g\""
Jul 27 17:34:06.676123 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:34:06.676101    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-84fc959947-j9x2v"]
Jul 27 17:34:06.796599 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:34:06.796558    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/492d3c37-e7bb-4598-84b8-544d13e956ba-cert\") pod \"llmisvc-controller-manager-84fc959947-j9x2v\" (UID: \"492d3c37-e7bb-4598-84b8-544d13e956ba\") " pod="kserve/llmisvc-controller-manager-84fc959947-j9x2v"
Jul 27 17:34:06.796775 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:34:06.796614    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9g5z\" (UniqueName: \"kubernetes.io/projected/492d3c37-e7bb-4598-84b8-544d13e956ba-kube-api-access-m9g5z\") pod \"llmisvc-controller-manager-84fc959947-j9x2v\" (UID: \"492d3c37-e7bb-4598-84b8-544d13e956ba\") " pod="kserve/llmisvc-controller-manager-84fc959947-j9x2v"
Jul 27 17:34:06.897353 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:34:06.897310    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9g5z\" (UniqueName: \"kubernetes.io/projected/492d3c37-e7bb-4598-84b8-544d13e956ba-kube-api-access-m9g5z\") pod \"llmisvc-controller-manager-84fc959947-j9x2v\" (UID: \"492d3c37-e7bb-4598-84b8-544d13e956ba\") " pod="kserve/llmisvc-controller-manager-84fc959947-j9x2v"
Jul 27 17:34:06.897520 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:34:06.897401    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/492d3c37-e7bb-4598-84b8-544d13e956ba-cert\") pod \"llmisvc-controller-manager-84fc959947-j9x2v\" (UID: \"492d3c37-e7bb-4598-84b8-544d13e956ba\") " pod="kserve/llmisvc-controller-manager-84fc959947-j9x2v"
Jul 27 17:34:06.899835 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:34:06.899807    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/492d3c37-e7bb-4598-84b8-544d13e956ba-cert\") pod \"llmisvc-controller-manager-84fc959947-j9x2v\" (UID: \"492d3c37-e7bb-4598-84b8-544d13e956ba\") " pod="kserve/llmisvc-controller-manager-84fc959947-j9x2v"
Jul 27 17:34:06.906540 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:34:06.906511    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9g5z\" (UniqueName: \"kubernetes.io/projected/492d3c37-e7bb-4598-84b8-544d13e956ba-kube-api-access-m9g5z\") pod \"llmisvc-controller-manager-84fc959947-j9x2v\" (UID: \"492d3c37-e7bb-4598-84b8-544d13e956ba\") " pod="kserve/llmisvc-controller-manager-84fc959947-j9x2v"
Jul 27 17:34:06.976636 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:34:06.976600    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-84fc959947-j9x2v"
Jul 27 17:34:07.111233 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:34:07.111200    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-84fc959947-j9x2v"]
Jul 27 17:34:07.114336 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:34:07.114307    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod492d3c37_e7bb_4598_84b8_544d13e956ba.slice/crio-aba17380151b37513202711b44ebdea99baf57501a89d601e4a5a4ef86fecc26 WatchSource:0}: Error finding container aba17380151b37513202711b44ebdea99baf57501a89d601e4a5a4ef86fecc26: Status 404 returned error can't find the container with id aba17380151b37513202711b44ebdea99baf57501a89d601e4a5a4ef86fecc26
Jul 27 17:34:08.022847 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:34:08.022815    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-84fc959947-j9x2v" event={"ID":"492d3c37-e7bb-4598-84b8-544d13e956ba","Type":"ContainerStarted","Data":"aba17380151b37513202711b44ebdea99baf57501a89d601e4a5a4ef86fecc26"}
Jul 27 17:34:08.723271 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:34:08.723201    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:34:10.034976 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:34:10.034824    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-84fc959947-j9x2v" event={"ID":"492d3c37-e7bb-4598-84b8-544d13e956ba","Type":"ContainerStarted","Data":"ad5b6b7a6ead1e50acb50a3f99d935e9f2396eb46b3f49b9bf94159deebe9861"}
Jul 27 17:34:10.036702 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:34:10.035262    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-84fc959947-j9x2v"
Jul 27 17:34:10.054075 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:34:10.054021    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-84fc959947-j9x2v" podStartSLOduration=1.496736609 podStartE2EDuration="4.054004027s" totalImagesPullingTime="2.557267418s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:34:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:34:10.053577169 +0000 UTC m=+392.196979221" watchObservedRunningTime="2026-07-27 17:34:10.054004027 +0000 UTC m=+392.197406078"
Jul 27 17:34:25.069311 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:34:25.069275    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:34:25.170554 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:34:25.170525    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:34:41.042761 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:34:41.042727    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-84fc959947-j9x2v"
Jul 27 17:34:57.971785 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:34:57.971745    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:34:58.034238 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:34:58.034200    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:34:59.817937 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:34:59.817856    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:34:59.872798 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:34:59.872766    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:35:08.071992 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:08.071950    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:35:08.128888 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:08.128856    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:35:19.074563 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:19.074519    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77ddc7856-pzqmk"]
Jul 27 17:35:19.199846 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:19.199802    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:35:19.299655 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:19.299621    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:35:19.402091 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:19.402053    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:35:22.136324 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:22.136289    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:35:22.170195 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:22.170159    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-jjfkl"]
Jul 27 17:35:22.172605 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:22.172586    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-jjfkl"
Jul 27 17:35:22.175537 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:22.175510    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 27 17:35:22.175666 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:22.175581    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-89sxg\""
Jul 27 17:35:22.185583 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:22.185555    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-jjfkl"]
Jul 27 17:35:22.221149 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:22.221113    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:35:22.235378 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:22.235348    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/562877e9-4748-48fb-8fe6-d27f9f13f93c-tls-certs\") pod \"model-serving-api-7b954ffc95-jjfkl\" (UID: \"562877e9-4748-48fb-8fe6-d27f9f13f93c\") " pod="kserve/model-serving-api-7b954ffc95-jjfkl"
Jul 27 17:35:22.235518 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:22.235385    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hv4r\" (UniqueName: \"kubernetes.io/projected/562877e9-4748-48fb-8fe6-d27f9f13f93c-kube-api-access-8hv4r\") pod \"model-serving-api-7b954ffc95-jjfkl\" (UID: \"562877e9-4748-48fb-8fe6-d27f9f13f93c\") " pod="kserve/model-serving-api-7b954ffc95-jjfkl"
Jul 27 17:35:22.312014 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:22.311981    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:35:22.336209 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:22.336177    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/562877e9-4748-48fb-8fe6-d27f9f13f93c-tls-certs\") pod \"model-serving-api-7b954ffc95-jjfkl\" (UID: \"562877e9-4748-48fb-8fe6-d27f9f13f93c\") " pod="kserve/model-serving-api-7b954ffc95-jjfkl"
Jul 27 17:35:22.336345 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:22.336218    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hv4r\" (UniqueName: \"kubernetes.io/projected/562877e9-4748-48fb-8fe6-d27f9f13f93c-kube-api-access-8hv4r\") pod \"model-serving-api-7b954ffc95-jjfkl\" (UID: \"562877e9-4748-48fb-8fe6-d27f9f13f93c\") " pod="kserve/model-serving-api-7b954ffc95-jjfkl"
Jul 27 17:35:22.338708 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:22.338680    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/562877e9-4748-48fb-8fe6-d27f9f13f93c-tls-certs\") pod \"model-serving-api-7b954ffc95-jjfkl\" (UID: \"562877e9-4748-48fb-8fe6-d27f9f13f93c\") " pod="kserve/model-serving-api-7b954ffc95-jjfkl"
Jul 27 17:35:22.346718 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:22.346691    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hv4r\" (UniqueName: \"kubernetes.io/projected/562877e9-4748-48fb-8fe6-d27f9f13f93c-kube-api-access-8hv4r\") pod \"model-serving-api-7b954ffc95-jjfkl\" (UID: \"562877e9-4748-48fb-8fe6-d27f9f13f93c\") " pod="kserve/model-serving-api-7b954ffc95-jjfkl"
Jul 27 17:35:22.413830 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:22.413790    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:35:22.483414 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:22.483380    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-jjfkl"
Jul 27 17:35:22.614399 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:22.614358    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-jjfkl"]
Jul 27 17:35:22.617119 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:35:22.617083    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod562877e9_4748_48fb_8fe6_d27f9f13f93c.slice/crio-bc781393b2f148021eee2764358cc67793de2f460a874d5d22102fb587aa986e WatchSource:0}: Error finding container bc781393b2f148021eee2764358cc67793de2f460a874d5d22102fb587aa986e: Status 404 returned error can't find the container with id bc781393b2f148021eee2764358cc67793de2f460a874d5d22102fb587aa986e
Jul 27 17:35:22.856862 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:22.856775    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:35:22.920098 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:22.920057    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:35:23.356004 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:23.355964    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-jjfkl" event={"ID":"562877e9-4748-48fb-8fe6-d27f9f13f93c","Type":"ContainerStarted","Data":"bc781393b2f148021eee2764358cc67793de2f460a874d5d22102fb587aa986e"}
Jul 27 17:35:25.366301 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:25.366187    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-jjfkl" event={"ID":"562877e9-4748-48fb-8fe6-d27f9f13f93c","Type":"ContainerStarted","Data":"edee5cde7a1415ac02d3a1247a571d81104a89bd59d91316a3f35eb8179316ec"}
Jul 27 17:35:25.366706 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:25.366300    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-jjfkl"
Jul 27 17:35:25.387546 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:25.387493    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-jjfkl" podStartSLOduration=1.039788054 podStartE2EDuration="3.387474498s" totalImagesPullingTime="2.347686444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:35:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:35:25.3848322 +0000 UTC m=+467.528234252" watchObservedRunningTime="2026-07-27 17:35:25.387474498 +0000 UTC m=+467.530876549"
Jul 27 17:35:36.374044 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:36.374014    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-jjfkl"
Jul 27 17:35:44.094376 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:44.094304    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-77ddc7856-pzqmk" podUID="66a0649f-1102-4dd9-9b70-49d34f92c512" containerName="console" containerID="cri-o://bce03b917fb06840178e708dd738ad226f476f8828f9dd6ae8e598b5c4edb532" gracePeriod=15
Jul 27 17:35:44.347067 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:44.347000    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77ddc7856-pzqmk"
Jul 27 17:35:44.399527 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:44.399499    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66a0649f-1102-4dd9-9b70-49d34f92c512-service-ca\") pod \"66a0649f-1102-4dd9-9b70-49d34f92c512\" (UID: \"66a0649f-1102-4dd9-9b70-49d34f92c512\") "
Jul 27 17:35:44.399697 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:44.399550    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66a0649f-1102-4dd9-9b70-49d34f92c512-oauth-serving-cert\") pod \"66a0649f-1102-4dd9-9b70-49d34f92c512\" (UID: \"66a0649f-1102-4dd9-9b70-49d34f92c512\") "
Jul 27 17:35:44.399697 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:44.399591    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66a0649f-1102-4dd9-9b70-49d34f92c512-console-config\") pod \"66a0649f-1102-4dd9-9b70-49d34f92c512\" (UID: \"66a0649f-1102-4dd9-9b70-49d34f92c512\") "
Jul 27 17:35:44.399767 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:44.399707    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66a0649f-1102-4dd9-9b70-49d34f92c512-console-serving-cert\") pod \"66a0649f-1102-4dd9-9b70-49d34f92c512\" (UID: \"66a0649f-1102-4dd9-9b70-49d34f92c512\") "
Jul 27 17:35:44.399767 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:44.399756    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9r59w\" (UniqueName: \"kubernetes.io/projected/66a0649f-1102-4dd9-9b70-49d34f92c512-kube-api-access-9r59w\") pod \"66a0649f-1102-4dd9-9b70-49d34f92c512\" (UID: \"66a0649f-1102-4dd9-9b70-49d34f92c512\") "
Jul 27 17:35:44.399870 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:44.399804    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66a0649f-1102-4dd9-9b70-49d34f92c512-console-oauth-config\") pod \"66a0649f-1102-4dd9-9b70-49d34f92c512\" (UID: \"66a0649f-1102-4dd9-9b70-49d34f92c512\") "
Jul 27 17:35:44.399870 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:44.399846    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66a0649f-1102-4dd9-9b70-49d34f92c512-trusted-ca-bundle\") pod \"66a0649f-1102-4dd9-9b70-49d34f92c512\" (UID: \"66a0649f-1102-4dd9-9b70-49d34f92c512\") "
Jul 27 17:35:44.399972 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:44.399916    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66a0649f-1102-4dd9-9b70-49d34f92c512-service-ca" (OuterVolumeSpecName: "service-ca") pod "66a0649f-1102-4dd9-9b70-49d34f92c512" (UID: "66a0649f-1102-4dd9-9b70-49d34f92c512"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:35:44.400044 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:44.400018    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66a0649f-1102-4dd9-9b70-49d34f92c512-console-config" (OuterVolumeSpecName: "console-config") pod "66a0649f-1102-4dd9-9b70-49d34f92c512" (UID: "66a0649f-1102-4dd9-9b70-49d34f92c512"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:35:44.400141 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:44.400085    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66a0649f-1102-4dd9-9b70-49d34f92c512-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "66a0649f-1102-4dd9-9b70-49d34f92c512" (UID: "66a0649f-1102-4dd9-9b70-49d34f92c512"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:35:44.400209 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:44.400196    2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66a0649f-1102-4dd9-9b70-49d34f92c512-console-config\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:35:44.400279 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:44.400214    2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66a0649f-1102-4dd9-9b70-49d34f92c512-service-ca\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:35:44.400374 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:44.400345    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66a0649f-1102-4dd9-9b70-49d34f92c512-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "66a0649f-1102-4dd9-9b70-49d34f92c512" (UID: "66a0649f-1102-4dd9-9b70-49d34f92c512"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:35:44.401982 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:44.401952    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66a0649f-1102-4dd9-9b70-49d34f92c512-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "66a0649f-1102-4dd9-9b70-49d34f92c512" (UID: "66a0649f-1102-4dd9-9b70-49d34f92c512"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:35:44.402094 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:44.401989    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66a0649f-1102-4dd9-9b70-49d34f92c512-kube-api-access-9r59w" (OuterVolumeSpecName: "kube-api-access-9r59w") pod "66a0649f-1102-4dd9-9b70-49d34f92c512" (UID: "66a0649f-1102-4dd9-9b70-49d34f92c512"). InnerVolumeSpecName "kube-api-access-9r59w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:35:44.402094 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:44.402055    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66a0649f-1102-4dd9-9b70-49d34f92c512-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "66a0649f-1102-4dd9-9b70-49d34f92c512" (UID: "66a0649f-1102-4dd9-9b70-49d34f92c512"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:35:44.456905 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:44.456864    2573 generic.go:356] "Generic (PLEG): container finished" podID="66a0649f-1102-4dd9-9b70-49d34f92c512" containerID="bce03b917fb06840178e708dd738ad226f476f8828f9dd6ae8e598b5c4edb532" exitCode=0
Jul 27 17:35:44.457094 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:44.456929    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77ddc7856-pzqmk"
Jul 27 17:35:44.457094 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:44.456951    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77ddc7856-pzqmk" event={"ID":"66a0649f-1102-4dd9-9b70-49d34f92c512","Type":"ContainerDied","Data":"bce03b917fb06840178e708dd738ad226f476f8828f9dd6ae8e598b5c4edb532"}
Jul 27 17:35:44.457094 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:44.457028    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77ddc7856-pzqmk" event={"ID":"66a0649f-1102-4dd9-9b70-49d34f92c512","Type":"ContainerDied","Data":"3366a002287d1011e0eb874997df99fb30a0cc341da55794bb0f1b6e41b67eba"}
Jul 27 17:35:44.457094 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:44.457051    2573 scope.go:117] "RemoveContainer" containerID="bce03b917fb06840178e708dd738ad226f476f8828f9dd6ae8e598b5c4edb532"
Jul 27 17:35:44.465734 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:44.465710    2573 scope.go:117] "RemoveContainer" containerID="bce03b917fb06840178e708dd738ad226f476f8828f9dd6ae8e598b5c4edb532"
Jul 27 17:35:44.466029 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:35:44.466009    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bce03b917fb06840178e708dd738ad226f476f8828f9dd6ae8e598b5c4edb532\": container with ID starting with bce03b917fb06840178e708dd738ad226f476f8828f9dd6ae8e598b5c4edb532 not found: ID does not exist" containerID="bce03b917fb06840178e708dd738ad226f476f8828f9dd6ae8e598b5c4edb532"
Jul 27 17:35:44.466111 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:44.466037    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bce03b917fb06840178e708dd738ad226f476f8828f9dd6ae8e598b5c4edb532"} err="failed to get container status \"bce03b917fb06840178e708dd738ad226f476f8828f9dd6ae8e598b5c4edb532\": rpc error: code = NotFound desc = could not find container \"bce03b917fb06840178e708dd738ad226f476f8828f9dd6ae8e598b5c4edb532\": container with ID starting with bce03b917fb06840178e708dd738ad226f476f8828f9dd6ae8e598b5c4edb532 not found: ID does not exist"
Jul 27 17:35:44.479431 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:44.479404    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77ddc7856-pzqmk"]
Jul 27 17:35:44.482816 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:44.482791    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-77ddc7856-pzqmk"]
Jul 27 17:35:44.501440 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:44.501404    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r59w\" (UniqueName: \"kubernetes.io/projected/66a0649f-1102-4dd9-9b70-49d34f92c512-kube-api-access-9r59w\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:35:44.501440 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:44.501441    2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66a0649f-1102-4dd9-9b70-49d34f92c512-console-oauth-config\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:35:44.501653 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:44.501455    2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66a0649f-1102-4dd9-9b70-49d34f92c512-trusted-ca-bundle\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:35:44.501653 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:44.501467    2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66a0649f-1102-4dd9-9b70-49d34f92c512-oauth-serving-cert\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:35:44.501653 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:44.501481    2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66a0649f-1102-4dd9-9b70-49d34f92c512-console-serving-cert\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:35:46.468634 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:46.468602    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66a0649f-1102-4dd9-9b70-49d34f92c512" path="/var/lib/kubelet/pods/66a0649f-1102-4dd9-9b70-49d34f92c512/volumes"
Jul 27 17:35:48.364412 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:35:48.364381    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:36:52.977767 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:36:52.977734    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:37:00.141689 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:37:00.141651    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-rxrjn"]
Jul 27 17:37:00.142079 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:37:00.142001    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66a0649f-1102-4dd9-9b70-49d34f92c512" containerName="console"
Jul 27 17:37:00.142079 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:37:00.142012    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="66a0649f-1102-4dd9-9b70-49d34f92c512" containerName="console"
Jul 27 17:37:00.142149 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:37:00.142090    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="66a0649f-1102-4dd9-9b70-49d34f92c512" containerName="console"
Jul 27 17:37:00.144798 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:37:00.144780    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-rxrjn"
Jul 27 17:37:00.147751 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:37:00.147717    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-wrc7l\""
Jul 27 17:37:00.147751 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:37:00.147725    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Jul 27 17:37:00.153488 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:37:00.153460    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-rxrjn"]
Jul 27 17:37:00.223385 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:37:00.223350    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:37:00.236356 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:37:00.236324    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvhnv\" (UniqueName: \"kubernetes.io/projected/0ba231a3-9ff6-43f6-be66-194b52fd6bbf-kube-api-access-kvhnv\") pod \"seaweedfs-tls-custom-6644bf5d6c-rxrjn\" (UID: \"0ba231a3-9ff6-43f6-be66-194b52fd6bbf\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-rxrjn"
Jul 27 17:37:00.236497 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:37:00.236419    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0ba231a3-9ff6-43f6-be66-194b52fd6bbf-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-rxrjn\" (UID: \"0ba231a3-9ff6-43f6-be66-194b52fd6bbf\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-rxrjn"
Jul 27 17:37:00.336819 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:37:00.336781    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0ba231a3-9ff6-43f6-be66-194b52fd6bbf-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-rxrjn\" (UID: \"0ba231a3-9ff6-43f6-be66-194b52fd6bbf\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-rxrjn"
Jul 27 17:37:00.336954 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:37:00.336857    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvhnv\" (UniqueName: \"kubernetes.io/projected/0ba231a3-9ff6-43f6-be66-194b52fd6bbf-kube-api-access-kvhnv\") pod \"seaweedfs-tls-custom-6644bf5d6c-rxrjn\" (UID: \"0ba231a3-9ff6-43f6-be66-194b52fd6bbf\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-rxrjn"
Jul 27 17:37:00.337166 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:37:00.337148    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0ba231a3-9ff6-43f6-be66-194b52fd6bbf-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-rxrjn\" (UID: \"0ba231a3-9ff6-43f6-be66-194b52fd6bbf\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-rxrjn"
Jul 27 17:37:00.347231 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:37:00.347196    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvhnv\" (UniqueName: \"kubernetes.io/projected/0ba231a3-9ff6-43f6-be66-194b52fd6bbf-kube-api-access-kvhnv\") pod \"seaweedfs-tls-custom-6644bf5d6c-rxrjn\" (UID: \"0ba231a3-9ff6-43f6-be66-194b52fd6bbf\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-rxrjn"
Jul 27 17:37:00.454563 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:37:00.454466    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-rxrjn"
Jul 27 17:37:00.588005 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:37:00.587970    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-rxrjn"]
Jul 27 17:37:00.592263 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:37:00.592223    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ba231a3_9ff6_43f6_be66_194b52fd6bbf.slice/crio-5b9fe779af992ab7814c3e64407c48ee6714983399354ec06a441d0e153d90f3 WatchSource:0}: Error finding container 5b9fe779af992ab7814c3e64407c48ee6714983399354ec06a441d0e153d90f3: Status 404 returned error can't find the container with id 5b9fe779af992ab7814c3e64407c48ee6714983399354ec06a441d0e153d90f3
Jul 27 17:37:00.788766 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:37:00.788683    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-rxrjn" event={"ID":"0ba231a3-9ff6-43f6-be66-194b52fd6bbf","Type":"ContainerStarted","Data":"5b9fe779af992ab7814c3e64407c48ee6714983399354ec06a441d0e153d90f3"}
Jul 27 17:37:05.811062 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:37:05.811025    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-rxrjn" event={"ID":"0ba231a3-9ff6-43f6-be66-194b52fd6bbf","Type":"ContainerStarted","Data":"8e0f481644baab7309f3af68a2c45c16514ae4e3c32c9ba3810d511bdcbea59e"}
Jul 27 17:37:07.838973 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:37:07.838916    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-rxrjn" podStartSLOduration=3.581310106 podStartE2EDuration="7.838900117s" totalImagesPullingTime="4.257590011s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:37:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:37:05.834771002 +0000 UTC m=+567.978173054" watchObservedRunningTime="2026-07-27 17:37:07.838900117 +0000 UTC m=+569.982302168"
Jul 27 17:37:07.839419 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:37:07.839397    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-rxrjn"]
Jul 27 17:37:07.839596 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:37:07.839576    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-rxrjn" podUID="0ba231a3-9ff6-43f6-be66-194b52fd6bbf" containerName="seaweedfs-tls-custom" containerID="cri-o://8e0f481644baab7309f3af68a2c45c16514ae4e3c32c9ba3810d511bdcbea59e" gracePeriod=30
Jul 27 17:37:35.603405 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:37:35.603378    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-rxrjn"
Jul 27 17:37:35.637909 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:37:35.637887    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0ba231a3-9ff6-43f6-be66-194b52fd6bbf-data\") pod \"0ba231a3-9ff6-43f6-be66-194b52fd6bbf\" (UID: \"0ba231a3-9ff6-43f6-be66-194b52fd6bbf\") "
Jul 27 17:37:35.638066 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:37:35.637966    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvhnv\" (UniqueName: \"kubernetes.io/projected/0ba231a3-9ff6-43f6-be66-194b52fd6bbf-kube-api-access-kvhnv\") pod \"0ba231a3-9ff6-43f6-be66-194b52fd6bbf\" (UID: \"0ba231a3-9ff6-43f6-be66-194b52fd6bbf\") "
Jul 27 17:37:35.639788 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:37:35.639746    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ba231a3-9ff6-43f6-be66-194b52fd6bbf-data" (OuterVolumeSpecName: "data") pod "0ba231a3-9ff6-43f6-be66-194b52fd6bbf" (UID: "0ba231a3-9ff6-43f6-be66-194b52fd6bbf"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:37:35.640433 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:37:35.640402    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ba231a3-9ff6-43f6-be66-194b52fd6bbf-kube-api-access-kvhnv" (OuterVolumeSpecName: "kube-api-access-kvhnv") pod "0ba231a3-9ff6-43f6-be66-194b52fd6bbf" (UID: "0ba231a3-9ff6-43f6-be66-194b52fd6bbf"). InnerVolumeSpecName "kube-api-access-kvhnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:37:35.739118 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:37:35.739075    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvhnv\" (UniqueName: \"kubernetes.io/projected/0ba231a3-9ff6-43f6-be66-194b52fd6bbf-kube-api-access-kvhnv\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:37:35.739118 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:37:35.739109    2573 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0ba231a3-9ff6-43f6-be66-194b52fd6bbf-data\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:37:35.939211 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:37:35.939119    2573 generic.go:356] "Generic (PLEG): container finished" podID="0ba231a3-9ff6-43f6-be66-194b52fd6bbf" containerID="8e0f481644baab7309f3af68a2c45c16514ae4e3c32c9ba3810d511bdcbea59e" exitCode=0
Jul 27 17:37:35.939211 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:37:35.939168    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-rxrjn"
Jul 27 17:37:35.939475 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:37:35.939213    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-rxrjn" event={"ID":"0ba231a3-9ff6-43f6-be66-194b52fd6bbf","Type":"ContainerDied","Data":"8e0f481644baab7309f3af68a2c45c16514ae4e3c32c9ba3810d511bdcbea59e"}
Jul 27 17:37:35.939475 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:37:35.939292    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-rxrjn" event={"ID":"0ba231a3-9ff6-43f6-be66-194b52fd6bbf","Type":"ContainerDied","Data":"5b9fe779af992ab7814c3e64407c48ee6714983399354ec06a441d0e153d90f3"}
Jul 27 17:37:35.939475 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:37:35.939321    2573 scope.go:117] "RemoveContainer" containerID="8e0f481644baab7309f3af68a2c45c16514ae4e3c32c9ba3810d511bdcbea59e"
Jul 27 17:37:35.949420 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:37:35.949400    2573 scope.go:117] "RemoveContainer" containerID="8e0f481644baab7309f3af68a2c45c16514ae4e3c32c9ba3810d511bdcbea59e"
Jul 27 17:37:35.949738 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:37:35.949718    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e0f481644baab7309f3af68a2c45c16514ae4e3c32c9ba3810d511bdcbea59e\": container with ID starting with 8e0f481644baab7309f3af68a2c45c16514ae4e3c32c9ba3810d511bdcbea59e not found: ID does not exist" containerID="8e0f481644baab7309f3af68a2c45c16514ae4e3c32c9ba3810d511bdcbea59e"
Jul 27 17:37:35.949817 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:37:35.949749    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e0f481644baab7309f3af68a2c45c16514ae4e3c32c9ba3810d511bdcbea59e"} err="failed to get container status \"8e0f481644baab7309f3af68a2c45c16514ae4e3c32c9ba3810d511bdcbea59e\": rpc error: code = NotFound desc = could not find container \"8e0f481644baab7309f3af68a2c45c16514ae4e3c32c9ba3810d511bdcbea59e\": container with ID starting with 8e0f481644baab7309f3af68a2c45c16514ae4e3c32c9ba3810d511bdcbea59e not found: ID does not exist"
Jul 27 17:37:35.961854 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:37:35.961825    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-rxrjn"]
Jul 27 17:37:35.965363 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:37:35.965340    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-rxrjn"]
Jul 27 17:37:36.469221 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:37:36.469184    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ba231a3-9ff6-43f6-be66-194b52fd6bbf" path="/var/lib/kubelet/pods/0ba231a3-9ff6-43f6-be66-194b52fd6bbf/volumes"
Jul 27 17:37:38.276812 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:37:38.276785    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vc2tz_87dac797-f8d2-4d36-999f-f8ce855dbeff/ovn-acl-logging/0.log"
Jul 27 17:37:38.280817 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:37:38.280796    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vc2tz_87dac797-f8d2-4d36-999f-f8ce855dbeff/ovn-acl-logging/0.log"
Jul 27 17:37:53.238023 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:37:53.237981    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:37:53.843849 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:37:53.843810    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:38:09.087388 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:38:09.087353    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:38:09.187863 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:38:09.187827    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:38:20.443126 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:38:20.443090    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:38:20.557563 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:38:20.557530    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d"]
Jul 27 17:38:20.557876 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:38:20.557865    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ba231a3-9ff6-43f6-be66-194b52fd6bbf" containerName="seaweedfs-tls-custom"
Jul 27 17:38:20.557925 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:38:20.557878    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ba231a3-9ff6-43f6-be66-194b52fd6bbf" containerName="seaweedfs-tls-custom"
Jul 27 17:38:20.557959 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:38:20.557935    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ba231a3-9ff6-43f6-be66-194b52fd6bbf" containerName="seaweedfs-tls-custom"
Jul 27 17:38:20.561112 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:38:20.561088    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d"
Jul 27 17:38:20.564219 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:38:20.564197    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 17:38:20.564377 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:38:20.564339    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Jul 27 17:38:20.564430 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:38:20.564390    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Jul 27 17:38:20.564430 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:38:20.564419    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 17:38:20.564764 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:38:20.564742    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z6ld7\""
Jul 27 17:38:20.565584 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:38:20.565565    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 17:38:20.565667 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:38:20.565602    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 17:38:20.571109 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:38:20.571084    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d"]
Jul 27 17:38:20.604509 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:38:20.604475    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37948065-229b-415a-80e9-80135ce60bd3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d\" (UID: \"37948065-229b-415a-80e9-80135ce60bd3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d"
Jul 27 17:38:20.604689 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:38:20.604517    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37948065-229b-415a-80e9-80135ce60bd3-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d\" (UID: \"37948065-229b-415a-80e9-80135ce60bd3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d"
Jul 27 17:38:20.604689 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:38:20.604538    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37948065-229b-415a-80e9-80135ce60bd3-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d\" (UID: \"37948065-229b-415a-80e9-80135ce60bd3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d"
Jul 27 17:38:20.604689 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:38:20.604565    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37948065-229b-415a-80e9-80135ce60bd3-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d\" (UID: \"37948065-229b-415a-80e9-80135ce60bd3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d"
Jul 27 17:38:20.604689 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:38:20.604635    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdcgd\" (UniqueName: \"kubernetes.io/projected/37948065-229b-415a-80e9-80135ce60bd3-kube-api-access-rdcgd\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d\" (UID: \"37948065-229b-415a-80e9-80135ce60bd3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d"
Jul 27 17:38:20.705317 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:38:20.705200    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdcgd\" (UniqueName: \"kubernetes.io/projected/37948065-229b-415a-80e9-80135ce60bd3-kube-api-access-rdcgd\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d\" (UID: \"37948065-229b-415a-80e9-80135ce60bd3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d"
Jul 27 17:38:20.705317 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:38:20.705283    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37948065-229b-415a-80e9-80135ce60bd3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d\" (UID: \"37948065-229b-415a-80e9-80135ce60bd3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d"
Jul 27 17:38:20.705524 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:38:20.705320    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37948065-229b-415a-80e9-80135ce60bd3-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d\" (UID: \"37948065-229b-415a-80e9-80135ce60bd3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d"
Jul 27 17:38:20.705524 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:38:20.705341    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37948065-229b-415a-80e9-80135ce60bd3-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d\" (UID: \"37948065-229b-415a-80e9-80135ce60bd3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d"
Jul 27 17:38:20.705524 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:38:20.705365    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37948065-229b-415a-80e9-80135ce60bd3-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d\" (UID: \"37948065-229b-415a-80e9-80135ce60bd3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d"
Jul 27 17:38:20.705524 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:38:20.705449    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-serving-cert: secret "isvc-sklearn-batcher-predictor-serving-cert" not found
Jul 27 17:38:20.705524 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:38:20.705527    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37948065-229b-415a-80e9-80135ce60bd3-proxy-tls podName:37948065-229b-415a-80e9-80135ce60bd3 nodeName:}" failed. No retries permitted until 2026-07-27 17:38:21.205507382 +0000 UTC m=+643.348909417 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/37948065-229b-415a-80e9-80135ce60bd3-proxy-tls") pod "isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d" (UID: "37948065-229b-415a-80e9-80135ce60bd3") : secret "isvc-sklearn-batcher-predictor-serving-cert" not found
Jul 27 17:38:20.705817 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:38:20.705799    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37948065-229b-415a-80e9-80135ce60bd3-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d\" (UID: \"37948065-229b-415a-80e9-80135ce60bd3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d"
Jul 27 17:38:20.706109 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:38:20.706087    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37948065-229b-415a-80e9-80135ce60bd3-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d\" (UID: \"37948065-229b-415a-80e9-80135ce60bd3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d"
Jul 27 17:38:20.706149 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:38:20.706097    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37948065-229b-415a-80e9-80135ce60bd3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d\" (UID: \"37948065-229b-415a-80e9-80135ce60bd3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d"
Jul 27 17:38:20.714539 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:38:20.714514    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdcgd\" (UniqueName: \"kubernetes.io/projected/37948065-229b-415a-80e9-80135ce60bd3-kube-api-access-rdcgd\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d\" (UID: \"37948065-229b-415a-80e9-80135ce60bd3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d"
Jul 27 17:38:20.868971 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:38:20.868933    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:38:21.209397 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:38:21.209361    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37948065-229b-415a-80e9-80135ce60bd3-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d\" (UID: \"37948065-229b-415a-80e9-80135ce60bd3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d"
Jul 27 17:38:21.212003 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:38:21.211971    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37948065-229b-415a-80e9-80135ce60bd3-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d\" (UID: \"37948065-229b-415a-80e9-80135ce60bd3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d"
Jul 27 17:38:21.472692 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:38:21.472607    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d"
Jul 27 17:38:21.614403 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:38:21.614378    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d"]
Jul 27 17:38:21.617102 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:38:21.617073    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37948065_229b_415a_80e9_80135ce60bd3.slice/crio-0ec7e240d4c87b2d67e17ac5eabd64961f13053dcf286e0c07f7b42c34901048 WatchSource:0}: Error finding container 0ec7e240d4c87b2d67e17ac5eabd64961f13053dcf286e0c07f7b42c34901048: Status 404 returned error can't find the container with id 0ec7e240d4c87b2d67e17ac5eabd64961f13053dcf286e0c07f7b42c34901048
Jul 27 17:38:21.619340 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:38:21.619321    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 17:38:22.135668 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:38:22.135634    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d" event={"ID":"37948065-229b-415a-80e9-80135ce60bd3","Type":"ContainerStarted","Data":"0ec7e240d4c87b2d67e17ac5eabd64961f13053dcf286e0c07f7b42c34901048"}
Jul 27 17:38:26.156546 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:38:26.156495    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d" event={"ID":"37948065-229b-415a-80e9-80135ce60bd3","Type":"ContainerStarted","Data":"261576af76f625798cba3895ccb75b164d0b0b0cbc55b17bb7bdec184a674f5e"}
Jul 27 17:38:29.171002 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:38:29.170968    2573 generic.go:356] "Generic (PLEG): container finished" podID="37948065-229b-415a-80e9-80135ce60bd3" containerID="261576af76f625798cba3895ccb75b164d0b0b0cbc55b17bb7bdec184a674f5e" exitCode=0
Jul 27 17:38:29.171406 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:38:29.171041    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d" event={"ID":"37948065-229b-415a-80e9-80135ce60bd3","Type":"ContainerDied","Data":"261576af76f625798cba3895ccb75b164d0b0b0cbc55b17bb7bdec184a674f5e"}
Jul 27 17:38:44.250422 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:38:44.250383    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d" event={"ID":"37948065-229b-415a-80e9-80135ce60bd3","Type":"ContainerStarted","Data":"28174831a9263f99e31baa088c887ed3d8c3f8bbf73f7753588fc7a10af7646b"}
Jul 27 17:38:46.260980 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:38:46.260944    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d" event={"ID":"37948065-229b-415a-80e9-80135ce60bd3","Type":"ContainerStarted","Data":"5f91a2b9b5166351c1da8d2b83e3386ad807ba65c8104f6af87a5915caa1fbb1"}
Jul 27 17:38:49.277230 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:38:49.277189    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d" event={"ID":"37948065-229b-415a-80e9-80135ce60bd3","Type":"ContainerStarted","Data":"ebe9c59bd18726fe145dbf7cfd9e72945ec03ff9341b5ea8b760887d490411a1"}
Jul 27 17:38:49.277740 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:38:49.277306    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d"
Jul 27 17:38:49.277740 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:38:49.277345    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d"
Jul 27 17:38:49.278770 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:38:49.278745    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d" podUID="37948065-229b-415a-80e9-80135ce60bd3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 27 17:38:49.303311 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:38:49.303258    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d" podStartSLOduration=2.83147716 podStartE2EDuration="29.303230214s" totalImagesPullingTime="22.752168255s" totalInitContainerRuntime="3.719584799s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:38:20 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:38:49.301989721 +0000 UTC m=+671.445391764" watchObservedRunningTime="2026-07-27 17:38:49.303230214 +0000 UTC m=+671.446632332"
Jul 27 17:38:50.281625 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:38:50.281578    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d"
Jul 27 17:38:50.282080 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:38:50.281773    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d" podUID="37948065-229b-415a-80e9-80135ce60bd3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 27 17:38:50.282780 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:38:50.282753    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d" podUID="37948065-229b-415a-80e9-80135ce60bd3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 17:38:51.286682 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:38:51.286638    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d" podUID="37948065-229b-415a-80e9-80135ce60bd3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 27 17:38:51.287120 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:38:51.287044    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d" podUID="37948065-229b-415a-80e9-80135ce60bd3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 17:38:51.291231 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:38:51.291209    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d"
Jul 27 17:38:52.291263 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:38:52.291216    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d" podUID="37948065-229b-415a-80e9-80135ce60bd3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 27 17:38:52.291730 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:38:52.291697    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d" podUID="37948065-229b-415a-80e9-80135ce60bd3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 17:39:02.291592 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:39:02.291545    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d" podUID="37948065-229b-415a-80e9-80135ce60bd3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 27 17:39:02.292152 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:39:02.292071    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d" podUID="37948065-229b-415a-80e9-80135ce60bd3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 17:39:12.291757 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:39:12.291702    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d" podUID="37948065-229b-415a-80e9-80135ce60bd3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 27 17:39:12.292227 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:39:12.292180    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d" podUID="37948065-229b-415a-80e9-80135ce60bd3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 17:39:18.206879 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:39:18.206838    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:39:22.292000 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:39:22.291950    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d" podUID="37948065-229b-415a-80e9-80135ce60bd3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 27 17:39:22.292445 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:39:22.292412    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d" podUID="37948065-229b-415a-80e9-80135ce60bd3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 17:39:32.291196 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:39:32.291105    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d" podUID="37948065-229b-415a-80e9-80135ce60bd3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 27 17:39:32.291788 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:39:32.291763    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d" podUID="37948065-229b-415a-80e9-80135ce60bd3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 17:39:36.966852 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:39:36.966813    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:39:42.292117 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:39:42.292062    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d" podUID="37948065-229b-415a-80e9-80135ce60bd3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 27 17:39:42.294882 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:39:42.292475    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d" podUID="37948065-229b-415a-80e9-80135ce60bd3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 17:39:52.291924 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:39:52.291875    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d" podUID="37948065-229b-415a-80e9-80135ce60bd3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 27 17:39:52.292518 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:39:52.292439    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d" podUID="37948065-229b-415a-80e9-80135ce60bd3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 17:40:02.292204 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:02.292169    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d"
Jul 27 17:40:02.292678 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:02.292460    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d"
Jul 27 17:40:15.603195 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:15.603156    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:40:15.671951 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:15.671900    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d"]
Jul 27 17:40:15.672286 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:15.672234    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d" podUID="37948065-229b-415a-80e9-80135ce60bd3" containerName="kserve-container" containerID="cri-o://28174831a9263f99e31baa088c887ed3d8c3f8bbf73f7753588fc7a10af7646b" gracePeriod=30
Jul 27 17:40:15.672286 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:15.672234    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d" podUID="37948065-229b-415a-80e9-80135ce60bd3" containerName="agent" containerID="cri-o://ebe9c59bd18726fe145dbf7cfd9e72945ec03ff9341b5ea8b760887d490411a1" gracePeriod=30
Jul 27 17:40:15.672507 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:15.672279    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d" podUID="37948065-229b-415a-80e9-80135ce60bd3" containerName="kube-rbac-proxy" containerID="cri-o://5f91a2b9b5166351c1da8d2b83e3386ad807ba65c8104f6af87a5915caa1fbb1" gracePeriod=30
Jul 27 17:40:15.692098 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:15.692070    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:40:15.699358 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:15.699333    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz"]
Jul 27 17:40:15.702900 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:15.702876    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz"
Jul 27 17:40:15.706009 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:15.705988    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Jul 27 17:40:15.706126 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:15.706112    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Jul 27 17:40:15.715007 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:15.714983    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz"]
Jul 27 17:40:15.792944 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:15.792915    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:40:15.801007 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:15.800984    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/100e1c22-8fa3-439b-bc52-5ac62d3dfbf0-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz\" (UID: \"100e1c22-8fa3-439b-bc52-5ac62d3dfbf0\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz"
Jul 27 17:40:15.801105 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:15.801023    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/100e1c22-8fa3-439b-bc52-5ac62d3dfbf0-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz\" (UID: \"100e1c22-8fa3-439b-bc52-5ac62d3dfbf0\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz"
Jul 27 17:40:15.801105 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:15.801091    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/100e1c22-8fa3-439b-bc52-5ac62d3dfbf0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz\" (UID: \"100e1c22-8fa3-439b-bc52-5ac62d3dfbf0\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz"
Jul 27 17:40:15.801211 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:15.801191    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjcbf\" (UniqueName: \"kubernetes.io/projected/100e1c22-8fa3-439b-bc52-5ac62d3dfbf0-kube-api-access-qjcbf\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz\" (UID: \"100e1c22-8fa3-439b-bc52-5ac62d3dfbf0\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz"
Jul 27 17:40:15.801282 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:15.801227    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/100e1c22-8fa3-439b-bc52-5ac62d3dfbf0-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz\" (UID: \"100e1c22-8fa3-439b-bc52-5ac62d3dfbf0\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz"
Jul 27 17:40:15.902664 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:15.902630    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjcbf\" (UniqueName: \"kubernetes.io/projected/100e1c22-8fa3-439b-bc52-5ac62d3dfbf0-kube-api-access-qjcbf\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz\" (UID: \"100e1c22-8fa3-439b-bc52-5ac62d3dfbf0\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz"
Jul 27 17:40:15.902871 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:15.902669    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/100e1c22-8fa3-439b-bc52-5ac62d3dfbf0-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz\" (UID: \"100e1c22-8fa3-439b-bc52-5ac62d3dfbf0\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz"
Jul 27 17:40:15.902871 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:15.902696    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/100e1c22-8fa3-439b-bc52-5ac62d3dfbf0-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz\" (UID: \"100e1c22-8fa3-439b-bc52-5ac62d3dfbf0\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz"
Jul 27 17:40:15.902871 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:15.902732    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/100e1c22-8fa3-439b-bc52-5ac62d3dfbf0-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz\" (UID: \"100e1c22-8fa3-439b-bc52-5ac62d3dfbf0\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz"
Jul 27 17:40:15.902871 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:15.902785    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/100e1c22-8fa3-439b-bc52-5ac62d3dfbf0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz\" (UID: \"100e1c22-8fa3-439b-bc52-5ac62d3dfbf0\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz"
Jul 27 17:40:15.903199 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:15.903167    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/100e1c22-8fa3-439b-bc52-5ac62d3dfbf0-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz\" (UID: \"100e1c22-8fa3-439b-bc52-5ac62d3dfbf0\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz"
Jul 27 17:40:15.903485 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:15.903464    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/100e1c22-8fa3-439b-bc52-5ac62d3dfbf0-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz\" (UID: \"100e1c22-8fa3-439b-bc52-5ac62d3dfbf0\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz"
Jul 27 17:40:15.903570 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:15.903522    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/100e1c22-8fa3-439b-bc52-5ac62d3dfbf0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz\" (UID: \"100e1c22-8fa3-439b-bc52-5ac62d3dfbf0\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz"
Jul 27 17:40:15.905073 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:15.905056    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/100e1c22-8fa3-439b-bc52-5ac62d3dfbf0-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz\" (UID: \"100e1c22-8fa3-439b-bc52-5ac62d3dfbf0\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz"
Jul 27 17:40:15.912456 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:15.912438    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjcbf\" (UniqueName: \"kubernetes.io/projected/100e1c22-8fa3-439b-bc52-5ac62d3dfbf0-kube-api-access-qjcbf\") pod \"isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz\" (UID: \"100e1c22-8fa3-439b-bc52-5ac62d3dfbf0\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz"
Jul 27 17:40:16.012625 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:16.012574    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz"
Jul 27 17:40:16.151243 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:16.151216    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz"]
Jul 27 17:40:16.153951 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:40:16.153885    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod100e1c22_8fa3_439b_bc52_5ac62d3dfbf0.slice/crio-381b71b24bcd24e1b7dd78dcce64a6df1bf947527c50289bd35d5aa45cb8860d WatchSource:0}: Error finding container 381b71b24bcd24e1b7dd78dcce64a6df1bf947527c50289bd35d5aa45cb8860d: Status 404 returned error can't find the container with id 381b71b24bcd24e1b7dd78dcce64a6df1bf947527c50289bd35d5aa45cb8860d
Jul 27 17:40:16.287705 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:16.287664    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d" podUID="37948065-229b-415a-80e9-80135ce60bd3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Jul 27 17:40:16.663512 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:16.663477    2573 generic.go:356] "Generic (PLEG): container finished" podID="37948065-229b-415a-80e9-80135ce60bd3" containerID="5f91a2b9b5166351c1da8d2b83e3386ad807ba65c8104f6af87a5915caa1fbb1" exitCode=2
Jul 27 17:40:16.663955 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:16.663547    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d" event={"ID":"37948065-229b-415a-80e9-80135ce60bd3","Type":"ContainerDied","Data":"5f91a2b9b5166351c1da8d2b83e3386ad807ba65c8104f6af87a5915caa1fbb1"}
Jul 27 17:40:16.665166 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:16.665135    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz" event={"ID":"100e1c22-8fa3-439b-bc52-5ac62d3dfbf0","Type":"ContainerStarted","Data":"2b677af56b01b9aa9dad448bb2c25058e831a56e6de4201fbad4ae90eee1df27"}
Jul 27 17:40:16.665315 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:16.665202    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz" event={"ID":"100e1c22-8fa3-439b-bc52-5ac62d3dfbf0","Type":"ContainerStarted","Data":"381b71b24bcd24e1b7dd78dcce64a6df1bf947527c50289bd35d5aa45cb8860d"}
Jul 27 17:40:20.684910 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:20.684875    2573 generic.go:356] "Generic (PLEG): container finished" podID="37948065-229b-415a-80e9-80135ce60bd3" containerID="28174831a9263f99e31baa088c887ed3d8c3f8bbf73f7753588fc7a10af7646b" exitCode=0
Jul 27 17:40:20.685312 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:20.684951    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d" event={"ID":"37948065-229b-415a-80e9-80135ce60bd3","Type":"ContainerDied","Data":"28174831a9263f99e31baa088c887ed3d8c3f8bbf73f7753588fc7a10af7646b"}
Jul 27 17:40:20.686479 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:20.686457    2573 generic.go:356] "Generic (PLEG): container finished" podID="100e1c22-8fa3-439b-bc52-5ac62d3dfbf0" containerID="2b677af56b01b9aa9dad448bb2c25058e831a56e6de4201fbad4ae90eee1df27" exitCode=0
Jul 27 17:40:20.686600 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:20.686541    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz" event={"ID":"100e1c22-8fa3-439b-bc52-5ac62d3dfbf0","Type":"ContainerDied","Data":"2b677af56b01b9aa9dad448bb2c25058e831a56e6de4201fbad4ae90eee1df27"}
Jul 27 17:40:21.287695 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:21.287656    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d" podUID="37948065-229b-415a-80e9-80135ce60bd3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Jul 27 17:40:21.694019 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:21.693985    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz" event={"ID":"100e1c22-8fa3-439b-bc52-5ac62d3dfbf0","Type":"ContainerStarted","Data":"2c7001c2211f433e2ddc63a12ba0ff62236482384192dcab1067daf08cae309e"}
Jul 27 17:40:21.694430 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:21.694052    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz" event={"ID":"100e1c22-8fa3-439b-bc52-5ac62d3dfbf0","Type":"ContainerStarted","Data":"208c96e10323a8175db6f3642d1f631fb5d237ede88465bddec9db4e47631857"}
Jul 27 17:40:21.694430 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:21.694077    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz" event={"ID":"100e1c22-8fa3-439b-bc52-5ac62d3dfbf0","Type":"ContainerStarted","Data":"e5a8cdb1d64920cb87b6817fab3ce24d84fcb1b0ad4e40e862c6a84ed3d9f960"}
Jul 27 17:40:21.694561 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:21.694481    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz"
Jul 27 17:40:21.694603 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:21.694560    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz"
Jul 27 17:40:21.694603 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:21.694600    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz"
Jul 27 17:40:21.695905 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:21.695872    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz" podUID="100e1c22-8fa3-439b-bc52-5ac62d3dfbf0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:5000: connect: connection refused"
Jul 27 17:40:21.696640 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:21.696614    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz" podUID="100e1c22-8fa3-439b-bc52-5ac62d3dfbf0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 17:40:21.720394 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:21.720337    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz" podStartSLOduration=3.022600045 podStartE2EDuration="6.720320943s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.697720898s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:40:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:40:21.718780475 +0000 UTC m=+763.862182527" watchObservedRunningTime="2026-07-27 17:40:21.720320943 +0000 UTC m=+763.863723043"
Jul 27 17:40:22.291494 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:22.291445    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d" podUID="37948065-229b-415a-80e9-80135ce60bd3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 27 17:40:22.291795 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:22.291759    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d" podUID="37948065-229b-415a-80e9-80135ce60bd3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 17:40:22.698343 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:22.698301    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz" podUID="100e1c22-8fa3-439b-bc52-5ac62d3dfbf0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:5000: connect: connection refused"
Jul 27 17:40:22.698731 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:22.698704    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz" podUID="100e1c22-8fa3-439b-bc52-5ac62d3dfbf0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 17:40:26.287297 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:26.287217    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d" podUID="37948065-229b-415a-80e9-80135ce60bd3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Jul 27 17:40:26.287763 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:26.287420    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d"
Jul 27 17:40:27.702919 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:27.702889    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz"
Jul 27 17:40:27.703616 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:27.703585    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz" podUID="100e1c22-8fa3-439b-bc52-5ac62d3dfbf0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:5000: connect: connection refused"
Jul 27 17:40:27.703864 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:27.703843    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz" podUID="100e1c22-8fa3-439b-bc52-5ac62d3dfbf0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 17:40:31.287697 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:31.287650    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d" podUID="37948065-229b-415a-80e9-80135ce60bd3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Jul 27 17:40:32.291620 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:32.291573    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d" podUID="37948065-229b-415a-80e9-80135ce60bd3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 27 17:40:32.292021 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:32.291993    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d" podUID="37948065-229b-415a-80e9-80135ce60bd3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 17:40:36.286904 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:36.286865    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d" podUID="37948065-229b-415a-80e9-80135ce60bd3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Jul 27 17:40:37.703808 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:37.703768    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz" podUID="100e1c22-8fa3-439b-bc52-5ac62d3dfbf0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:5000: connect: connection refused"
Jul 27 17:40:37.704276 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:37.704231    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz" podUID="100e1c22-8fa3-439b-bc52-5ac62d3dfbf0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 17:40:41.287591 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:41.287548    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d" podUID="37948065-229b-415a-80e9-80135ce60bd3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Jul 27 17:40:42.292157 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:42.292102    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d" podUID="37948065-229b-415a-80e9-80135ce60bd3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 27 17:40:42.292659 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:42.292313    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d"
Jul 27 17:40:42.292659 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:42.292451    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d" podUID="37948065-229b-415a-80e9-80135ce60bd3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 17:40:42.292659 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:42.292577    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d"
Jul 27 17:40:45.806331 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:45.806297    2573 generic.go:356] "Generic (PLEG): container finished" podID="37948065-229b-415a-80e9-80135ce60bd3" containerID="ebe9c59bd18726fe145dbf7cfd9e72945ec03ff9341b5ea8b760887d490411a1" exitCode=0
Jul 27 17:40:45.806718 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:45.806370    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d" event={"ID":"37948065-229b-415a-80e9-80135ce60bd3","Type":"ContainerDied","Data":"ebe9c59bd18726fe145dbf7cfd9e72945ec03ff9341b5ea8b760887d490411a1"}
Jul 27 17:40:45.827710 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:45.827682    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d"
Jul 27 17:40:45.969191 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:45.969101    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37948065-229b-415a-80e9-80135ce60bd3-kube-rbac-proxy-sar-config\") pod \"37948065-229b-415a-80e9-80135ce60bd3\" (UID: \"37948065-229b-415a-80e9-80135ce60bd3\") "
Jul 27 17:40:45.969191 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:45.969175    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37948065-229b-415a-80e9-80135ce60bd3-proxy-tls\") pod \"37948065-229b-415a-80e9-80135ce60bd3\" (UID: \"37948065-229b-415a-80e9-80135ce60bd3\") "
Jul 27 17:40:45.969461 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:45.969200    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37948065-229b-415a-80e9-80135ce60bd3-kserve-provision-location\") pod \"37948065-229b-415a-80e9-80135ce60bd3\" (UID: \"37948065-229b-415a-80e9-80135ce60bd3\") "
Jul 27 17:40:45.969461 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:45.969217    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rdcgd\" (UniqueName: \"kubernetes.io/projected/37948065-229b-415a-80e9-80135ce60bd3-kube-api-access-rdcgd\") pod \"37948065-229b-415a-80e9-80135ce60bd3\" (UID: \"37948065-229b-415a-80e9-80135ce60bd3\") "
Jul 27 17:40:45.969461 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:45.969281    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37948065-229b-415a-80e9-80135ce60bd3-cabundle-cert\") pod \"37948065-229b-415a-80e9-80135ce60bd3\" (UID: \"37948065-229b-415a-80e9-80135ce60bd3\") "
Jul 27 17:40:45.969642 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:45.969579    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37948065-229b-415a-80e9-80135ce60bd3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "37948065-229b-415a-80e9-80135ce60bd3" (UID: "37948065-229b-415a-80e9-80135ce60bd3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:40:45.969642 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:45.969583    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37948065-229b-415a-80e9-80135ce60bd3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "37948065-229b-415a-80e9-80135ce60bd3" (UID: "37948065-229b-415a-80e9-80135ce60bd3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:40:45.969812 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:45.969781    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37948065-229b-415a-80e9-80135ce60bd3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "37948065-229b-415a-80e9-80135ce60bd3" (UID: "37948065-229b-415a-80e9-80135ce60bd3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:40:45.971551 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:45.971526    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37948065-229b-415a-80e9-80135ce60bd3-kube-api-access-rdcgd" (OuterVolumeSpecName: "kube-api-access-rdcgd") pod "37948065-229b-415a-80e9-80135ce60bd3" (UID: "37948065-229b-415a-80e9-80135ce60bd3"). InnerVolumeSpecName "kube-api-access-rdcgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:40:45.971640 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:45.971529    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37948065-229b-415a-80e9-80135ce60bd3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "37948065-229b-415a-80e9-80135ce60bd3" (UID: "37948065-229b-415a-80e9-80135ce60bd3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:40:46.070358 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:46.070320    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37948065-229b-415a-80e9-80135ce60bd3-proxy-tls\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:40:46.070358 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:46.070355    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37948065-229b-415a-80e9-80135ce60bd3-kserve-provision-location\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:40:46.070358 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:46.070366    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdcgd\" (UniqueName: \"kubernetes.io/projected/37948065-229b-415a-80e9-80135ce60bd3-kube-api-access-rdcgd\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:40:46.070598 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:46.070375    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37948065-229b-415a-80e9-80135ce60bd3-cabundle-cert\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:40:46.070598 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:46.070384    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37948065-229b-415a-80e9-80135ce60bd3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:40:46.813583 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:46.813503    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d"
Jul 27 17:40:46.814038 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:46.813502    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d" event={"ID":"37948065-229b-415a-80e9-80135ce60bd3","Type":"ContainerDied","Data":"0ec7e240d4c87b2d67e17ac5eabd64961f13053dcf286e0c07f7b42c34901048"}
Jul 27 17:40:46.814038 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:46.813648    2573 scope.go:117] "RemoveContainer" containerID="ebe9c59bd18726fe145dbf7cfd9e72945ec03ff9341b5ea8b760887d490411a1"
Jul 27 17:40:46.821764 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:46.821744    2573 scope.go:117] "RemoveContainer" containerID="5f91a2b9b5166351c1da8d2b83e3386ad807ba65c8104f6af87a5915caa1fbb1"
Jul 27 17:40:46.829145 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:46.829122    2573 scope.go:117] "RemoveContainer" containerID="28174831a9263f99e31baa088c887ed3d8c3f8bbf73f7753588fc7a10af7646b"
Jul 27 17:40:46.837013 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:46.836989    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d"]
Jul 27 17:40:46.837940 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:46.837922    2573 scope.go:117] "RemoveContainer" containerID="261576af76f625798cba3895ccb75b164d0b0b0cbc55b17bb7bdec184a674f5e"
Jul 27 17:40:46.840450 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:46.840428    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-86cb8f94cb-7fz4d"]
Jul 27 17:40:47.704266 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:47.704215    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz" podUID="100e1c22-8fa3-439b-bc52-5ac62d3dfbf0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:5000: connect: connection refused"
Jul 27 17:40:47.704718 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:47.704692    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz" podUID="100e1c22-8fa3-439b-bc52-5ac62d3dfbf0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 17:40:48.470259 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:48.470214    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37948065-229b-415a-80e9-80135ce60bd3" path="/var/lib/kubelet/pods/37948065-229b-415a-80e9-80135ce60bd3/volumes"
Jul 27 17:40:57.703489 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:57.703445    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz" podUID="100e1c22-8fa3-439b-bc52-5ac62d3dfbf0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:5000: connect: connection refused"
Jul 27 17:40:57.703998 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:40:57.703846    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz" podUID="100e1c22-8fa3-439b-bc52-5ac62d3dfbf0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 17:41:03.575574 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:41:03.575492    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:41:03.641767 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:41:03.641728    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:41:07.703725 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:41:07.703683    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz" podUID="100e1c22-8fa3-439b-bc52-5ac62d3dfbf0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:5000: connect: connection refused"
Jul 27 17:41:07.704190 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:41:07.704162    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz" podUID="100e1c22-8fa3-439b-bc52-5ac62d3dfbf0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 17:41:17.704153 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:41:17.704112    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz" podUID="100e1c22-8fa3-439b-bc52-5ac62d3dfbf0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:5000: connect: connection refused"
Jul 27 17:41:17.704649 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:41:17.704597    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz" podUID="100e1c22-8fa3-439b-bc52-5ac62d3dfbf0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 17:41:27.704473 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:41:27.704428    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz" podUID="100e1c22-8fa3-439b-bc52-5ac62d3dfbf0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:5000: connect: connection refused"
Jul 27 17:41:27.704966 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:41:27.704943    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz" podUID="100e1c22-8fa3-439b-bc52-5ac62d3dfbf0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 17:41:37.704292 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:41:37.704239    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz"
Jul 27 17:41:37.704752 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:41:37.704382    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz"
Jul 27 17:41:50.793740 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:41:50.793703    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:41:50.839715 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:41:50.839675    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mfbmn"]
Jul 27 17:41:50.840194 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:41:50.840171    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37948065-229b-415a-80e9-80135ce60bd3" containerName="kserve-container"
Jul 27 17:41:50.840194 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:41:50.840193    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="37948065-229b-415a-80e9-80135ce60bd3" containerName="kserve-container"
Jul 27 17:41:50.840408 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:41:50.840213    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37948065-229b-415a-80e9-80135ce60bd3" containerName="kube-rbac-proxy"
Jul 27 17:41:50.840408 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:41:50.840222    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="37948065-229b-415a-80e9-80135ce60bd3" containerName="kube-rbac-proxy"
Jul 27 17:41:50.840408 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:41:50.840264    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37948065-229b-415a-80e9-80135ce60bd3" containerName="agent"
Jul 27 17:41:50.840408 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:41:50.840274    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="37948065-229b-415a-80e9-80135ce60bd3" containerName="agent"
Jul 27 17:41:50.840408 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:41:50.840301    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37948065-229b-415a-80e9-80135ce60bd3" containerName="storage-initializer"
Jul 27 17:41:50.840408 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:41:50.840311    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="37948065-229b-415a-80e9-80135ce60bd3" containerName="storage-initializer"
Jul 27 17:41:50.840408 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:41:50.840395    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="37948065-229b-415a-80e9-80135ce60bd3" containerName="kube-rbac-proxy"
Jul 27 17:41:50.840408 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:41:50.840407    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="37948065-229b-415a-80e9-80135ce60bd3" containerName="agent"
Jul 27 17:41:50.840711 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:41:50.840417    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="37948065-229b-415a-80e9-80135ce60bd3" containerName="kserve-container"
Jul 27 17:41:50.844807 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:41:50.844785    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mfbmn"
Jul 27 17:41:50.847601 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:41:50.847579    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Jul 27 17:41:50.847720 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:41:50.847619    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Jul 27 17:41:50.854198 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:41:50.854166    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mfbmn"]
Jul 27 17:41:50.860469 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:41:50.860442    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz"]
Jul 27 17:41:50.860767 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:41:50.860742    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz" podUID="100e1c22-8fa3-439b-bc52-5ac62d3dfbf0" containerName="kserve-container" containerID="cri-o://e5a8cdb1d64920cb87b6817fab3ce24d84fcb1b0ad4e40e862c6a84ed3d9f960" gracePeriod=30
Jul 27 17:41:50.860868 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:41:50.860811    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz" podUID="100e1c22-8fa3-439b-bc52-5ac62d3dfbf0" containerName="agent" containerID="cri-o://2c7001c2211f433e2ddc63a12ba0ff62236482384192dcab1067daf08cae309e" gracePeriod=30
Jul 27 17:41:50.860868 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:41:50.860812    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz" podUID="100e1c22-8fa3-439b-bc52-5ac62d3dfbf0" containerName="kube-rbac-proxy" containerID="cri-o://208c96e10323a8175db6f3642d1f631fb5d237ede88465bddec9db4e47631857" gracePeriod=30
Jul 27 17:41:50.869901 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:41:50.869871    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:41:50.920743 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:41:50.920710    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8bfg\" (UniqueName: \"kubernetes.io/projected/8cd54928-6fd8-4a48-b5ba-691799e5c214-kube-api-access-d8bfg\") pod \"message-dumper-predictor-6575bcfdfd-mfbmn\" (UID: \"8cd54928-6fd8-4a48-b5ba-691799e5c214\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mfbmn"
Jul 27 17:41:50.920929 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:41:50.920767    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8cd54928-6fd8-4a48-b5ba-691799e5c214-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-mfbmn\" (UID: \"8cd54928-6fd8-4a48-b5ba-691799e5c214\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mfbmn"
Jul 27 17:41:50.920929 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:41:50.920786    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8cd54928-6fd8-4a48-b5ba-691799e5c214-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-mfbmn\" (UID: \"8cd54928-6fd8-4a48-b5ba-691799e5c214\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mfbmn"
Jul 27 17:41:51.021901 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:41:51.021861    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8cd54928-6fd8-4a48-b5ba-691799e5c214-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-mfbmn\" (UID: \"8cd54928-6fd8-4a48-b5ba-691799e5c214\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mfbmn"
Jul 27 17:41:51.021901 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:41:51.021901    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8cd54928-6fd8-4a48-b5ba-691799e5c214-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-mfbmn\" (UID: \"8cd54928-6fd8-4a48-b5ba-691799e5c214\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mfbmn"
Jul 27 17:41:51.022125 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:41:51.021957    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8bfg\" (UniqueName: \"kubernetes.io/projected/8cd54928-6fd8-4a48-b5ba-691799e5c214-kube-api-access-d8bfg\") pod \"message-dumper-predictor-6575bcfdfd-mfbmn\" (UID: \"8cd54928-6fd8-4a48-b5ba-691799e5c214\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mfbmn"
Jul 27 17:41:51.022665 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:41:51.022634    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8cd54928-6fd8-4a48-b5ba-691799e5c214-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-mfbmn\" (UID: \"8cd54928-6fd8-4a48-b5ba-691799e5c214\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mfbmn"
Jul 27 17:41:51.024679 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:41:51.024655    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8cd54928-6fd8-4a48-b5ba-691799e5c214-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-mfbmn\" (UID: \"8cd54928-6fd8-4a48-b5ba-691799e5c214\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mfbmn"
Jul 27 17:41:51.033596 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:41:51.033574    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8bfg\" (UniqueName: \"kubernetes.io/projected/8cd54928-6fd8-4a48-b5ba-691799e5c214-kube-api-access-d8bfg\") pod \"message-dumper-predictor-6575bcfdfd-mfbmn\" (UID: \"8cd54928-6fd8-4a48-b5ba-691799e5c214\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mfbmn"
Jul 27 17:41:51.049393 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:41:51.049307    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:41:51.103138 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:41:51.103108    2573 generic.go:356] "Generic (PLEG): container finished" podID="100e1c22-8fa3-439b-bc52-5ac62d3dfbf0" containerID="208c96e10323a8175db6f3642d1f631fb5d237ede88465bddec9db4e47631857" exitCode=2
Jul 27 17:41:51.103331 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:41:51.103175    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz" event={"ID":"100e1c22-8fa3-439b-bc52-5ac62d3dfbf0","Type":"ContainerDied","Data":"208c96e10323a8175db6f3642d1f631fb5d237ede88465bddec9db4e47631857"}
Jul 27 17:41:51.157131 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:41:51.157092    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mfbmn"
Jul 27 17:41:51.297617 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:41:51.297587    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mfbmn"]
Jul 27 17:41:51.300098 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:41:51.300029    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cd54928_6fd8_4a48_b5ba_691799e5c214.slice/crio-7e3660cce62e1457c895ea21427b4ebe97e98e5fc768deaf89b94339897886c0 WatchSource:0}: Error finding container 7e3660cce62e1457c895ea21427b4ebe97e98e5fc768deaf89b94339897886c0: Status 404 returned error can't find the container with id 7e3660cce62e1457c895ea21427b4ebe97e98e5fc768deaf89b94339897886c0
Jul 27 17:41:52.108418 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:41:52.108383    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mfbmn" event={"ID":"8cd54928-6fd8-4a48-b5ba-691799e5c214","Type":"ContainerStarted","Data":"7e3660cce62e1457c895ea21427b4ebe97e98e5fc768deaf89b94339897886c0"}
Jul 27 17:41:52.699038 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:41:52.699008    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz" podUID="100e1c22-8fa3-439b-bc52-5ac62d3dfbf0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 27 17:41:53.114536 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:41:53.114494    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mfbmn" event={"ID":"8cd54928-6fd8-4a48-b5ba-691799e5c214","Type":"ContainerStarted","Data":"6d8e799564fa23814a38911cc65e0a239af83d5d433fa0513e216fc46450433c"}
Jul 27 17:41:53.114975 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:41:53.114566    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mfbmn" event={"ID":"8cd54928-6fd8-4a48-b5ba-691799e5c214","Type":"ContainerStarted","Data":"894718fb4505d6916dc65fe57da59ad641501b95b070bc3f9b28201d16991e35"}
Jul 27 17:41:53.114975 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:41:53.114651    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mfbmn"
Jul 27 17:41:53.135508 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:41:53.135461    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mfbmn" podStartSLOduration=1.830113396 podStartE2EDuration="3.135444566s" totalImagesPullingTime="1.30533117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:41:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:41:53.133821854 +0000 UTC m=+855.277223905" watchObservedRunningTime="2026-07-27 17:41:53.135444566 +0000 UTC m=+855.278846616"
Jul 27 17:41:54.119228 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:41:54.119193    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mfbmn"
Jul 27 17:41:54.121177 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:41:54.121156    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mfbmn"
Jul 27 17:41:56.130612 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:41:56.130530    2573 generic.go:356] "Generic (PLEG): container finished" podID="100e1c22-8fa3-439b-bc52-5ac62d3dfbf0" containerID="e5a8cdb1d64920cb87b6817fab3ce24d84fcb1b0ad4e40e862c6a84ed3d9f960" exitCode=0
Jul 27 17:41:56.130989 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:41:56.130605    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz" event={"ID":"100e1c22-8fa3-439b-bc52-5ac62d3dfbf0","Type":"ContainerDied","Data":"e5a8cdb1d64920cb87b6817fab3ce24d84fcb1b0ad4e40e862c6a84ed3d9f960"}
Jul 27 17:41:57.698822 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:41:57.698782    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz" podUID="100e1c22-8fa3-439b-bc52-5ac62d3dfbf0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 27 17:41:57.704054 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:41:57.704024    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz" podUID="100e1c22-8fa3-439b-bc52-5ac62d3dfbf0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:5000: connect: connection refused"
Jul 27 17:41:57.704392 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:41:57.704369    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz" podUID="100e1c22-8fa3-439b-bc52-5ac62d3dfbf0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 17:42:01.135469 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:01.135439    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mfbmn"
Jul 27 17:42:02.699140 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:02.699094    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz" podUID="100e1c22-8fa3-439b-bc52-5ac62d3dfbf0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 27 17:42:02.699601 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:02.699244    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz"
Jul 27 17:42:07.698634 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:07.698585    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz" podUID="100e1c22-8fa3-439b-bc52-5ac62d3dfbf0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 27 17:42:07.703920 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:07.703886    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz" podUID="100e1c22-8fa3-439b-bc52-5ac62d3dfbf0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:5000: connect: connection refused"
Jul 27 17:42:07.704221 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:07.704196    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz" podUID="100e1c22-8fa3-439b-bc52-5ac62d3dfbf0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 17:42:10.848030 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:10.847994    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:42:10.900313 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:10.900273    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk"]
Jul 27 17:42:10.906828 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:10.906809    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk"
Jul 27 17:42:10.909729 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:10.909705    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Jul 27 17:42:10.909862 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:10.909813    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Jul 27 17:42:10.916490 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:10.916468    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk"]
Jul 27 17:42:10.991679 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:10.991642    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f91da0c-d541-465e-a81d-b0f7d063b361-cabundle-cert\") pod \"isvc-logger-predictor-6b6746bb7-ww2wk\" (UID: \"9f91da0c-d541-465e-a81d-b0f7d063b361\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk"
Jul 27 17:42:10.991854 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:10.991686    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f91da0c-d541-465e-a81d-b0f7d063b361-proxy-tls\") pod \"isvc-logger-predictor-6b6746bb7-ww2wk\" (UID: \"9f91da0c-d541-465e-a81d-b0f7d063b361\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk"
Jul 27 17:42:10.991854 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:10.991723    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f91da0c-d541-465e-a81d-b0f7d063b361-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-6b6746bb7-ww2wk\" (UID: \"9f91da0c-d541-465e-a81d-b0f7d063b361\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk"
Jul 27 17:42:10.991854 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:10.991760    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f91da0c-d541-465e-a81d-b0f7d063b361-kserve-provision-location\") pod \"isvc-logger-predictor-6b6746bb7-ww2wk\" (UID: \"9f91da0c-d541-465e-a81d-b0f7d063b361\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk"
Jul 27 17:42:10.991854 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:10.991843    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msm8p\" (UniqueName: \"kubernetes.io/projected/9f91da0c-d541-465e-a81d-b0f7d063b361-kube-api-access-msm8p\") pod \"isvc-logger-predictor-6b6746bb7-ww2wk\" (UID: \"9f91da0c-d541-465e-a81d-b0f7d063b361\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk"
Jul 27 17:42:11.092921 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:11.092885    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msm8p\" (UniqueName: \"kubernetes.io/projected/9f91da0c-d541-465e-a81d-b0f7d063b361-kube-api-access-msm8p\") pod \"isvc-logger-predictor-6b6746bb7-ww2wk\" (UID: \"9f91da0c-d541-465e-a81d-b0f7d063b361\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk"
Jul 27 17:42:11.093273 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:11.092943    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f91da0c-d541-465e-a81d-b0f7d063b361-cabundle-cert\") pod \"isvc-logger-predictor-6b6746bb7-ww2wk\" (UID: \"9f91da0c-d541-465e-a81d-b0f7d063b361\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk"
Jul 27 17:42:11.093273 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:11.092971    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f91da0c-d541-465e-a81d-b0f7d063b361-proxy-tls\") pod \"isvc-logger-predictor-6b6746bb7-ww2wk\" (UID: \"9f91da0c-d541-465e-a81d-b0f7d063b361\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk"
Jul 27 17:42:11.093273 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:11.092991    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f91da0c-d541-465e-a81d-b0f7d063b361-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-6b6746bb7-ww2wk\" (UID: \"9f91da0c-d541-465e-a81d-b0f7d063b361\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk"
Jul 27 17:42:11.093273 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:11.093015    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f91da0c-d541-465e-a81d-b0f7d063b361-kserve-provision-location\") pod \"isvc-logger-predictor-6b6746bb7-ww2wk\" (UID: \"9f91da0c-d541-465e-a81d-b0f7d063b361\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk"
Jul 27 17:42:11.093273 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:42:11.093136    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-predictor-serving-cert: secret "isvc-logger-predictor-serving-cert" not found
Jul 27 17:42:11.093273 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:42:11.093231    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f91da0c-d541-465e-a81d-b0f7d063b361-proxy-tls podName:9f91da0c-d541-465e-a81d-b0f7d063b361 nodeName:}" failed. No retries permitted until 2026-07-27 17:42:11.59320957 +0000 UTC m=+873.736611601 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9f91da0c-d541-465e-a81d-b0f7d063b361-proxy-tls") pod "isvc-logger-predictor-6b6746bb7-ww2wk" (UID: "9f91da0c-d541-465e-a81d-b0f7d063b361") : secret "isvc-logger-predictor-serving-cert" not found
Jul 27 17:42:11.093589 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:11.093555    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f91da0c-d541-465e-a81d-b0f7d063b361-kserve-provision-location\") pod \"isvc-logger-predictor-6b6746bb7-ww2wk\" (UID: \"9f91da0c-d541-465e-a81d-b0f7d063b361\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk"
Jul 27 17:42:11.093827 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:11.093808    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f91da0c-d541-465e-a81d-b0f7d063b361-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-6b6746bb7-ww2wk\" (UID: \"9f91da0c-d541-465e-a81d-b0f7d063b361\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk"
Jul 27 17:42:11.093868 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:11.093827    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f91da0c-d541-465e-a81d-b0f7d063b361-cabundle-cert\") pod \"isvc-logger-predictor-6b6746bb7-ww2wk\" (UID: \"9f91da0c-d541-465e-a81d-b0f7d063b361\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk"
Jul 27 17:42:11.104456 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:11.104404    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msm8p\" (UniqueName: \"kubernetes.io/projected/9f91da0c-d541-465e-a81d-b0f7d063b361-kube-api-access-msm8p\") pod \"isvc-logger-predictor-6b6746bb7-ww2wk\" (UID: \"9f91da0c-d541-465e-a81d-b0f7d063b361\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk"
Jul 27 17:42:11.328568 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:11.328535    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:42:11.598697 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:11.598655    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f91da0c-d541-465e-a81d-b0f7d063b361-proxy-tls\") pod \"isvc-logger-predictor-6b6746bb7-ww2wk\" (UID: \"9f91da0c-d541-465e-a81d-b0f7d063b361\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk"
Jul 27 17:42:11.601109 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:11.601087    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f91da0c-d541-465e-a81d-b0f7d063b361-proxy-tls\") pod \"isvc-logger-predictor-6b6746bb7-ww2wk\" (UID: \"9f91da0c-d541-465e-a81d-b0f7d063b361\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk"
Jul 27 17:42:11.818014 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:11.817979    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk"
Jul 27 17:42:11.956687 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:11.956661    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk"]
Jul 27 17:42:11.959348 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:42:11.959319    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f91da0c_d541_465e_a81d_b0f7d063b361.slice/crio-c5af4be8973179051b603b1a98598b6c563d1d79a7e5ed8398df1e93956435ca WatchSource:0}: Error finding container c5af4be8973179051b603b1a98598b6c563d1d79a7e5ed8398df1e93956435ca: Status 404 returned error can't find the container with id c5af4be8973179051b603b1a98598b6c563d1d79a7e5ed8398df1e93956435ca
Jul 27 17:42:12.210083 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:12.210040    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk" event={"ID":"9f91da0c-d541-465e-a81d-b0f7d063b361","Type":"ContainerStarted","Data":"99e80a31306f3b926b977908e6d0049a6ee488a9f2fa08d860eaf4235869c796"}
Jul 27 17:42:12.210266 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:12.210134    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk" event={"ID":"9f91da0c-d541-465e-a81d-b0f7d063b361","Type":"ContainerStarted","Data":"c5af4be8973179051b603b1a98598b6c563d1d79a7e5ed8398df1e93956435ca"}
Jul 27 17:42:12.698447 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:12.698407    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz" podUID="100e1c22-8fa3-439b-bc52-5ac62d3dfbf0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 27 17:42:16.229318 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:16.229285    2573 generic.go:356] "Generic (PLEG): container finished" podID="9f91da0c-d541-465e-a81d-b0f7d063b361" containerID="99e80a31306f3b926b977908e6d0049a6ee488a9f2fa08d860eaf4235869c796" exitCode=0
Jul 27 17:42:16.229700 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:16.229359    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk" event={"ID":"9f91da0c-d541-465e-a81d-b0f7d063b361","Type":"ContainerDied","Data":"99e80a31306f3b926b977908e6d0049a6ee488a9f2fa08d860eaf4235869c796"}
Jul 27 17:42:17.237461 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:17.237426    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk" event={"ID":"9f91da0c-d541-465e-a81d-b0f7d063b361","Type":"ContainerStarted","Data":"56b61c780753ba29c4985cb9928dbe84e69d984104e49b699ddad0280d32f6ce"}
Jul 27 17:42:17.237997 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:17.237498    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk" event={"ID":"9f91da0c-d541-465e-a81d-b0f7d063b361","Type":"ContainerStarted","Data":"f81bbf008642e8dfbc7fe4e4871aef73324d3b1cd8b8fbea12a864d15a6b889c"}
Jul 27 17:42:17.237997 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:17.237527    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk" event={"ID":"9f91da0c-d541-465e-a81d-b0f7d063b361","Type":"ContainerStarted","Data":"c7ba02a430567f46b564290d2ed4eeb9260874af31593af3922534f3d910f961"}
Jul 27 17:42:17.237997 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:17.237751    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk"
Jul 27 17:42:17.237997 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:17.237819    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk"
Jul 27 17:42:17.239399 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:17.239371    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk" podUID="9f91da0c-d541-465e-a81d-b0f7d063b361" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 27 17:42:17.264900 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:17.264848    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk" podStartSLOduration=3.647660112 podStartE2EDuration="7.264831253s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.617171141s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:42:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:42:17.263002004 +0000 UTC m=+879.406404055" watchObservedRunningTime="2026-07-27 17:42:17.264831253 +0000 UTC m=+879.408233303"
Jul 27 17:42:17.698450 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:17.698406    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz" podUID="100e1c22-8fa3-439b-bc52-5ac62d3dfbf0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 27 17:42:17.703762 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:17.703733    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz" podUID="100e1c22-8fa3-439b-bc52-5ac62d3dfbf0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:5000: connect: connection refused"
Jul 27 17:42:17.703934 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:17.703918    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz"
Jul 27 17:42:17.704160 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:17.704137    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz" podUID="100e1c22-8fa3-439b-bc52-5ac62d3dfbf0" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 17:42:17.704299 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:17.704280    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz"
Jul 27 17:42:18.242692 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:18.242655    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk"
Jul 27 17:42:18.243176 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:18.242690    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk" podUID="9f91da0c-d541-465e-a81d-b0f7d063b361" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 27 17:42:18.243998 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:18.243968    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk" podUID="9f91da0c-d541-465e-a81d-b0f7d063b361" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 17:42:19.247108 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:19.247066    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk" podUID="9f91da0c-d541-465e-a81d-b0f7d063b361" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 27 17:42:19.247633 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:19.247609    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk" podUID="9f91da0c-d541-465e-a81d-b0f7d063b361" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 17:42:21.017996 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:21.017966    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz"
Jul 27 17:42:21.081299 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:21.081242    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/100e1c22-8fa3-439b-bc52-5ac62d3dfbf0-cabundle-cert\") pod \"100e1c22-8fa3-439b-bc52-5ac62d3dfbf0\" (UID: \"100e1c22-8fa3-439b-bc52-5ac62d3dfbf0\") "
Jul 27 17:42:21.081481 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:21.081305    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/100e1c22-8fa3-439b-bc52-5ac62d3dfbf0-kube-rbac-proxy-sar-config\") pod \"100e1c22-8fa3-439b-bc52-5ac62d3dfbf0\" (UID: \"100e1c22-8fa3-439b-bc52-5ac62d3dfbf0\") "
Jul 27 17:42:21.081481 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:21.081387    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/100e1c22-8fa3-439b-bc52-5ac62d3dfbf0-kserve-provision-location\") pod \"100e1c22-8fa3-439b-bc52-5ac62d3dfbf0\" (UID: \"100e1c22-8fa3-439b-bc52-5ac62d3dfbf0\") "
Jul 27 17:42:21.081481 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:21.081415    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/100e1c22-8fa3-439b-bc52-5ac62d3dfbf0-proxy-tls\") pod \"100e1c22-8fa3-439b-bc52-5ac62d3dfbf0\" (UID: \"100e1c22-8fa3-439b-bc52-5ac62d3dfbf0\") "
Jul 27 17:42:21.081730 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:21.081498    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjcbf\" (UniqueName: \"kubernetes.io/projected/100e1c22-8fa3-439b-bc52-5ac62d3dfbf0-kube-api-access-qjcbf\") pod \"100e1c22-8fa3-439b-bc52-5ac62d3dfbf0\" (UID: \"100e1c22-8fa3-439b-bc52-5ac62d3dfbf0\") "
Jul 27 17:42:21.081730 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:21.081646    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/100e1c22-8fa3-439b-bc52-5ac62d3dfbf0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "100e1c22-8fa3-439b-bc52-5ac62d3dfbf0" (UID: "100e1c22-8fa3-439b-bc52-5ac62d3dfbf0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:42:21.081730 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:21.081693    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/100e1c22-8fa3-439b-bc52-5ac62d3dfbf0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "100e1c22-8fa3-439b-bc52-5ac62d3dfbf0" (UID: "100e1c22-8fa3-439b-bc52-5ac62d3dfbf0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:42:21.081730 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:21.081693    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/100e1c22-8fa3-439b-bc52-5ac62d3dfbf0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "100e1c22-8fa3-439b-bc52-5ac62d3dfbf0" (UID: "100e1c22-8fa3-439b-bc52-5ac62d3dfbf0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:42:21.081879 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:21.081743    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/100e1c22-8fa3-439b-bc52-5ac62d3dfbf0-cabundle-cert\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:42:21.083625 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:21.083596    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/100e1c22-8fa3-439b-bc52-5ac62d3dfbf0-kube-api-access-qjcbf" (OuterVolumeSpecName: "kube-api-access-qjcbf") pod "100e1c22-8fa3-439b-bc52-5ac62d3dfbf0" (UID: "100e1c22-8fa3-439b-bc52-5ac62d3dfbf0"). InnerVolumeSpecName "kube-api-access-qjcbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:42:21.083625 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:21.083615    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/100e1c22-8fa3-439b-bc52-5ac62d3dfbf0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "100e1c22-8fa3-439b-bc52-5ac62d3dfbf0" (UID: "100e1c22-8fa3-439b-bc52-5ac62d3dfbf0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:42:21.183035 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:21.182999    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/100e1c22-8fa3-439b-bc52-5ac62d3dfbf0-kserve-provision-location\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:42:21.183035 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:21.183030    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/100e1c22-8fa3-439b-bc52-5ac62d3dfbf0-proxy-tls\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:42:21.183035 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:21.183042    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjcbf\" (UniqueName: \"kubernetes.io/projected/100e1c22-8fa3-439b-bc52-5ac62d3dfbf0-kube-api-access-qjcbf\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:42:21.183305 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:21.183052    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/100e1c22-8fa3-439b-bc52-5ac62d3dfbf0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:42:21.257230 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:21.257192    2573 generic.go:356] "Generic (PLEG): container finished" podID="100e1c22-8fa3-439b-bc52-5ac62d3dfbf0" containerID="2c7001c2211f433e2ddc63a12ba0ff62236482384192dcab1067daf08cae309e" exitCode=0
Jul 27 17:42:21.257385 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:21.257226    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz" event={"ID":"100e1c22-8fa3-439b-bc52-5ac62d3dfbf0","Type":"ContainerDied","Data":"2c7001c2211f433e2ddc63a12ba0ff62236482384192dcab1067daf08cae309e"}
Jul 27 17:42:21.257385 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:21.257272    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz"
Jul 27 17:42:21.257385 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:21.257311    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz" event={"ID":"100e1c22-8fa3-439b-bc52-5ac62d3dfbf0","Type":"ContainerDied","Data":"381b71b24bcd24e1b7dd78dcce64a6df1bf947527c50289bd35d5aa45cb8860d"}
Jul 27 17:42:21.257385 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:21.257343    2573 scope.go:117] "RemoveContainer" containerID="2c7001c2211f433e2ddc63a12ba0ff62236482384192dcab1067daf08cae309e"
Jul 27 17:42:21.265729 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:21.265707    2573 scope.go:117] "RemoveContainer" containerID="208c96e10323a8175db6f3642d1f631fb5d237ede88465bddec9db4e47631857"
Jul 27 17:42:21.273271 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:21.273226    2573 scope.go:117] "RemoveContainer" containerID="e5a8cdb1d64920cb87b6817fab3ce24d84fcb1b0ad4e40e862c6a84ed3d9f960"
Jul 27 17:42:21.280888 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:21.280823    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz"]
Jul 27 17:42:21.280941 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:21.280908    2573 scope.go:117] "RemoveContainer" containerID="2b677af56b01b9aa9dad448bb2c25058e831a56e6de4201fbad4ae90eee1df27"
Jul 27 17:42:21.288289 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:21.288239    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-55c4fb9c75-j9nmz"]
Jul 27 17:42:21.289184 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:21.289163    2573 scope.go:117] "RemoveContainer" containerID="2c7001c2211f433e2ddc63a12ba0ff62236482384192dcab1067daf08cae309e"
Jul 27 17:42:21.289526 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:42:21.289505    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c7001c2211f433e2ddc63a12ba0ff62236482384192dcab1067daf08cae309e\": container with ID starting with 2c7001c2211f433e2ddc63a12ba0ff62236482384192dcab1067daf08cae309e not found: ID does not exist" containerID="2c7001c2211f433e2ddc63a12ba0ff62236482384192dcab1067daf08cae309e"
Jul 27 17:42:21.289605 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:21.289534    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c7001c2211f433e2ddc63a12ba0ff62236482384192dcab1067daf08cae309e"} err="failed to get container status \"2c7001c2211f433e2ddc63a12ba0ff62236482384192dcab1067daf08cae309e\": rpc error: code = NotFound desc = could not find container \"2c7001c2211f433e2ddc63a12ba0ff62236482384192dcab1067daf08cae309e\": container with ID starting with 2c7001c2211f433e2ddc63a12ba0ff62236482384192dcab1067daf08cae309e not found: ID does not exist"
Jul 27 17:42:21.289605 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:21.289554    2573 scope.go:117] "RemoveContainer" containerID="208c96e10323a8175db6f3642d1f631fb5d237ede88465bddec9db4e47631857"
Jul 27 17:42:21.289848 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:42:21.289821    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"208c96e10323a8175db6f3642d1f631fb5d237ede88465bddec9db4e47631857\": container with ID starting with 208c96e10323a8175db6f3642d1f631fb5d237ede88465bddec9db4e47631857 not found: ID does not exist" containerID="208c96e10323a8175db6f3642d1f631fb5d237ede88465bddec9db4e47631857"
Jul 27 17:42:21.289894 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:21.289859    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"208c96e10323a8175db6f3642d1f631fb5d237ede88465bddec9db4e47631857"} err="failed to get container status \"208c96e10323a8175db6f3642d1f631fb5d237ede88465bddec9db4e47631857\": rpc error: code = NotFound desc = could not find container \"208c96e10323a8175db6f3642d1f631fb5d237ede88465bddec9db4e47631857\": container with ID starting with 208c96e10323a8175db6f3642d1f631fb5d237ede88465bddec9db4e47631857 not found: ID does not exist"
Jul 27 17:42:21.289894 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:21.289884    2573 scope.go:117] "RemoveContainer" containerID="e5a8cdb1d64920cb87b6817fab3ce24d84fcb1b0ad4e40e862c6a84ed3d9f960"
Jul 27 17:42:21.290160 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:42:21.290137    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5a8cdb1d64920cb87b6817fab3ce24d84fcb1b0ad4e40e862c6a84ed3d9f960\": container with ID starting with e5a8cdb1d64920cb87b6817fab3ce24d84fcb1b0ad4e40e862c6a84ed3d9f960 not found: ID does not exist" containerID="e5a8cdb1d64920cb87b6817fab3ce24d84fcb1b0ad4e40e862c6a84ed3d9f960"
Jul 27 17:42:21.290279 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:21.290169    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5a8cdb1d64920cb87b6817fab3ce24d84fcb1b0ad4e40e862c6a84ed3d9f960"} err="failed to get container status \"e5a8cdb1d64920cb87b6817fab3ce24d84fcb1b0ad4e40e862c6a84ed3d9f960\": rpc error: code = NotFound desc = could not find container \"e5a8cdb1d64920cb87b6817fab3ce24d84fcb1b0ad4e40e862c6a84ed3d9f960\": container with ID starting with e5a8cdb1d64920cb87b6817fab3ce24d84fcb1b0ad4e40e862c6a84ed3d9f960 not found: ID does not exist"
Jul 27 17:42:21.290279 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:21.290184    2573 scope.go:117] "RemoveContainer" containerID="2b677af56b01b9aa9dad448bb2c25058e831a56e6de4201fbad4ae90eee1df27"
Jul 27 17:42:21.290528 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:42:21.290512    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b677af56b01b9aa9dad448bb2c25058e831a56e6de4201fbad4ae90eee1df27\": container with ID starting with 2b677af56b01b9aa9dad448bb2c25058e831a56e6de4201fbad4ae90eee1df27 not found: ID does not exist" containerID="2b677af56b01b9aa9dad448bb2c25058e831a56e6de4201fbad4ae90eee1df27"
Jul 27 17:42:21.290571 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:21.290533    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b677af56b01b9aa9dad448bb2c25058e831a56e6de4201fbad4ae90eee1df27"} err="failed to get container status \"2b677af56b01b9aa9dad448bb2c25058e831a56e6de4201fbad4ae90eee1df27\": rpc error: code = NotFound desc = could not find container \"2b677af56b01b9aa9dad448bb2c25058e831a56e6de4201fbad4ae90eee1df27\": container with ID starting with 2b677af56b01b9aa9dad448bb2c25058e831a56e6de4201fbad4ae90eee1df27 not found: ID does not exist"
Jul 27 17:42:22.474759 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:22.474722    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="100e1c22-8fa3-439b-bc52-5ac62d3dfbf0" path="/var/lib/kubelet/pods/100e1c22-8fa3-439b-bc52-5ac62d3dfbf0/volumes"
Jul 27 17:42:24.251780 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:24.251748    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk"
Jul 27 17:42:24.252477 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:24.252434    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk" podUID="9f91da0c-d541-465e-a81d-b0f7d063b361" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 27 17:42:24.252828 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:24.252798    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk" podUID="9f91da0c-d541-465e-a81d-b0f7d063b361" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 17:42:34.252554 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:34.252457    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk" podUID="9f91da0c-d541-465e-a81d-b0f7d063b361" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 27 17:42:34.252967 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:34.252945    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk" podUID="9f91da0c-d541-465e-a81d-b0f7d063b361" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 17:42:38.306431 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:38.306401    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vc2tz_87dac797-f8d2-4d36-999f-f8ce855dbeff/ovn-acl-logging/0.log"
Jul 27 17:42:38.307573 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:38.307549    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vc2tz_87dac797-f8d2-4d36-999f-f8ce855dbeff/ovn-acl-logging/0.log"
Jul 27 17:42:44.252584 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:44.252541    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk" podUID="9f91da0c-d541-465e-a81d-b0f7d063b361" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 27 17:42:44.253032 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:44.253006    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk" podUID="9f91da0c-d541-465e-a81d-b0f7d063b361" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 17:42:54.252632 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:54.252589    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk" podUID="9f91da0c-d541-465e-a81d-b0f7d063b361" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 27 17:42:54.255031 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:42:54.253096    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk" podUID="9f91da0c-d541-465e-a81d-b0f7d063b361" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 17:43:04.252446 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:04.252396    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk" podUID="9f91da0c-d541-465e-a81d-b0f7d063b361" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 27 17:43:04.252863 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:04.252756    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk" podUID="9f91da0c-d541-465e-a81d-b0f7d063b361" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 17:43:14.252526 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:14.252479    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk" podUID="9f91da0c-d541-465e-a81d-b0f7d063b361" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 27 17:43:14.253048 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:14.252969    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk" podUID="9f91da0c-d541-465e-a81d-b0f7d063b361" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 17:43:24.252667 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:24.252620    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk" podUID="9f91da0c-d541-465e-a81d-b0f7d063b361" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 27 17:43:24.253163 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:24.253121    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk" podUID="9f91da0c-d541-465e-a81d-b0f7d063b361" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 17:43:34.253461 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:34.253427    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk"
Jul 27 17:43:34.253852 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:34.253731    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk"
Jul 27 17:43:45.877324 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:45.877283    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-mfbmn_8cd54928-6fd8-4a48-b5ba-691799e5c214/kserve-container/0.log"
Jul 27 17:43:46.113780 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.113749    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk"]
Jul 27 17:43:46.114053 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.114004    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk" podUID="9f91da0c-d541-465e-a81d-b0f7d063b361" containerName="kserve-container" containerID="cri-o://c7ba02a430567f46b564290d2ed4eeb9260874af31593af3922534f3d910f961" gracePeriod=30
Jul 27 17:43:46.114211 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.114054    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk" podUID="9f91da0c-d541-465e-a81d-b0f7d063b361" containerName="kube-rbac-proxy" containerID="cri-o://f81bbf008642e8dfbc7fe4e4871aef73324d3b1cd8b8fbea12a864d15a6b889c" gracePeriod=30
Jul 27 17:43:46.114211 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.114026    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk" podUID="9f91da0c-d541-465e-a81d-b0f7d063b361" containerName="agent" containerID="cri-o://56b61c780753ba29c4985cb9928dbe84e69d984104e49b699ddad0280d32f6ce" gracePeriod=30
Jul 27 17:43:46.117610 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.117582    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:43:46.174749 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.174713    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxrp"]
Jul 27 17:43:46.175106 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.175092    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="100e1c22-8fa3-439b-bc52-5ac62d3dfbf0" containerName="storage-initializer"
Jul 27 17:43:46.175151 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.175110    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="100e1c22-8fa3-439b-bc52-5ac62d3dfbf0" containerName="storage-initializer"
Jul 27 17:43:46.175151 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.175121    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="100e1c22-8fa3-439b-bc52-5ac62d3dfbf0" containerName="kserve-container"
Jul 27 17:43:46.175151 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.175127    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="100e1c22-8fa3-439b-bc52-5ac62d3dfbf0" containerName="kserve-container"
Jul 27 17:43:46.175151 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.175134    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="100e1c22-8fa3-439b-bc52-5ac62d3dfbf0" containerName="agent"
Jul 27 17:43:46.175151 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.175140    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="100e1c22-8fa3-439b-bc52-5ac62d3dfbf0" containerName="agent"
Jul 27 17:43:46.175340 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.175159    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="100e1c22-8fa3-439b-bc52-5ac62d3dfbf0" containerName="kube-rbac-proxy"
Jul 27 17:43:46.175340 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.175164    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="100e1c22-8fa3-439b-bc52-5ac62d3dfbf0" containerName="kube-rbac-proxy"
Jul 27 17:43:46.175340 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.175221    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="100e1c22-8fa3-439b-bc52-5ac62d3dfbf0" containerName="kube-rbac-proxy"
Jul 27 17:43:46.175340 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.175229    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="100e1c22-8fa3-439b-bc52-5ac62d3dfbf0" containerName="agent"
Jul 27 17:43:46.175340 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.175236    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="100e1c22-8fa3-439b-bc52-5ac62d3dfbf0" containerName="kserve-container"
Jul 27 17:43:46.178365 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.178344    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxrp"
Jul 27 17:43:46.179396 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.179365    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:43:46.181101 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.181082    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Jul 27 17:43:46.181306 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.181292    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Jul 27 17:43:46.188285 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.188263    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxrp"]
Jul 27 17:43:46.231143 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.231113    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mfbmn"]
Jul 27 17:43:46.231458 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.231427    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mfbmn" podUID="8cd54928-6fd8-4a48-b5ba-691799e5c214" containerName="kserve-container" containerID="cri-o://894718fb4505d6916dc65fe57da59ad641501b95b070bc3f9b28201d16991e35" gracePeriod=30
Jul 27 17:43:46.231589 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.231484    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mfbmn" podUID="8cd54928-6fd8-4a48-b5ba-691799e5c214" containerName="kube-rbac-proxy" containerID="cri-o://6d8e799564fa23814a38911cc65e0a239af83d5d433fa0513e216fc46450433c" gracePeriod=30
Jul 27 17:43:46.239875 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.239847    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:43:46.284224 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.284186    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-fvxrp\" (UID: \"cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxrp"
Jul 27 17:43:46.284412 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.284242    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-fvxrp\" (UID: \"cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxrp"
Jul 27 17:43:46.284412 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.284319    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-fvxrp\" (UID: \"cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxrp"
Jul 27 17:43:46.284412 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.284360    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67ft4\" (UniqueName: \"kubernetes.io/projected/cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1-kube-api-access-67ft4\") pod \"isvc-autogluon-v1-predictor-574b5b7495-fvxrp\" (UID: \"cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxrp"
Jul 27 17:43:46.284412 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.284388    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-fvxrp\" (UID: \"cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxrp"
Jul 27 17:43:46.385198 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.385166    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-fvxrp\" (UID: \"cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxrp"
Jul 27 17:43:46.385381 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.385215    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67ft4\" (UniqueName: \"kubernetes.io/projected/cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1-kube-api-access-67ft4\") pod \"isvc-autogluon-v1-predictor-574b5b7495-fvxrp\" (UID: \"cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxrp"
Jul 27 17:43:46.385381 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.385352    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-fvxrp\" (UID: \"cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxrp"
Jul 27 17:43:46.385500 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.385415    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-fvxrp\" (UID: \"cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxrp"
Jul 27 17:43:46.385500 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.385478    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-fvxrp\" (UID: \"cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxrp"
Jul 27 17:43:46.385646 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:43:46.385613    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v1-predictor-serving-cert: secret "isvc-autogluon-v1-predictor-serving-cert" not found
Jul 27 17:43:46.385783 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:43:46.385695    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1-proxy-tls podName:cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1 nodeName:}" failed. No retries permitted until 2026-07-27 17:43:46.885673142 +0000 UTC m=+969.029075172 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1-proxy-tls") pod "isvc-autogluon-v1-predictor-574b5b7495-fvxrp" (UID: "cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1") : secret "isvc-autogluon-v1-predictor-serving-cert" not found
Jul 27 17:43:46.385858 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.385838    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-fvxrp\" (UID: \"cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxrp"
Jul 27 17:43:46.386067 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.386042    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-fvxrp\" (UID: \"cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxrp"
Jul 27 17:43:46.386147 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.386101    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-fvxrp\" (UID: \"cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxrp"
Jul 27 17:43:46.394875 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.394842    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67ft4\" (UniqueName: \"kubernetes.io/projected/cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1-kube-api-access-67ft4\") pod \"isvc-autogluon-v1-predictor-574b5b7495-fvxrp\" (UID: \"cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxrp"
Jul 27 17:43:46.444555 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.444459    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:43:46.496057 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.496034    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mfbmn"
Jul 27 17:43:46.587160 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.587107    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8bfg\" (UniqueName: \"kubernetes.io/projected/8cd54928-6fd8-4a48-b5ba-691799e5c214-kube-api-access-d8bfg\") pod \"8cd54928-6fd8-4a48-b5ba-691799e5c214\" (UID: \"8cd54928-6fd8-4a48-b5ba-691799e5c214\") "
Jul 27 17:43:46.587394 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.587206    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8cd54928-6fd8-4a48-b5ba-691799e5c214-proxy-tls\") pod \"8cd54928-6fd8-4a48-b5ba-691799e5c214\" (UID: \"8cd54928-6fd8-4a48-b5ba-691799e5c214\") "
Jul 27 17:43:46.587394 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.587233    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8cd54928-6fd8-4a48-b5ba-691799e5c214-kube-rbac-proxy-sar-config\") pod \"8cd54928-6fd8-4a48-b5ba-691799e5c214\" (UID: \"8cd54928-6fd8-4a48-b5ba-691799e5c214\") "
Jul 27 17:43:46.587601 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.587576    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cd54928-6fd8-4a48-b5ba-691799e5c214-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8cd54928-6fd8-4a48-b5ba-691799e5c214" (UID: "8cd54928-6fd8-4a48-b5ba-691799e5c214"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:43:46.589427 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.589398    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cd54928-6fd8-4a48-b5ba-691799e5c214-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8cd54928-6fd8-4a48-b5ba-691799e5c214" (UID: "8cd54928-6fd8-4a48-b5ba-691799e5c214"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:43:46.589496 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.589408    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cd54928-6fd8-4a48-b5ba-691799e5c214-kube-api-access-d8bfg" (OuterVolumeSpecName: "kube-api-access-d8bfg") pod "8cd54928-6fd8-4a48-b5ba-691799e5c214" (UID: "8cd54928-6fd8-4a48-b5ba-691799e5c214"). InnerVolumeSpecName "kube-api-access-d8bfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:43:46.657120 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.657081    2573 generic.go:356] "Generic (PLEG): container finished" podID="8cd54928-6fd8-4a48-b5ba-691799e5c214" containerID="6d8e799564fa23814a38911cc65e0a239af83d5d433fa0513e216fc46450433c" exitCode=2
Jul 27 17:43:46.657120 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.657112    2573 generic.go:356] "Generic (PLEG): container finished" podID="8cd54928-6fd8-4a48-b5ba-691799e5c214" containerID="894718fb4505d6916dc65fe57da59ad641501b95b070bc3f9b28201d16991e35" exitCode=2
Jul 27 17:43:46.657392 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.657137    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mfbmn"
Jul 27 17:43:46.657392 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.657164    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mfbmn" event={"ID":"8cd54928-6fd8-4a48-b5ba-691799e5c214","Type":"ContainerDied","Data":"6d8e799564fa23814a38911cc65e0a239af83d5d433fa0513e216fc46450433c"}
Jul 27 17:43:46.657392 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.657224    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mfbmn" event={"ID":"8cd54928-6fd8-4a48-b5ba-691799e5c214","Type":"ContainerDied","Data":"894718fb4505d6916dc65fe57da59ad641501b95b070bc3f9b28201d16991e35"}
Jul 27 17:43:46.657392 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.657261    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mfbmn" event={"ID":"8cd54928-6fd8-4a48-b5ba-691799e5c214","Type":"ContainerDied","Data":"7e3660cce62e1457c895ea21427b4ebe97e98e5fc768deaf89b94339897886c0"}
Jul 27 17:43:46.657392 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.657276    2573 scope.go:117] "RemoveContainer" containerID="6d8e799564fa23814a38911cc65e0a239af83d5d433fa0513e216fc46450433c"
Jul 27 17:43:46.659922 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.659898    2573 generic.go:356] "Generic (PLEG): container finished" podID="9f91da0c-d541-465e-a81d-b0f7d063b361" containerID="f81bbf008642e8dfbc7fe4e4871aef73324d3b1cd8b8fbea12a864d15a6b889c" exitCode=2
Jul 27 17:43:46.660031 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.659959    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk" event={"ID":"9f91da0c-d541-465e-a81d-b0f7d063b361","Type":"ContainerDied","Data":"f81bbf008642e8dfbc7fe4e4871aef73324d3b1cd8b8fbea12a864d15a6b889c"}
Jul 27 17:43:46.666566 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.666546    2573 scope.go:117] "RemoveContainer" containerID="894718fb4505d6916dc65fe57da59ad641501b95b070bc3f9b28201d16991e35"
Jul 27 17:43:46.674266 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.674231    2573 scope.go:117] "RemoveContainer" containerID="6d8e799564fa23814a38911cc65e0a239af83d5d433fa0513e216fc46450433c"
Jul 27 17:43:46.674536 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:43:46.674517    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d8e799564fa23814a38911cc65e0a239af83d5d433fa0513e216fc46450433c\": container with ID starting with 6d8e799564fa23814a38911cc65e0a239af83d5d433fa0513e216fc46450433c not found: ID does not exist" containerID="6d8e799564fa23814a38911cc65e0a239af83d5d433fa0513e216fc46450433c"
Jul 27 17:43:46.674603 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.674544    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d8e799564fa23814a38911cc65e0a239af83d5d433fa0513e216fc46450433c"} err="failed to get container status \"6d8e799564fa23814a38911cc65e0a239af83d5d433fa0513e216fc46450433c\": rpc error: code = NotFound desc = could not find container \"6d8e799564fa23814a38911cc65e0a239af83d5d433fa0513e216fc46450433c\": container with ID starting with 6d8e799564fa23814a38911cc65e0a239af83d5d433fa0513e216fc46450433c not found: ID does not exist"
Jul 27 17:43:46.674603 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.674562    2573 scope.go:117] "RemoveContainer" containerID="894718fb4505d6916dc65fe57da59ad641501b95b070bc3f9b28201d16991e35"
Jul 27 17:43:46.674804 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:43:46.674788    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"894718fb4505d6916dc65fe57da59ad641501b95b070bc3f9b28201d16991e35\": container with ID starting with 894718fb4505d6916dc65fe57da59ad641501b95b070bc3f9b28201d16991e35 not found: ID does not exist" containerID="894718fb4505d6916dc65fe57da59ad641501b95b070bc3f9b28201d16991e35"
Jul 27 17:43:46.674845 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.674808    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"894718fb4505d6916dc65fe57da59ad641501b95b070bc3f9b28201d16991e35"} err="failed to get container status \"894718fb4505d6916dc65fe57da59ad641501b95b070bc3f9b28201d16991e35\": rpc error: code = NotFound desc = could not find container \"894718fb4505d6916dc65fe57da59ad641501b95b070bc3f9b28201d16991e35\": container with ID starting with 894718fb4505d6916dc65fe57da59ad641501b95b070bc3f9b28201d16991e35 not found: ID does not exist"
Jul 27 17:43:46.674845 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.674822    2573 scope.go:117] "RemoveContainer" containerID="6d8e799564fa23814a38911cc65e0a239af83d5d433fa0513e216fc46450433c"
Jul 27 17:43:46.675006 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.674987    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d8e799564fa23814a38911cc65e0a239af83d5d433fa0513e216fc46450433c"} err="failed to get container status \"6d8e799564fa23814a38911cc65e0a239af83d5d433fa0513e216fc46450433c\": rpc error: code = NotFound desc = could not find container \"6d8e799564fa23814a38911cc65e0a239af83d5d433fa0513e216fc46450433c\": container with ID starting with 6d8e799564fa23814a38911cc65e0a239af83d5d433fa0513e216fc46450433c not found: ID does not exist"
Jul 27 17:43:46.675050 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.675006    2573 scope.go:117] "RemoveContainer" containerID="894718fb4505d6916dc65fe57da59ad641501b95b070bc3f9b28201d16991e35"
Jul 27 17:43:46.675226 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.675210    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"894718fb4505d6916dc65fe57da59ad641501b95b070bc3f9b28201d16991e35"} err="failed to get container status \"894718fb4505d6916dc65fe57da59ad641501b95b070bc3f9b28201d16991e35\": rpc error: code = NotFound desc = could not find container \"894718fb4505d6916dc65fe57da59ad641501b95b070bc3f9b28201d16991e35\": container with ID starting with 894718fb4505d6916dc65fe57da59ad641501b95b070bc3f9b28201d16991e35 not found: ID does not exist"
Jul 27 17:43:46.680730 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.680706    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mfbmn"]
Jul 27 17:43:46.684432 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.684408    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mfbmn"]
Jul 27 17:43:46.688095 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.688064    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8bfg\" (UniqueName: \"kubernetes.io/projected/8cd54928-6fd8-4a48-b5ba-691799e5c214-kube-api-access-d8bfg\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:43:46.688095 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.688100    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8cd54928-6fd8-4a48-b5ba-691799e5c214-proxy-tls\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:43:46.688200 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.688112    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8cd54928-6fd8-4a48-b5ba-691799e5c214-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:43:46.890305 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.890242    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-fvxrp\" (UID: \"cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxrp"
Jul 27 17:43:46.892627 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:46.892607    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-fvxrp\" (UID: \"cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxrp"
Jul 27 17:43:47.089232 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:47.089199    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxrp"
Jul 27 17:43:47.231123 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:47.231098    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxrp"]
Jul 27 17:43:47.233637 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:43:47.233605    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfe84b87_e1cf_4445_add3_5ff3b7d8b5c1.slice/crio-64d308ab5f50e25994cf8011e21276eab4d79802e3e745dd3a33e593795513c4 WatchSource:0}: Error finding container 64d308ab5f50e25994cf8011e21276eab4d79802e3e745dd3a33e593795513c4: Status 404 returned error can't find the container with id 64d308ab5f50e25994cf8011e21276eab4d79802e3e745dd3a33e593795513c4
Jul 27 17:43:47.667331 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:47.667292    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxrp" event={"ID":"cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1","Type":"ContainerStarted","Data":"29dd4300c3e9763657570529c9910b12fa45e81c9b4aec7ace95cb2da16f50cb"}
Jul 27 17:43:47.667542 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:47.667360    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxrp" event={"ID":"cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1","Type":"ContainerStarted","Data":"64d308ab5f50e25994cf8011e21276eab4d79802e3e745dd3a33e593795513c4"}
Jul 27 17:43:48.469460 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:48.469426    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cd54928-6fd8-4a48-b5ba-691799e5c214" path="/var/lib/kubelet/pods/8cd54928-6fd8-4a48-b5ba-691799e5c214/volumes"
Jul 27 17:43:49.248150 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:49.248101    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk" podUID="9f91da0c-d541-465e-a81d-b0f7d063b361" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 27 17:43:51.691111 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:51.691074    2573 generic.go:356] "Generic (PLEG): container finished" podID="9f91da0c-d541-465e-a81d-b0f7d063b361" containerID="c7ba02a430567f46b564290d2ed4eeb9260874af31593af3922534f3d910f961" exitCode=0
Jul 27 17:43:51.691516 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:51.691147    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk" event={"ID":"9f91da0c-d541-465e-a81d-b0f7d063b361","Type":"ContainerDied","Data":"c7ba02a430567f46b564290d2ed4eeb9260874af31593af3922534f3d910f961"}
Jul 27 17:43:54.247567 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:54.247524    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk" podUID="9f91da0c-d541-465e-a81d-b0f7d063b361" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 27 17:43:54.252840 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:54.252808    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk" podUID="9f91da0c-d541-465e-a81d-b0f7d063b361" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 27 17:43:54.253228 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:54.253202    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk" podUID="9f91da0c-d541-465e-a81d-b0f7d063b361" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 17:43:59.247523 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:59.247475    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk" podUID="9f91da0c-d541-465e-a81d-b0f7d063b361" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 27 17:43:59.247921 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:43:59.247622    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk"
Jul 27 17:44:04.248358 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:44:04.248315    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk" podUID="9f91da0c-d541-465e-a81d-b0f7d063b361" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 27 17:44:04.252734 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:44:04.252707    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk" podUID="9f91da0c-d541-465e-a81d-b0f7d063b361" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 27 17:44:04.253014 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:44:04.252994    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk" podUID="9f91da0c-d541-465e-a81d-b0f7d063b361" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 17:44:09.247911 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:44:09.247864    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk" podUID="9f91da0c-d541-465e-a81d-b0f7d063b361" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 27 17:44:14.248211 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:44:14.248168    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk" podUID="9f91da0c-d541-465e-a81d-b0f7d063b361" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 27 17:44:14.252636 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:44:14.252598    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk" podUID="9f91da0c-d541-465e-a81d-b0f7d063b361" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 27 17:44:14.252791 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:44:14.252734    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk"
Jul 27 17:44:14.253005 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:44:14.252983    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk" podUID="9f91da0c-d541-465e-a81d-b0f7d063b361" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 17:44:14.253108 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:44:14.253095    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk"
Jul 27 17:44:16.265046 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:44:16.265019    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk"
Jul 27 17:44:16.340732 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:44:16.340696    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f91da0c-d541-465e-a81d-b0f7d063b361-proxy-tls\") pod \"9f91da0c-d541-465e-a81d-b0f7d063b361\" (UID: \"9f91da0c-d541-465e-a81d-b0f7d063b361\") "
Jul 27 17:44:16.340884 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:44:16.340756    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f91da0c-d541-465e-a81d-b0f7d063b361-cabundle-cert\") pod \"9f91da0c-d541-465e-a81d-b0f7d063b361\" (UID: \"9f91da0c-d541-465e-a81d-b0f7d063b361\") "
Jul 27 17:44:16.340884 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:44:16.340788    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f91da0c-d541-465e-a81d-b0f7d063b361-kserve-provision-location\") pod \"9f91da0c-d541-465e-a81d-b0f7d063b361\" (UID: \"9f91da0c-d541-465e-a81d-b0f7d063b361\") "
Jul 27 17:44:16.340884 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:44:16.340808    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f91da0c-d541-465e-a81d-b0f7d063b361-kube-rbac-proxy-sar-config\") pod \"9f91da0c-d541-465e-a81d-b0f7d063b361\" (UID: \"9f91da0c-d541-465e-a81d-b0f7d063b361\") "
Jul 27 17:44:16.340884 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:44:16.340857    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-msm8p\" (UniqueName: \"kubernetes.io/projected/9f91da0c-d541-465e-a81d-b0f7d063b361-kube-api-access-msm8p\") pod \"9f91da0c-d541-465e-a81d-b0f7d063b361\" (UID: \"9f91da0c-d541-465e-a81d-b0f7d063b361\") "
Jul 27 17:44:16.341141 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:44:16.341111    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f91da0c-d541-465e-a81d-b0f7d063b361-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9f91da0c-d541-465e-a81d-b0f7d063b361" (UID: "9f91da0c-d541-465e-a81d-b0f7d063b361"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:44:16.341141 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:44:16.341128    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f91da0c-d541-465e-a81d-b0f7d063b361-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9f91da0c-d541-465e-a81d-b0f7d063b361" (UID: "9f91da0c-d541-465e-a81d-b0f7d063b361"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:44:16.341282 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:44:16.341160    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f91da0c-d541-465e-a81d-b0f7d063b361-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9f91da0c-d541-465e-a81d-b0f7d063b361" (UID: "9f91da0c-d541-465e-a81d-b0f7d063b361"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:44:16.342871 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:44:16.342849    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f91da0c-d541-465e-a81d-b0f7d063b361-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9f91da0c-d541-465e-a81d-b0f7d063b361" (UID: "9f91da0c-d541-465e-a81d-b0f7d063b361"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:44:16.343057 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:44:16.343035    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f91da0c-d541-465e-a81d-b0f7d063b361-kube-api-access-msm8p" (OuterVolumeSpecName: "kube-api-access-msm8p") pod "9f91da0c-d541-465e-a81d-b0f7d063b361" (UID: "9f91da0c-d541-465e-a81d-b0f7d063b361"). InnerVolumeSpecName "kube-api-access-msm8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:44:16.441584 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:44:16.441553    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f91da0c-d541-465e-a81d-b0f7d063b361-proxy-tls\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:44:16.441584 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:44:16.441581    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f91da0c-d541-465e-a81d-b0f7d063b361-cabundle-cert\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:44:16.441788 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:44:16.441592    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f91da0c-d541-465e-a81d-b0f7d063b361-kserve-provision-location\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:44:16.441788 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:44:16.441602    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f91da0c-d541-465e-a81d-b0f7d063b361-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:44:16.441788 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:44:16.441633    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msm8p\" (UniqueName: \"kubernetes.io/projected/9f91da0c-d541-465e-a81d-b0f7d063b361-kube-api-access-msm8p\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:44:16.805563 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:44:16.805470    2573 generic.go:356] "Generic (PLEG): container finished" podID="cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1" containerID="29dd4300c3e9763657570529c9910b12fa45e81c9b4aec7ace95cb2da16f50cb" exitCode=0
Jul 27 17:44:16.805563 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:44:16.805548    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxrp" event={"ID":"cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1","Type":"ContainerDied","Data":"29dd4300c3e9763657570529c9910b12fa45e81c9b4aec7ace95cb2da16f50cb"}
Jul 27 17:44:16.807128 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:44:16.807104    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 17:44:16.808100 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:44:16.808079    2573 generic.go:356] "Generic (PLEG): container finished" podID="9f91da0c-d541-465e-a81d-b0f7d063b361" containerID="56b61c780753ba29c4985cb9928dbe84e69d984104e49b699ddad0280d32f6ce" exitCode=0
Jul 27 17:44:16.808186 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:44:16.808151    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk"
Jul 27 17:44:16.808186 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:44:16.808167    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk" event={"ID":"9f91da0c-d541-465e-a81d-b0f7d063b361","Type":"ContainerDied","Data":"56b61c780753ba29c4985cb9928dbe84e69d984104e49b699ddad0280d32f6ce"}
Jul 27 17:44:16.808288 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:44:16.808263    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk" event={"ID":"9f91da0c-d541-465e-a81d-b0f7d063b361","Type":"ContainerDied","Data":"c5af4be8973179051b603b1a98598b6c563d1d79a7e5ed8398df1e93956435ca"}
Jul 27 17:44:16.808288 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:44:16.808281    2573 scope.go:117] "RemoveContainer" containerID="56b61c780753ba29c4985cb9928dbe84e69d984104e49b699ddad0280d32f6ce"
Jul 27 17:44:16.817055 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:44:16.817038    2573 scope.go:117] "RemoveContainer" containerID="f81bbf008642e8dfbc7fe4e4871aef73324d3b1cd8b8fbea12a864d15a6b889c"
Jul 27 17:44:16.824713 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:44:16.824696    2573 scope.go:117] "RemoveContainer" containerID="c7ba02a430567f46b564290d2ed4eeb9260874af31593af3922534f3d910f961"
Jul 27 17:44:16.832593 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:44:16.832577    2573 scope.go:117] "RemoveContainer" containerID="99e80a31306f3b926b977908e6d0049a6ee488a9f2fa08d860eaf4235869c796"
Jul 27 17:44:16.840005 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:44:16.839987    2573 scope.go:117] "RemoveContainer" containerID="56b61c780753ba29c4985cb9928dbe84e69d984104e49b699ddad0280d32f6ce"
Jul 27 17:44:16.840300 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:44:16.840277    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56b61c780753ba29c4985cb9928dbe84e69d984104e49b699ddad0280d32f6ce\": container with ID starting with 56b61c780753ba29c4985cb9928dbe84e69d984104e49b699ddad0280d32f6ce not found: ID does not exist" containerID="56b61c780753ba29c4985cb9928dbe84e69d984104e49b699ddad0280d32f6ce"
Jul 27 17:44:16.840368 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:44:16.840313    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56b61c780753ba29c4985cb9928dbe84e69d984104e49b699ddad0280d32f6ce"} err="failed to get container status \"56b61c780753ba29c4985cb9928dbe84e69d984104e49b699ddad0280d32f6ce\": rpc error: code = NotFound desc = could not find container \"56b61c780753ba29c4985cb9928dbe84e69d984104e49b699ddad0280d32f6ce\": container with ID starting with 56b61c780753ba29c4985cb9928dbe84e69d984104e49b699ddad0280d32f6ce not found: ID does not exist"
Jul 27 17:44:16.840368 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:44:16.840339    2573 scope.go:117] "RemoveContainer" containerID="f81bbf008642e8dfbc7fe4e4871aef73324d3b1cd8b8fbea12a864d15a6b889c"
Jul 27 17:44:16.840553 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:44:16.840538    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f81bbf008642e8dfbc7fe4e4871aef73324d3b1cd8b8fbea12a864d15a6b889c\": container with ID starting with f81bbf008642e8dfbc7fe4e4871aef73324d3b1cd8b8fbea12a864d15a6b889c not found: ID does not exist" containerID="f81bbf008642e8dfbc7fe4e4871aef73324d3b1cd8b8fbea12a864d15a6b889c"
Jul 27 17:44:16.840600 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:44:16.840559    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f81bbf008642e8dfbc7fe4e4871aef73324d3b1cd8b8fbea12a864d15a6b889c"} err="failed to get container status \"f81bbf008642e8dfbc7fe4e4871aef73324d3b1cd8b8fbea12a864d15a6b889c\": rpc error: code = NotFound desc = could not find container \"f81bbf008642e8dfbc7fe4e4871aef73324d3b1cd8b8fbea12a864d15a6b889c\": container with ID starting with f81bbf008642e8dfbc7fe4e4871aef73324d3b1cd8b8fbea12a864d15a6b889c not found: ID does not exist"
Jul 27 17:44:16.840600 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:44:16.840573    2573 scope.go:117] "RemoveContainer" containerID="c7ba02a430567f46b564290d2ed4eeb9260874af31593af3922534f3d910f961"
Jul 27 17:44:16.840817 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:44:16.840796    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7ba02a430567f46b564290d2ed4eeb9260874af31593af3922534f3d910f961\": container with ID starting with c7ba02a430567f46b564290d2ed4eeb9260874af31593af3922534f3d910f961 not found: ID does not exist" containerID="c7ba02a430567f46b564290d2ed4eeb9260874af31593af3922534f3d910f961"
Jul 27 17:44:16.840856 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:44:16.840822    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7ba02a430567f46b564290d2ed4eeb9260874af31593af3922534f3d910f961"} err="failed to get container status \"c7ba02a430567f46b564290d2ed4eeb9260874af31593af3922534f3d910f961\": rpc error: code = NotFound desc = could not find container \"c7ba02a430567f46b564290d2ed4eeb9260874af31593af3922534f3d910f961\": container with ID starting with c7ba02a430567f46b564290d2ed4eeb9260874af31593af3922534f3d910f961 not found: ID does not exist"
Jul 27 17:44:16.840856 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:44:16.840837    2573 scope.go:117] "RemoveContainer" containerID="99e80a31306f3b926b977908e6d0049a6ee488a9f2fa08d860eaf4235869c796"
Jul 27 17:44:16.841039 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:44:16.841024    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99e80a31306f3b926b977908e6d0049a6ee488a9f2fa08d860eaf4235869c796\": container with ID starting with 99e80a31306f3b926b977908e6d0049a6ee488a9f2fa08d860eaf4235869c796 not found: ID does not exist" containerID="99e80a31306f3b926b977908e6d0049a6ee488a9f2fa08d860eaf4235869c796"
Jul 27 17:44:16.841079 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:44:16.841044    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99e80a31306f3b926b977908e6d0049a6ee488a9f2fa08d860eaf4235869c796"} err="failed to get container status \"99e80a31306f3b926b977908e6d0049a6ee488a9f2fa08d860eaf4235869c796\": rpc error: code = NotFound desc = could not find container \"99e80a31306f3b926b977908e6d0049a6ee488a9f2fa08d860eaf4235869c796\": container with ID starting with 99e80a31306f3b926b977908e6d0049a6ee488a9f2fa08d860eaf4235869c796 not found: ID does not exist"
Jul 27 17:44:16.865643 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:44:16.865613    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk"]
Jul 27 17:44:16.874438 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:44:16.874404    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6b6746bb7-ww2wk"]
Jul 27 17:44:18.471360 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:44:18.471325    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f91da0c-d541-465e-a81d-b0f7d063b361" path="/var/lib/kubelet/pods/9f91da0c-d541-465e-a81d-b0f7d063b361/volumes"
Jul 27 17:45:03.897981 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:45:03.897926    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:46:12.391788 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:12.391756    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxrp" event={"ID":"cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1","Type":"ContainerStarted","Data":"8df4cb48535ac9237e8366569296d6b5d044f84b7211a54e6306b8817cf6658a"}
Jul 27 17:46:12.392314 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:12.391826    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxrp" event={"ID":"cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1","Type":"ContainerStarted","Data":"425cf3a4de9480c326393504f03cd6bc4bd0e0fa7c6fdca6235ac9df09acbd7f"}
Jul 27 17:46:12.392314 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:12.391929    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxrp"
Jul 27 17:46:12.423168 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:12.423098    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxrp" podStartSLOduration=2.139708024 podStartE2EDuration="2m26.423075619s" totalImagesPullingTime="1m55.045778458s" totalInitContainerRuntime="29.237589137s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:43:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:46:12.419446386 +0000 UTC m=+1114.562848439" watchObservedRunningTime="2026-07-27 17:46:12.423075619 +0000 UTC m=+1114.566477671"
Jul 27 17:46:13.397207 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:13.397174    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxrp"
Jul 27 17:46:13.398442 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:13.398411    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxrp" podUID="cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 27 17:46:14.401408 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:14.401363    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxrp" podUID="cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 27 17:46:19.406402 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:19.406369    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxrp"
Jul 27 17:46:19.406980 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:19.406955    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxrp" podUID="cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 27 17:46:29.408109 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:29.408070    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxrp"
Jul 27 17:46:42.632688 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:42.632650    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxrp"]
Jul 27 17:46:42.633196 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:42.632956    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxrp" podUID="cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1" containerName="kserve-container" containerID="cri-o://425cf3a4de9480c326393504f03cd6bc4bd0e0fa7c6fdca6235ac9df09acbd7f" gracePeriod=30
Jul 27 17:46:42.633196 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:42.632992    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxrp" podUID="cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1" containerName="kube-rbac-proxy" containerID="cri-o://8df4cb48535ac9237e8366569296d6b5d044f84b7211a54e6306b8817cf6658a" gracePeriod=30
Jul 27 17:46:42.669096 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:42.669064    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:46:43.535126 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:43.535094    2573 generic.go:356] "Generic (PLEG): container finished" podID="cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1" containerID="8df4cb48535ac9237e8366569296d6b5d044f84b7211a54e6306b8817cf6658a" exitCode=2
Jul 27 17:46:43.535337 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:43.535172    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxrp" event={"ID":"cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1","Type":"ContainerDied","Data":"8df4cb48535ac9237e8366569296d6b5d044f84b7211a54e6306b8817cf6658a"}
Jul 27 17:46:44.215938 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:44.215907    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxrp"
Jul 27 17:46:44.341441 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:44.341344    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1-kserve-provision-location\") pod \"cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1\" (UID: \"cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1\") "
Jul 27 17:46:44.341441 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:44.341394    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1-kube-rbac-proxy-sar-config\") pod \"cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1\" (UID: \"cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1\") "
Jul 27 17:46:44.341441 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:44.341426    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-67ft4\" (UniqueName: \"kubernetes.io/projected/cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1-kube-api-access-67ft4\") pod \"cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1\" (UID: \"cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1\") "
Jul 27 17:46:44.341737 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:44.341456    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1-cabundle-cert\") pod \"cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1\" (UID: \"cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1\") "
Jul 27 17:46:44.341737 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:44.341474    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1-proxy-tls\") pod \"cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1\" (UID: \"cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1\") "
Jul 27 17:46:44.341848 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:44.341828    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1" (UID: "cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:46:44.341909 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:44.341868    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1" (UID: "cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:46:44.343562 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:44.343537    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1-kube-api-access-67ft4" (OuterVolumeSpecName: "kube-api-access-67ft4") pod "cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1" (UID: "cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1"). InnerVolumeSpecName "kube-api-access-67ft4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:46:44.343691 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:44.343622    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1" (UID: "cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:46:44.346277 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:44.346235    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1" (UID: "cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:46:44.442179 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:44.442132    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1-kserve-provision-location\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:46:44.442179 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:44.442168    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:46:44.442179 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:44.442183    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67ft4\" (UniqueName: \"kubernetes.io/projected/cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1-kube-api-access-67ft4\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:46:44.442452 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:44.442195    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1-cabundle-cert\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:46:44.442452 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:44.442206    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1-proxy-tls\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:46:44.542014 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:44.541977    2573 generic.go:356] "Generic (PLEG): container finished" podID="cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1" containerID="425cf3a4de9480c326393504f03cd6bc4bd0e0fa7c6fdca6235ac9df09acbd7f" exitCode=0
Jul 27 17:46:44.542183 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:44.542035    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxrp"
Jul 27 17:46:44.542183 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:44.542060    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxrp" event={"ID":"cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1","Type":"ContainerDied","Data":"425cf3a4de9480c326393504f03cd6bc4bd0e0fa7c6fdca6235ac9df09acbd7f"}
Jul 27 17:46:44.542183 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:44.542123    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxrp" event={"ID":"cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1","Type":"ContainerDied","Data":"64d308ab5f50e25994cf8011e21276eab4d79802e3e745dd3a33e593795513c4"}
Jul 27 17:46:44.542183 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:44.542151    2573 scope.go:117] "RemoveContainer" containerID="8df4cb48535ac9237e8366569296d6b5d044f84b7211a54e6306b8817cf6658a"
Jul 27 17:46:44.550746 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:44.550730    2573 scope.go:117] "RemoveContainer" containerID="425cf3a4de9480c326393504f03cd6bc4bd0e0fa7c6fdca6235ac9df09acbd7f"
Jul 27 17:46:44.558380 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:44.558359    2573 scope.go:117] "RemoveContainer" containerID="29dd4300c3e9763657570529c9910b12fa45e81c9b4aec7ace95cb2da16f50cb"
Jul 27 17:46:44.561715 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:44.561691    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxrp"]
Jul 27 17:46:44.565848 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:44.565824    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-fvxrp"]
Jul 27 17:46:44.566865 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:44.566849    2573 scope.go:117] "RemoveContainer" containerID="8df4cb48535ac9237e8366569296d6b5d044f84b7211a54e6306b8817cf6658a"
Jul 27 17:46:44.567118 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:46:44.567094    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8df4cb48535ac9237e8366569296d6b5d044f84b7211a54e6306b8817cf6658a\": container with ID starting with 8df4cb48535ac9237e8366569296d6b5d044f84b7211a54e6306b8817cf6658a not found: ID does not exist" containerID="8df4cb48535ac9237e8366569296d6b5d044f84b7211a54e6306b8817cf6658a"
Jul 27 17:46:44.567233 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:44.567121    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8df4cb48535ac9237e8366569296d6b5d044f84b7211a54e6306b8817cf6658a"} err="failed to get container status \"8df4cb48535ac9237e8366569296d6b5d044f84b7211a54e6306b8817cf6658a\": rpc error: code = NotFound desc = could not find container \"8df4cb48535ac9237e8366569296d6b5d044f84b7211a54e6306b8817cf6658a\": container with ID starting with 8df4cb48535ac9237e8366569296d6b5d044f84b7211a54e6306b8817cf6658a not found: ID does not exist"
Jul 27 17:46:44.567233 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:44.567141    2573 scope.go:117] "RemoveContainer" containerID="425cf3a4de9480c326393504f03cd6bc4bd0e0fa7c6fdca6235ac9df09acbd7f"
Jul 27 17:46:44.567396 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:46:44.567380    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"425cf3a4de9480c326393504f03cd6bc4bd0e0fa7c6fdca6235ac9df09acbd7f\": container with ID starting with 425cf3a4de9480c326393504f03cd6bc4bd0e0fa7c6fdca6235ac9df09acbd7f not found: ID does not exist" containerID="425cf3a4de9480c326393504f03cd6bc4bd0e0fa7c6fdca6235ac9df09acbd7f"
Jul 27 17:46:44.567435 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:44.567400    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"425cf3a4de9480c326393504f03cd6bc4bd0e0fa7c6fdca6235ac9df09acbd7f"} err="failed to get container status \"425cf3a4de9480c326393504f03cd6bc4bd0e0fa7c6fdca6235ac9df09acbd7f\": rpc error: code = NotFound desc = could not find container \"425cf3a4de9480c326393504f03cd6bc4bd0e0fa7c6fdca6235ac9df09acbd7f\": container with ID starting with 425cf3a4de9480c326393504f03cd6bc4bd0e0fa7c6fdca6235ac9df09acbd7f not found: ID does not exist"
Jul 27 17:46:44.567435 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:44.567415    2573 scope.go:117] "RemoveContainer" containerID="29dd4300c3e9763657570529c9910b12fa45e81c9b4aec7ace95cb2da16f50cb"
Jul 27 17:46:44.567684 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:46:44.567667    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29dd4300c3e9763657570529c9910b12fa45e81c9b4aec7ace95cb2da16f50cb\": container with ID starting with 29dd4300c3e9763657570529c9910b12fa45e81c9b4aec7ace95cb2da16f50cb not found: ID does not exist" containerID="29dd4300c3e9763657570529c9910b12fa45e81c9b4aec7ace95cb2da16f50cb"
Jul 27 17:46:44.567728 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:44.567690    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29dd4300c3e9763657570529c9910b12fa45e81c9b4aec7ace95cb2da16f50cb"} err="failed to get container status \"29dd4300c3e9763657570529c9910b12fa45e81c9b4aec7ace95cb2da16f50cb\": rpc error: code = NotFound desc = could not find container \"29dd4300c3e9763657570529c9910b12fa45e81c9b4aec7ace95cb2da16f50cb\": container with ID starting with 29dd4300c3e9763657570529c9910b12fa45e81c9b4aec7ace95cb2da16f50cb not found: ID does not exist"
Jul 27 17:46:46.469312 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:46.469278    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1" path="/var/lib/kubelet/pods/cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1/volumes"
Jul 27 17:46:47.708642 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:47.708614    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:46:47.763021 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:47.762988    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pxzlt"]
Jul 27 17:46:47.763323 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:47.763311    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cd54928-6fd8-4a48-b5ba-691799e5c214" containerName="kserve-container"
Jul 27 17:46:47.763377 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:47.763325    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cd54928-6fd8-4a48-b5ba-691799e5c214" containerName="kserve-container"
Jul 27 17:46:47.763377 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:47.763335    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cd54928-6fd8-4a48-b5ba-691799e5c214" containerName="kube-rbac-proxy"
Jul 27 17:46:47.763377 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:47.763340    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cd54928-6fd8-4a48-b5ba-691799e5c214" containerName="kube-rbac-proxy"
Jul 27 17:46:47.763377 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:47.763348    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1" containerName="kserve-container"
Jul 27 17:46:47.763377 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:47.763354    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1" containerName="kserve-container"
Jul 27 17:46:47.763377 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:47.763362    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1" containerName="kube-rbac-proxy"
Jul 27 17:46:47.763377 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:47.763367    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1" containerName="kube-rbac-proxy"
Jul 27 17:46:47.763377 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:47.763377    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f91da0c-d541-465e-a81d-b0f7d063b361" containerName="storage-initializer"
Jul 27 17:46:47.763622 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:47.763384    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f91da0c-d541-465e-a81d-b0f7d063b361" containerName="storage-initializer"
Jul 27 17:46:47.763622 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:47.763397    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f91da0c-d541-465e-a81d-b0f7d063b361" containerName="agent"
Jul 27 17:46:47.763622 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:47.763402    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f91da0c-d541-465e-a81d-b0f7d063b361" containerName="agent"
Jul 27 17:46:47.763622 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:47.763410    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f91da0c-d541-465e-a81d-b0f7d063b361" containerName="kserve-container"
Jul 27 17:46:47.763622 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:47.763415    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f91da0c-d541-465e-a81d-b0f7d063b361" containerName="kserve-container"
Jul 27 17:46:47.763622 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:47.763422    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f91da0c-d541-465e-a81d-b0f7d063b361" containerName="kube-rbac-proxy"
Jul 27 17:46:47.763622 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:47.763427    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f91da0c-d541-465e-a81d-b0f7d063b361" containerName="kube-rbac-proxy"
Jul 27 17:46:47.763622 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:47.763435    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1" containerName="storage-initializer"
Jul 27 17:46:47.763622 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:47.763440    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1" containerName="storage-initializer"
Jul 27 17:46:47.763622 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:47.763487    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f91da0c-d541-465e-a81d-b0f7d063b361" containerName="kube-rbac-proxy"
Jul 27 17:46:47.763622 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:47.763494    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cd54928-6fd8-4a48-b5ba-691799e5c214" containerName="kserve-container"
Jul 27 17:46:47.763622 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:47.763501    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1" containerName="kserve-container"
Jul 27 17:46:47.763622 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:47.763508    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfe84b87-e1cf-4445-add3-5ff3b7d8b5c1" containerName="kube-rbac-proxy"
Jul 27 17:46:47.763622 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:47.763523    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f91da0c-d541-465e-a81d-b0f7d063b361" containerName="kserve-container"
Jul 27 17:46:47.763622 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:47.763528    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cd54928-6fd8-4a48-b5ba-691799e5c214" containerName="kube-rbac-proxy"
Jul 27 17:46:47.763622 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:47.763533    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f91da0c-d541-465e-a81d-b0f7d063b361" containerName="agent"
Jul 27 17:46:47.801035 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:47.801006    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pxzlt"]
Jul 27 17:46:47.801190 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:47.801117    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pxzlt"
Jul 27 17:46:47.803903 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:47.803879    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Jul 27 17:46:47.804058 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:47.804015    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 17:46:47.804058 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:47.804016    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Jul 27 17:46:47.804168 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:47.804092    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 17:46:47.804218 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:47.804188    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 17:46:47.804415 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:47.804399    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 17:46:47.804468 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:47.804413    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z6ld7\""
Jul 27 17:46:47.869958 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:47.869924    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bfe8f9ab-8359-4727-ba69-fe903c33c7e6-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-pxzlt\" (UID: \"bfe8f9ab-8359-4727-ba69-fe903c33c7e6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pxzlt"
Jul 27 17:46:47.870154 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:47.869964    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bfe8f9ab-8359-4727-ba69-fe903c33c7e6-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-pxzlt\" (UID: \"bfe8f9ab-8359-4727-ba69-fe903c33c7e6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pxzlt"
Jul 27 17:46:47.870154 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:47.869996    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bfe8f9ab-8359-4727-ba69-fe903c33c7e6-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-pxzlt\" (UID: \"bfe8f9ab-8359-4727-ba69-fe903c33c7e6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pxzlt"
Jul 27 17:46:47.870154 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:47.870098    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bfe8f9ab-8359-4727-ba69-fe903c33c7e6-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-pxzlt\" (UID: \"bfe8f9ab-8359-4727-ba69-fe903c33c7e6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pxzlt"
Jul 27 17:46:47.870154 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:47.870132    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7njnj\" (UniqueName: \"kubernetes.io/projected/bfe8f9ab-8359-4727-ba69-fe903c33c7e6-kube-api-access-7njnj\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-pxzlt\" (UID: \"bfe8f9ab-8359-4727-ba69-fe903c33c7e6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pxzlt"
Jul 27 17:46:47.895625 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:47.895598    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:46:47.971417 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:47.971328    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bfe8f9ab-8359-4727-ba69-fe903c33c7e6-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-pxzlt\" (UID: \"bfe8f9ab-8359-4727-ba69-fe903c33c7e6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pxzlt"
Jul 27 17:46:47.971417 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:47.971376    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bfe8f9ab-8359-4727-ba69-fe903c33c7e6-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-pxzlt\" (UID: \"bfe8f9ab-8359-4727-ba69-fe903c33c7e6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pxzlt"
Jul 27 17:46:47.971417 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:47.971395    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7njnj\" (UniqueName: \"kubernetes.io/projected/bfe8f9ab-8359-4727-ba69-fe903c33c7e6-kube-api-access-7njnj\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-pxzlt\" (UID: \"bfe8f9ab-8359-4727-ba69-fe903c33c7e6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pxzlt"
Jul 27 17:46:47.971644 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:47.971449    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bfe8f9ab-8359-4727-ba69-fe903c33c7e6-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-pxzlt\" (UID: \"bfe8f9ab-8359-4727-ba69-fe903c33c7e6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pxzlt"
Jul 27 17:46:47.971644 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:47.971481    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bfe8f9ab-8359-4727-ba69-fe903c33c7e6-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-pxzlt\" (UID: \"bfe8f9ab-8359-4727-ba69-fe903c33c7e6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pxzlt"
Jul 27 17:46:47.971777 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:47.971758    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bfe8f9ab-8359-4727-ba69-fe903c33c7e6-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-pxzlt\" (UID: \"bfe8f9ab-8359-4727-ba69-fe903c33c7e6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pxzlt"
Jul 27 17:46:47.972146 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:47.972125    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bfe8f9ab-8359-4727-ba69-fe903c33c7e6-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-pxzlt\" (UID: \"bfe8f9ab-8359-4727-ba69-fe903c33c7e6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pxzlt"
Jul 27 17:46:47.972274 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:47.972128    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bfe8f9ab-8359-4727-ba69-fe903c33c7e6-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-pxzlt\" (UID: \"bfe8f9ab-8359-4727-ba69-fe903c33c7e6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pxzlt"
Jul 27 17:46:47.973802 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:47.973776    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bfe8f9ab-8359-4727-ba69-fe903c33c7e6-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-pxzlt\" (UID: \"bfe8f9ab-8359-4727-ba69-fe903c33c7e6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pxzlt"
Jul 27 17:46:47.983084 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:47.983061    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7njnj\" (UniqueName: \"kubernetes.io/projected/bfe8f9ab-8359-4727-ba69-fe903c33c7e6-kube-api-access-7njnj\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-pxzlt\" (UID: \"bfe8f9ab-8359-4727-ba69-fe903c33c7e6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pxzlt"
Jul 27 17:46:48.111811 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:48.111771    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pxzlt"
Jul 27 17:46:48.269313 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:48.269287    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pxzlt"]
Jul 27 17:46:48.271186 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:46:48.271160    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfe8f9ab_8359_4727_ba69_fe903c33c7e6.slice/crio-02c5980e698b90287b9576a34733c015568c65e2cff64b9d73e4add460f109f1 WatchSource:0}: Error finding container 02c5980e698b90287b9576a34733c015568c65e2cff64b9d73e4add460f109f1: Status 404 returned error can't find the container with id 02c5980e698b90287b9576a34733c015568c65e2cff64b9d73e4add460f109f1
Jul 27 17:46:48.560792 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:48.560694    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pxzlt" event={"ID":"bfe8f9ab-8359-4727-ba69-fe903c33c7e6","Type":"ContainerStarted","Data":"1ff7689a36e40381f73100e3c35ba102899b07e8a186a0a6514afae2c72ebeb3"}
Jul 27 17:46:48.560792 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:46:48.560753    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pxzlt" event={"ID":"bfe8f9ab-8359-4727-ba69-fe903c33c7e6","Type":"ContainerStarted","Data":"02c5980e698b90287b9576a34733c015568c65e2cff64b9d73e4add460f109f1"}
Jul 27 17:47:10.155530 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:47:10.155492    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:47:17.690476 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:47:17.690444    2573 generic.go:356] "Generic (PLEG): container finished" podID="bfe8f9ab-8359-4727-ba69-fe903c33c7e6" containerID="1ff7689a36e40381f73100e3c35ba102899b07e8a186a0a6514afae2c72ebeb3" exitCode=0
Jul 27 17:47:17.690884 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:47:17.690514    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pxzlt" event={"ID":"bfe8f9ab-8359-4727-ba69-fe903c33c7e6","Type":"ContainerDied","Data":"1ff7689a36e40381f73100e3c35ba102899b07e8a186a0a6514afae2c72ebeb3"}
Jul 27 17:47:18.697712 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:47:18.697671    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pxzlt" event={"ID":"bfe8f9ab-8359-4727-ba69-fe903c33c7e6","Type":"ContainerStarted","Data":"51cd72688d5325e07a99310eddf035418a30f3b4cc3b36878d934474a8a02c8e"}
Jul 27 17:47:18.698154 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:47:18.697745    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pxzlt" event={"ID":"bfe8f9ab-8359-4727-ba69-fe903c33c7e6","Type":"ContainerStarted","Data":"ba33550ea7ff9ae5ec6ee88a741337253ec9ff052154f2151242d02ccaf97bdd"}
Jul 27 17:47:18.698154 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:47:18.698075    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pxzlt"
Jul 27 17:47:18.723156 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:47:18.723101    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pxzlt" podStartSLOduration=2.46069927 podStartE2EDuration="31.723085942s" totalImagesPullingTime="0s" totalInitContainerRuntime="29.262386672s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:46:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:47:18.721595078 +0000 UTC m=+1180.864997128" watchObservedRunningTime="2026-07-27 17:47:18.723085942 +0000 UTC m=+1180.866487993"
Jul 27 17:47:19.702186 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:47:19.702155    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pxzlt"
Jul 27 17:47:25.712264 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:47:25.712226    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pxzlt"
Jul 27 17:47:38.333843 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:47:38.333813    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vc2tz_87dac797-f8d2-4d36-999f-f8ce855dbeff/ovn-acl-logging/0.log"
Jul 27 17:47:38.334317 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:47:38.334101    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vc2tz_87dac797-f8d2-4d36-999f-f8ce855dbeff/ovn-acl-logging/0.log"
Jul 27 17:48:21.739689 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:21.739653    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:48:21.796969 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:21.796936    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:48:47.467406 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:47.467316    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pxzlt"
Jul 27 17:48:54.207600 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:54.207565    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pxzlt"]
Jul 27 17:48:54.208128 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:54.207865    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pxzlt" podUID="bfe8f9ab-8359-4727-ba69-fe903c33c7e6" containerName="kserve-container" containerID="cri-o://ba33550ea7ff9ae5ec6ee88a741337253ec9ff052154f2151242d02ccaf97bdd" gracePeriod=30
Jul 27 17:48:54.208128 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:54.207898    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pxzlt" podUID="bfe8f9ab-8359-4727-ba69-fe903c33c7e6" containerName="kube-rbac-proxy" containerID="cri-o://51cd72688d5325e07a99310eddf035418a30f3b4cc3b36878d934474a8a02c8e" gracePeriod=30
Jul 27 17:48:54.229408 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:54.229377    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:48:55.124075 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:55.124033    2573 generic.go:356] "Generic (PLEG): container finished" podID="bfe8f9ab-8359-4727-ba69-fe903c33c7e6" containerID="51cd72688d5325e07a99310eddf035418a30f3b4cc3b36878d934474a8a02c8e" exitCode=2
Jul 27 17:48:55.124075 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:55.124077    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pxzlt" event={"ID":"bfe8f9ab-8359-4727-ba69-fe903c33c7e6","Type":"ContainerDied","Data":"51cd72688d5325e07a99310eddf035418a30f3b4cc3b36878d934474a8a02c8e"}
Jul 27 17:48:55.759933 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:55.759904    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pxzlt"
Jul 27 17:48:55.939959 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:55.939929    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7njnj\" (UniqueName: \"kubernetes.io/projected/bfe8f9ab-8359-4727-ba69-fe903c33c7e6-kube-api-access-7njnj\") pod \"bfe8f9ab-8359-4727-ba69-fe903c33c7e6\" (UID: \"bfe8f9ab-8359-4727-ba69-fe903c33c7e6\") "
Jul 27 17:48:55.940148 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:55.939971    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bfe8f9ab-8359-4727-ba69-fe903c33c7e6-cabundle-cert\") pod \"bfe8f9ab-8359-4727-ba69-fe903c33c7e6\" (UID: \"bfe8f9ab-8359-4727-ba69-fe903c33c7e6\") "
Jul 27 17:48:55.940148 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:55.940004    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bfe8f9ab-8359-4727-ba69-fe903c33c7e6-kserve-provision-location\") pod \"bfe8f9ab-8359-4727-ba69-fe903c33c7e6\" (UID: \"bfe8f9ab-8359-4727-ba69-fe903c33c7e6\") "
Jul 27 17:48:55.940148 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:55.940040    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bfe8f9ab-8359-4727-ba69-fe903c33c7e6-kube-rbac-proxy-sar-config\") pod \"bfe8f9ab-8359-4727-ba69-fe903c33c7e6\" (UID: \"bfe8f9ab-8359-4727-ba69-fe903c33c7e6\") "
Jul 27 17:48:55.940148 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:55.940088    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bfe8f9ab-8359-4727-ba69-fe903c33c7e6-proxy-tls\") pod \"bfe8f9ab-8359-4727-ba69-fe903c33c7e6\" (UID: \"bfe8f9ab-8359-4727-ba69-fe903c33c7e6\") "
Jul 27 17:48:55.940488 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:55.940460    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfe8f9ab-8359-4727-ba69-fe903c33c7e6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bfe8f9ab-8359-4727-ba69-fe903c33c7e6" (UID: "bfe8f9ab-8359-4727-ba69-fe903c33c7e6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:48:55.940582 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:55.940551    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfe8f9ab-8359-4727-ba69-fe903c33c7e6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bfe8f9ab-8359-4727-ba69-fe903c33c7e6" (UID: "bfe8f9ab-8359-4727-ba69-fe903c33c7e6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:48:55.942143 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:55.942109    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfe8f9ab-8359-4727-ba69-fe903c33c7e6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bfe8f9ab-8359-4727-ba69-fe903c33c7e6" (UID: "bfe8f9ab-8359-4727-ba69-fe903c33c7e6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:48:55.942276 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:55.942169    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfe8f9ab-8359-4727-ba69-fe903c33c7e6-kube-api-access-7njnj" (OuterVolumeSpecName: "kube-api-access-7njnj") pod "bfe8f9ab-8359-4727-ba69-fe903c33c7e6" (UID: "bfe8f9ab-8359-4727-ba69-fe903c33c7e6"). InnerVolumeSpecName "kube-api-access-7njnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:48:55.946652 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:55.946625    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfe8f9ab-8359-4727-ba69-fe903c33c7e6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bfe8f9ab-8359-4727-ba69-fe903c33c7e6" (UID: "bfe8f9ab-8359-4727-ba69-fe903c33c7e6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:48:56.040743 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:56.040703    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7njnj\" (UniqueName: \"kubernetes.io/projected/bfe8f9ab-8359-4727-ba69-fe903c33c7e6-kube-api-access-7njnj\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:48:56.040743 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:56.040735    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bfe8f9ab-8359-4727-ba69-fe903c33c7e6-cabundle-cert\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:48:56.040743 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:56.040744    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bfe8f9ab-8359-4727-ba69-fe903c33c7e6-kserve-provision-location\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:48:56.040743 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:56.040756    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bfe8f9ab-8359-4727-ba69-fe903c33c7e6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:48:56.041035 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:56.040771    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bfe8f9ab-8359-4727-ba69-fe903c33c7e6-proxy-tls\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:48:56.130601 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:56.130566    2573 generic.go:356] "Generic (PLEG): container finished" podID="bfe8f9ab-8359-4727-ba69-fe903c33c7e6" containerID="ba33550ea7ff9ae5ec6ee88a741337253ec9ff052154f2151242d02ccaf97bdd" exitCode=0
Jul 27 17:48:56.130784 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:56.130638    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pxzlt"
Jul 27 17:48:56.130784 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:56.130644    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pxzlt" event={"ID":"bfe8f9ab-8359-4727-ba69-fe903c33c7e6","Type":"ContainerDied","Data":"ba33550ea7ff9ae5ec6ee88a741337253ec9ff052154f2151242d02ccaf97bdd"}
Jul 27 17:48:56.130784 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:56.130712    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pxzlt" event={"ID":"bfe8f9ab-8359-4727-ba69-fe903c33c7e6","Type":"ContainerDied","Data":"02c5980e698b90287b9576a34733c015568c65e2cff64b9d73e4add460f109f1"}
Jul 27 17:48:56.130784 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:56.130739    2573 scope.go:117] "RemoveContainer" containerID="51cd72688d5325e07a99310eddf035418a30f3b4cc3b36878d934474a8a02c8e"
Jul 27 17:48:56.139387 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:56.139370    2573 scope.go:117] "RemoveContainer" containerID="ba33550ea7ff9ae5ec6ee88a741337253ec9ff052154f2151242d02ccaf97bdd"
Jul 27 17:48:56.146980 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:56.146963    2573 scope.go:117] "RemoveContainer" containerID="1ff7689a36e40381f73100e3c35ba102899b07e8a186a0a6514afae2c72ebeb3"
Jul 27 17:48:56.152959 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:56.152930    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pxzlt"]
Jul 27 17:48:56.154647 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:56.154611    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pxzlt"]
Jul 27 17:48:56.155048 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:56.155031    2573 scope.go:117] "RemoveContainer" containerID="51cd72688d5325e07a99310eddf035418a30f3b4cc3b36878d934474a8a02c8e"
Jul 27 17:48:56.155363 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:48:56.155340    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51cd72688d5325e07a99310eddf035418a30f3b4cc3b36878d934474a8a02c8e\": container with ID starting with 51cd72688d5325e07a99310eddf035418a30f3b4cc3b36878d934474a8a02c8e not found: ID does not exist" containerID="51cd72688d5325e07a99310eddf035418a30f3b4cc3b36878d934474a8a02c8e"
Jul 27 17:48:56.155456 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:56.155379    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51cd72688d5325e07a99310eddf035418a30f3b4cc3b36878d934474a8a02c8e"} err="failed to get container status \"51cd72688d5325e07a99310eddf035418a30f3b4cc3b36878d934474a8a02c8e\": rpc error: code = NotFound desc = could not find container \"51cd72688d5325e07a99310eddf035418a30f3b4cc3b36878d934474a8a02c8e\": container with ID starting with 51cd72688d5325e07a99310eddf035418a30f3b4cc3b36878d934474a8a02c8e not found: ID does not exist"
Jul 27 17:48:56.155456 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:56.155406    2573 scope.go:117] "RemoveContainer" containerID="ba33550ea7ff9ae5ec6ee88a741337253ec9ff052154f2151242d02ccaf97bdd"
Jul 27 17:48:56.155682 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:48:56.155665    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba33550ea7ff9ae5ec6ee88a741337253ec9ff052154f2151242d02ccaf97bdd\": container with ID starting with ba33550ea7ff9ae5ec6ee88a741337253ec9ff052154f2151242d02ccaf97bdd not found: ID does not exist" containerID="ba33550ea7ff9ae5ec6ee88a741337253ec9ff052154f2151242d02ccaf97bdd"
Jul 27 17:48:56.155721 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:56.155688    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba33550ea7ff9ae5ec6ee88a741337253ec9ff052154f2151242d02ccaf97bdd"} err="failed to get container status \"ba33550ea7ff9ae5ec6ee88a741337253ec9ff052154f2151242d02ccaf97bdd\": rpc error: code = NotFound desc = could not find container \"ba33550ea7ff9ae5ec6ee88a741337253ec9ff052154f2151242d02ccaf97bdd\": container with ID starting with ba33550ea7ff9ae5ec6ee88a741337253ec9ff052154f2151242d02ccaf97bdd not found: ID does not exist"
Jul 27 17:48:56.155721 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:56.155705    2573 scope.go:117] "RemoveContainer" containerID="1ff7689a36e40381f73100e3c35ba102899b07e8a186a0a6514afae2c72ebeb3"
Jul 27 17:48:56.155927 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:48:56.155909    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ff7689a36e40381f73100e3c35ba102899b07e8a186a0a6514afae2c72ebeb3\": container with ID starting with 1ff7689a36e40381f73100e3c35ba102899b07e8a186a0a6514afae2c72ebeb3 not found: ID does not exist" containerID="1ff7689a36e40381f73100e3c35ba102899b07e8a186a0a6514afae2c72ebeb3"
Jul 27 17:48:56.155983 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:56.155936    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ff7689a36e40381f73100e3c35ba102899b07e8a186a0a6514afae2c72ebeb3"} err="failed to get container status \"1ff7689a36e40381f73100e3c35ba102899b07e8a186a0a6514afae2c72ebeb3\": rpc error: code = NotFound desc = could not find container \"1ff7689a36e40381f73100e3c35ba102899b07e8a186a0a6514afae2c72ebeb3\": container with ID starting with 1ff7689a36e40381f73100e3c35ba102899b07e8a186a0a6514afae2c72ebeb3 not found: ID does not exist"
Jul 27 17:48:56.470836 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:56.470798    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfe8f9ab-8359-4727-ba69-fe903c33c7e6" path="/var/lib/kubelet/pods/bfe8f9ab-8359-4727-ba69-fe903c33c7e6/volumes"
Jul 27 17:48:56.707393 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:56.707352    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-pxzlt" podUID="bfe8f9ab-8359-4727-ba69-fe903c33c7e6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": context deadline exceeded"
Jul 27 17:48:59.367261 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:59.367185    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:48:59.443021 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:59.442986    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9f9nt"]
Jul 27 17:48:59.443350 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:59.443337    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfe8f9ab-8359-4727-ba69-fe903c33c7e6" containerName="kserve-container"
Jul 27 17:48:59.443406 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:59.443352    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfe8f9ab-8359-4727-ba69-fe903c33c7e6" containerName="kserve-container"
Jul 27 17:48:59.443406 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:59.443371    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfe8f9ab-8359-4727-ba69-fe903c33c7e6" containerName="storage-initializer"
Jul 27 17:48:59.443406 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:59.443377    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfe8f9ab-8359-4727-ba69-fe903c33c7e6" containerName="storage-initializer"
Jul 27 17:48:59.443406 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:59.443387    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfe8f9ab-8359-4727-ba69-fe903c33c7e6" containerName="kube-rbac-proxy"
Jul 27 17:48:59.443406 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:59.443393    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfe8f9ab-8359-4727-ba69-fe903c33c7e6" containerName="kube-rbac-proxy"
Jul 27 17:48:59.443578 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:59.443449    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfe8f9ab-8359-4727-ba69-fe903c33c7e6" containerName="kube-rbac-proxy"
Jul 27 17:48:59.443578 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:59.443460    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfe8f9ab-8359-4727-ba69-fe903c33c7e6" containerName="kserve-container"
Jul 27 17:48:59.446692 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:59.446676    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9f9nt"
Jul 27 17:48:59.449612 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:59.449583    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Jul 27 17:48:59.449896 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:59.449854    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 17:48:59.449896 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:59.449885    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 17:48:59.450092 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:59.449857    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Jul 27 17:48:59.450092 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:59.450042    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 17:48:59.451091 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:59.451071    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z6ld7\""
Jul 27 17:48:59.451190 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:59.451170    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 17:48:59.456291 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:59.456267    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9f9nt"]
Jul 27 17:48:59.505901 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:59.505866    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:48:59.567702 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:59.567662    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-9f9nt\" (UID: \"74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9f9nt"
Jul 27 17:48:59.567903 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:59.567723    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-9f9nt\" (UID: \"74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9f9nt"
Jul 27 17:48:59.567903 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:59.567747    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-9f9nt\" (UID: \"74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9f9nt"
Jul 27 17:48:59.567903 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:59.567774    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-9f9nt\" (UID: \"74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9f9nt"
Jul 27 17:48:59.567903 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:59.567794    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjrz7\" (UniqueName: \"kubernetes.io/projected/74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e-kube-api-access-jjrz7\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-9f9nt\" (UID: \"74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9f9nt"
Jul 27 17:48:59.668834 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:59.668796    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-9f9nt\" (UID: \"74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9f9nt"
Jul 27 17:48:59.669019 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:59.668842    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjrz7\" (UniqueName: \"kubernetes.io/projected/74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e-kube-api-access-jjrz7\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-9f9nt\" (UID: \"74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9f9nt"
Jul 27 17:48:59.669019 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:59.668934    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-9f9nt\" (UID: \"74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9f9nt"
Jul 27 17:48:59.669019 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:59.668965    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-9f9nt\" (UID: \"74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9f9nt"
Jul 27 17:48:59.669019 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:59.668995    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-9f9nt\" (UID: \"74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9f9nt"
Jul 27 17:48:59.669480 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:59.669459    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-9f9nt\" (UID: \"74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9f9nt"
Jul 27 17:48:59.669672 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:59.669650    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-9f9nt\" (UID: \"74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9f9nt"
Jul 27 17:48:59.669732 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:59.669653    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-9f9nt\" (UID: \"74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9f9nt"
Jul 27 17:48:59.671390 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:59.671370    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-9f9nt\" (UID: \"74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9f9nt"
Jul 27 17:48:59.678176 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:59.678149    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjrz7\" (UniqueName: \"kubernetes.io/projected/74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e-kube-api-access-jjrz7\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-9f9nt\" (UID: \"74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9f9nt"
Jul 27 17:48:59.757938 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:59.757899    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9f9nt"
Jul 27 17:48:59.892927 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:48:59.892901    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9f9nt"]
Jul 27 17:48:59.895394 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:48:59.895352    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74eb8bd7_99d2_4c98_bcab_37b1dc0cff5e.slice/crio-48cf655cc258309f8aee0ff9c6e3f1cd2f8597c948440aced0217f729352d731 WatchSource:0}: Error finding container 48cf655cc258309f8aee0ff9c6e3f1cd2f8597c948440aced0217f729352d731: Status 404 returned error can't find the container with id 48cf655cc258309f8aee0ff9c6e3f1cd2f8597c948440aced0217f729352d731
Jul 27 17:49:00.151650 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:49:00.151611    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9f9nt" event={"ID":"74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e","Type":"ContainerStarted","Data":"375e54be271528f9f67789239879a23b0729726d66fcc89c5adde47dab68a02c"}
Jul 27 17:49:00.151822 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:49:00.151674    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9f9nt" event={"ID":"74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e","Type":"ContainerStarted","Data":"48cf655cc258309f8aee0ff9c6e3f1cd2f8597c948440aced0217f729352d731"}
Jul 27 17:49:20.252844 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:49:20.252806    2573 generic.go:356] "Generic (PLEG): container finished" podID="74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e" containerID="375e54be271528f9f67789239879a23b0729726d66fcc89c5adde47dab68a02c" exitCode=0
Jul 27 17:49:20.253212 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:49:20.252872    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9f9nt" event={"ID":"74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e","Type":"ContainerDied","Data":"375e54be271528f9f67789239879a23b0729726d66fcc89c5adde47dab68a02c"}
Jul 27 17:49:21.259516 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:49:21.259479    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9f9nt" event={"ID":"74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e","Type":"ContainerStarted","Data":"99c57b794f64c219e3a0940e42bef1f1675bb0258391d7353074857ebacd2e94"}
Jul 27 17:49:21.259919 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:49:21.259550    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9f9nt" event={"ID":"74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e","Type":"ContainerStarted","Data":"3ac9b88e9c02bd5fcf1caa1366d16ac8f056f7957425d6c4764f5751defd4c38"}
Jul 27 17:49:21.259919 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:49:21.259760    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9f9nt"
Jul 27 17:49:21.282082 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:49:21.282031    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9f9nt" podStartSLOduration=2.072018886 podStartE2EDuration="22.282013449s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.209994563s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:48:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:49:21.281173718 +0000 UTC m=+1303.424575769" watchObservedRunningTime="2026-07-27 17:49:21.282013449 +0000 UTC m=+1303.425415499"
Jul 27 17:49:22.264385 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:49:22.264352    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9f9nt"
Jul 27 17:49:28.276313 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:49:28.276283    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9f9nt"
Jul 27 17:50:59.467881 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:50:59.467845    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9f9nt"
Jul 27 17:51:16.474376 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:16.474344    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9f9nt"]
Jul 27 17:51:16.474869 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:16.474639    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9f9nt" podUID="74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e" containerName="kserve-container" containerID="cri-o://3ac9b88e9c02bd5fcf1caa1366d16ac8f056f7957425d6c4764f5751defd4c38" gracePeriod=30
Jul 27 17:51:16.474869 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:16.474703    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9f9nt" podUID="74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e" containerName="kube-rbac-proxy" containerID="cri-o://99c57b794f64c219e3a0940e42bef1f1675bb0258391d7353074857ebacd2e94" gracePeriod=30
Jul 27 17:51:16.530625 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:16.530588    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:51:16.784317 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:16.784213    2573 generic.go:356] "Generic (PLEG): container finished" podID="74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e" containerID="99c57b794f64c219e3a0940e42bef1f1675bb0258391d7353074857ebacd2e94" exitCode=2
Jul 27 17:51:16.784317 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:16.784286    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9f9nt" event={"ID":"74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e","Type":"ContainerDied","Data":"99c57b794f64c219e3a0940e42bef1f1675bb0258391d7353074857ebacd2e94"}
Jul 27 17:51:17.792037 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:17.792007    2573 generic.go:356] "Generic (PLEG): container finished" podID="74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e" containerID="3ac9b88e9c02bd5fcf1caa1366d16ac8f056f7957425d6c4764f5751defd4c38" exitCode=0
Jul 27 17:51:17.792393 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:17.792094    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9f9nt" event={"ID":"74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e","Type":"ContainerDied","Data":"3ac9b88e9c02bd5fcf1caa1366d16ac8f056f7957425d6c4764f5751defd4c38"}
Jul 27 17:51:17.915793 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:17.915767    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9f9nt"
Jul 27 17:51:18.027489 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:18.027403    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e-kube-rbac-proxy-sar-config\") pod \"74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e\" (UID: \"74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e\") "
Jul 27 17:51:18.027489 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:18.027442    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e-kserve-provision-location\") pod \"74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e\" (UID: \"74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e\") "
Jul 27 17:51:18.027727 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:18.027499    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e-proxy-tls\") pod \"74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e\" (UID: \"74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e\") "
Jul 27 17:51:18.027727 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:18.027520    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjrz7\" (UniqueName: \"kubernetes.io/projected/74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e-kube-api-access-jjrz7\") pod \"74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e\" (UID: \"74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e\") "
Jul 27 17:51:18.027727 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:18.027602    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e-cabundle-cert\") pod \"74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e\" (UID: \"74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e\") "
Jul 27 17:51:18.027865 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:18.027779    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e" (UID: "74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:51:18.027962 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:18.027942    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:51:18.028018 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:18.027979    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e" (UID: "74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:51:18.029617 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:18.029593    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e-kube-api-access-jjrz7" (OuterVolumeSpecName: "kube-api-access-jjrz7") pod "74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e" (UID: "74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e"). InnerVolumeSpecName "kube-api-access-jjrz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:51:18.029789 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:18.029763    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e" (UID: "74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:51:18.032009 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:18.031988    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e" (UID: "74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:51:18.129339 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:18.129307    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e-kserve-provision-location\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:51:18.129339 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:18.129332    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e-proxy-tls\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:51:18.129339 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:18.129342    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjrz7\" (UniqueName: \"kubernetes.io/projected/74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e-kube-api-access-jjrz7\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:51:18.129577 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:18.129351    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e-cabundle-cert\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:51:18.798906 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:18.798821    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9f9nt"
Jul 27 17:51:18.798906 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:18.798840    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9f9nt" event={"ID":"74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e","Type":"ContainerDied","Data":"48cf655cc258309f8aee0ff9c6e3f1cd2f8597c948440aced0217f729352d731"}
Jul 27 17:51:18.798906 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:18.798906    2573 scope.go:117] "RemoveContainer" containerID="99c57b794f64c219e3a0940e42bef1f1675bb0258391d7353074857ebacd2e94"
Jul 27 17:51:18.807050 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:18.807033    2573 scope.go:117] "RemoveContainer" containerID="3ac9b88e9c02bd5fcf1caa1366d16ac8f056f7957425d6c4764f5751defd4c38"
Jul 27 17:51:18.814310 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:18.814292    2573 scope.go:117] "RemoveContainer" containerID="375e54be271528f9f67789239879a23b0729726d66fcc89c5adde47dab68a02c"
Jul 27 17:51:18.819045 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:18.819018    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9f9nt"]
Jul 27 17:51:18.822711 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:18.822690    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-9f9nt"]
Jul 27 17:51:18.863440 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:18.863408    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:51:20.468977 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:20.468946    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e" path="/var/lib/kubelet/pods/74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e/volumes"
Jul 27 17:51:21.556410 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:21.556381    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:51:21.616058 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:21.616024    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ftbxp"]
Jul 27 17:51:21.616364 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:21.616352    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e" containerName="kserve-container"
Jul 27 17:51:21.616420 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:21.616366    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e" containerName="kserve-container"
Jul 27 17:51:21.616420 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:21.616376    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e" containerName="storage-initializer"
Jul 27 17:51:21.616420 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:21.616382    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e" containerName="storage-initializer"
Jul 27 17:51:21.616420 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:21.616401    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e" containerName="kube-rbac-proxy"
Jul 27 17:51:21.616420 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:21.616407    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e" containerName="kube-rbac-proxy"
Jul 27 17:51:21.616576 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:21.616455    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e" containerName="kserve-container"
Jul 27 17:51:21.616576 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:21.616462    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="74eb8bd7-99d2-4c98-bcab-37b1dc0cff5e" containerName="kube-rbac-proxy"
Jul 27 17:51:21.622945 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:21.622649    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ftbxp"
Jul 27 17:51:21.625570 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:21.625542    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Jul 27 17:51:21.625851 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:21.625834    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 17:51:21.625941 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:21.625854    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z6ld7\""
Jul 27 17:51:21.626233 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:21.626208    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 17:51:21.627166 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:21.627147    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 17:51:21.627395 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:21.627379    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 17:51:21.627545 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:21.627523    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Jul 27 17:51:21.629318 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:21.629297    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ftbxp"]
Jul 27 17:51:21.657882 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:21.657846    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/379d8717-0ed3-4289-99eb-c613b9af3a7f-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-ftbxp\" (UID: \"379d8717-0ed3-4289-99eb-c613b9af3a7f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ftbxp"
Jul 27 17:51:21.657882 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:21.657885    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k69fb\" (UniqueName: \"kubernetes.io/projected/379d8717-0ed3-4289-99eb-c613b9af3a7f-kube-api-access-k69fb\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-ftbxp\" (UID: \"379d8717-0ed3-4289-99eb-c613b9af3a7f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ftbxp"
Jul 27 17:51:21.658086 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:21.657946    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/379d8717-0ed3-4289-99eb-c613b9af3a7f-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-ftbxp\" (UID: \"379d8717-0ed3-4289-99eb-c613b9af3a7f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ftbxp"
Jul 27 17:51:21.658086 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:21.658004    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/379d8717-0ed3-4289-99eb-c613b9af3a7f-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-ftbxp\" (UID: \"379d8717-0ed3-4289-99eb-c613b9af3a7f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ftbxp"
Jul 27 17:51:21.658086 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:21.658026    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/379d8717-0ed3-4289-99eb-c613b9af3a7f-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-ftbxp\" (UID: \"379d8717-0ed3-4289-99eb-c613b9af3a7f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ftbxp"
Jul 27 17:51:21.747909 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:21.747876    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:51:21.758778 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:21.758745    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/379d8717-0ed3-4289-99eb-c613b9af3a7f-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-ftbxp\" (UID: \"379d8717-0ed3-4289-99eb-c613b9af3a7f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ftbxp"
Jul 27 17:51:21.758909 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:21.758788    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k69fb\" (UniqueName: \"kubernetes.io/projected/379d8717-0ed3-4289-99eb-c613b9af3a7f-kube-api-access-k69fb\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-ftbxp\" (UID: \"379d8717-0ed3-4289-99eb-c613b9af3a7f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ftbxp"
Jul 27 17:51:21.758909 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:21.758840    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/379d8717-0ed3-4289-99eb-c613b9af3a7f-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-ftbxp\" (UID: \"379d8717-0ed3-4289-99eb-c613b9af3a7f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ftbxp"
Jul 27 17:51:21.758909 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:21.758867    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/379d8717-0ed3-4289-99eb-c613b9af3a7f-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-ftbxp\" (UID: \"379d8717-0ed3-4289-99eb-c613b9af3a7f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ftbxp"
Jul 27 17:51:21.758909 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:21.758891    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/379d8717-0ed3-4289-99eb-c613b9af3a7f-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-ftbxp\" (UID: \"379d8717-0ed3-4289-99eb-c613b9af3a7f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ftbxp"
Jul 27 17:51:21.759176 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:21.759156    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/379d8717-0ed3-4289-99eb-c613b9af3a7f-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-ftbxp\" (UID: \"379d8717-0ed3-4289-99eb-c613b9af3a7f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ftbxp"
Jul 27 17:51:21.759562 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:21.759540    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/379d8717-0ed3-4289-99eb-c613b9af3a7f-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-ftbxp\" (UID: \"379d8717-0ed3-4289-99eb-c613b9af3a7f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ftbxp"
Jul 27 17:51:21.759638 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:21.759560    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/379d8717-0ed3-4289-99eb-c613b9af3a7f-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-ftbxp\" (UID: \"379d8717-0ed3-4289-99eb-c613b9af3a7f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ftbxp"
Jul 27 17:51:21.761479 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:21.761459    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/379d8717-0ed3-4289-99eb-c613b9af3a7f-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-ftbxp\" (UID: \"379d8717-0ed3-4289-99eb-c613b9af3a7f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ftbxp"
Jul 27 17:51:21.766630 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:21.766603    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k69fb\" (UniqueName: \"kubernetes.io/projected/379d8717-0ed3-4289-99eb-c613b9af3a7f-kube-api-access-k69fb\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-ftbxp\" (UID: \"379d8717-0ed3-4289-99eb-c613b9af3a7f\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ftbxp"
Jul 27 17:51:21.934392 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:21.934354    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ftbxp"
Jul 27 17:51:22.073065 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:22.073036    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ftbxp"]
Jul 27 17:51:22.075322 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:51:22.075288    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod379d8717_0ed3_4289_99eb_c613b9af3a7f.slice/crio-3a25a878ae908a2cffd1339607b8bd75d6297a9c2b2f846760925bb2c2942473 WatchSource:0}: Error finding container 3a25a878ae908a2cffd1339607b8bd75d6297a9c2b2f846760925bb2c2942473: Status 404 returned error can't find the container with id 3a25a878ae908a2cffd1339607b8bd75d6297a9c2b2f846760925bb2c2942473
Jul 27 17:51:22.818540 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:22.818506    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ftbxp" event={"ID":"379d8717-0ed3-4289-99eb-c613b9af3a7f","Type":"ContainerStarted","Data":"2a5007ee04dd1c5bdb0aa4b6b1beee9823c2fe849dd66932a72bf7541051d656"}
Jul 27 17:51:22.818940 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:22.818566    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ftbxp" event={"ID":"379d8717-0ed3-4289-99eb-c613b9af3a7f","Type":"ContainerStarted","Data":"3a25a878ae908a2cffd1339607b8bd75d6297a9c2b2f846760925bb2c2942473"}
Jul 27 17:51:42.908198 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:42.908161    2573 generic.go:356] "Generic (PLEG): container finished" podID="379d8717-0ed3-4289-99eb-c613b9af3a7f" containerID="2a5007ee04dd1c5bdb0aa4b6b1beee9823c2fe849dd66932a72bf7541051d656" exitCode=0
Jul 27 17:51:42.908638 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:42.908234    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ftbxp" event={"ID":"379d8717-0ed3-4289-99eb-c613b9af3a7f","Type":"ContainerDied","Data":"2a5007ee04dd1c5bdb0aa4b6b1beee9823c2fe849dd66932a72bf7541051d656"}
Jul 27 17:51:43.914974 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:43.914938    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ftbxp" event={"ID":"379d8717-0ed3-4289-99eb-c613b9af3a7f","Type":"ContainerStarted","Data":"83f62e3afa158b3097f1f94acc5d5239cd714260e332eeac6a46d7c35543fc8b"}
Jul 27 17:51:43.915403 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:43.915000    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ftbxp" event={"ID":"379d8717-0ed3-4289-99eb-c613b9af3a7f","Type":"ContainerStarted","Data":"5f777a901fa2b9acc851e30a3125614d62f17fd631e9037439da0f23fbd09466"}
Jul 27 17:51:43.915403 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:43.915289    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ftbxp"
Jul 27 17:51:43.915403 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:43.915355    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ftbxp"
Jul 27 17:51:43.937656 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:43.937605    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ftbxp" podStartSLOduration=2.322621447 podStartE2EDuration="22.937589958s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.614968511s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:51:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:51:43.936985826 +0000 UTC m=+1446.080387890" watchObservedRunningTime="2026-07-27 17:51:43.937589958 +0000 UTC m=+1446.080992008"
Jul 27 17:51:49.926373 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:51:49.926343    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ftbxp"
Jul 27 17:52:38.358918 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:52:38.358881    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vc2tz_87dac797-f8d2-4d36-999f-f8ce855dbeff/ovn-acl-logging/0.log"
Jul 27 17:52:38.361339 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:52:38.361316    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vc2tz_87dac797-f8d2-4d36-999f-f8ce855dbeff/ovn-acl-logging/0.log"
Jul 27 17:53:18.471098 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:18.471058    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ftbxp"
Jul 27 17:53:20.173106 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:20.173071    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:53:28.002842 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:28.002800    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ftbxp"]
Jul 27 17:53:28.005368 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:28.003135    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ftbxp" podUID="379d8717-0ed3-4289-99eb-c613b9af3a7f" containerName="kserve-container" containerID="cri-o://5f777a901fa2b9acc851e30a3125614d62f17fd631e9037439da0f23fbd09466" gracePeriod=30
Jul 27 17:53:28.005368 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:28.003170    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ftbxp" podUID="379d8717-0ed3-4289-99eb-c613b9af3a7f" containerName="kube-rbac-proxy" containerID="cri-o://83f62e3afa158b3097f1f94acc5d5239cd714260e332eeac6a46d7c35543fc8b" gracePeriod=30
Jul 27 17:53:28.015065 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:28.015039    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:53:28.375434 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:28.375351    2573 generic.go:356] "Generic (PLEG): container finished" podID="379d8717-0ed3-4289-99eb-c613b9af3a7f" containerID="83f62e3afa158b3097f1f94acc5d5239cd714260e332eeac6a46d7c35543fc8b" exitCode=2
Jul 27 17:53:28.375615 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:28.375431    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ftbxp" event={"ID":"379d8717-0ed3-4289-99eb-c613b9af3a7f","Type":"ContainerDied","Data":"83f62e3afa158b3097f1f94acc5d5239cd714260e332eeac6a46d7c35543fc8b"}
Jul 27 17:53:28.466408 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:28.466368    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ftbxp" podUID="379d8717-0ed3-4289-99eb-c613b9af3a7f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.35:8080/v2/models/isvc-autogluon-v2-noslash/ready\": dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 27 17:53:29.382362 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:29.382332    2573 generic.go:356] "Generic (PLEG): container finished" podID="379d8717-0ed3-4289-99eb-c613b9af3a7f" containerID="5f777a901fa2b9acc851e30a3125614d62f17fd631e9037439da0f23fbd09466" exitCode=0
Jul 27 17:53:29.382734 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:29.382367    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ftbxp" event={"ID":"379d8717-0ed3-4289-99eb-c613b9af3a7f","Type":"ContainerDied","Data":"5f777a901fa2b9acc851e30a3125614d62f17fd631e9037439da0f23fbd09466"}
Jul 27 17:53:29.460459 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:29.460435    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ftbxp"
Jul 27 17:53:29.642300 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:29.642242    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k69fb\" (UniqueName: \"kubernetes.io/projected/379d8717-0ed3-4289-99eb-c613b9af3a7f-kube-api-access-k69fb\") pod \"379d8717-0ed3-4289-99eb-c613b9af3a7f\" (UID: \"379d8717-0ed3-4289-99eb-c613b9af3a7f\") "
Jul 27 17:53:29.642474 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:29.642320    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/379d8717-0ed3-4289-99eb-c613b9af3a7f-kube-rbac-proxy-sar-config\") pod \"379d8717-0ed3-4289-99eb-c613b9af3a7f\" (UID: \"379d8717-0ed3-4289-99eb-c613b9af3a7f\") "
Jul 27 17:53:29.642474 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:29.642362    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/379d8717-0ed3-4289-99eb-c613b9af3a7f-cabundle-cert\") pod \"379d8717-0ed3-4289-99eb-c613b9af3a7f\" (UID: \"379d8717-0ed3-4289-99eb-c613b9af3a7f\") "
Jul 27 17:53:29.642474 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:29.642392    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/379d8717-0ed3-4289-99eb-c613b9af3a7f-kserve-provision-location\") pod \"379d8717-0ed3-4289-99eb-c613b9af3a7f\" (UID: \"379d8717-0ed3-4289-99eb-c613b9af3a7f\") "
Jul 27 17:53:29.642474 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:29.642442    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/379d8717-0ed3-4289-99eb-c613b9af3a7f-proxy-tls\") pod \"379d8717-0ed3-4289-99eb-c613b9af3a7f\" (UID: \"379d8717-0ed3-4289-99eb-c613b9af3a7f\") "
Jul 27 17:53:29.642785 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:29.642749    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/379d8717-0ed3-4289-99eb-c613b9af3a7f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "379d8717-0ed3-4289-99eb-c613b9af3a7f" (UID: "379d8717-0ed3-4289-99eb-c613b9af3a7f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:53:29.642785 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:29.642751    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/379d8717-0ed3-4289-99eb-c613b9af3a7f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "379d8717-0ed3-4289-99eb-c613b9af3a7f" (UID: "379d8717-0ed3-4289-99eb-c613b9af3a7f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:53:29.644656 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:29.644626    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/379d8717-0ed3-4289-99eb-c613b9af3a7f-kube-api-access-k69fb" (OuterVolumeSpecName: "kube-api-access-k69fb") pod "379d8717-0ed3-4289-99eb-c613b9af3a7f" (UID: "379d8717-0ed3-4289-99eb-c613b9af3a7f"). InnerVolumeSpecName "kube-api-access-k69fb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:53:29.644781 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:29.644762    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/379d8717-0ed3-4289-99eb-c613b9af3a7f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "379d8717-0ed3-4289-99eb-c613b9af3a7f" (UID: "379d8717-0ed3-4289-99eb-c613b9af3a7f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:53:29.646619 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:29.646582    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/379d8717-0ed3-4289-99eb-c613b9af3a7f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "379d8717-0ed3-4289-99eb-c613b9af3a7f" (UID: "379d8717-0ed3-4289-99eb-c613b9af3a7f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:53:29.743589 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:29.743533    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k69fb\" (UniqueName: \"kubernetes.io/projected/379d8717-0ed3-4289-99eb-c613b9af3a7f-kube-api-access-k69fb\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:53:29.743589 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:29.743582    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/379d8717-0ed3-4289-99eb-c613b9af3a7f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:53:29.743589 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:29.743596    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/379d8717-0ed3-4289-99eb-c613b9af3a7f-cabundle-cert\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:53:29.743589 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:29.743608    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/379d8717-0ed3-4289-99eb-c613b9af3a7f-kserve-provision-location\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:53:29.743930 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:29.743622    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/379d8717-0ed3-4289-99eb-c613b9af3a7f-proxy-tls\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:53:30.388713 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:30.388686    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ftbxp"
Jul 27 17:53:30.388713 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:30.388697    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ftbxp" event={"ID":"379d8717-0ed3-4289-99eb-c613b9af3a7f","Type":"ContainerDied","Data":"3a25a878ae908a2cffd1339607b8bd75d6297a9c2b2f846760925bb2c2942473"}
Jul 27 17:53:30.389198 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:30.388759    2573 scope.go:117] "RemoveContainer" containerID="83f62e3afa158b3097f1f94acc5d5239cd714260e332eeac6a46d7c35543fc8b"
Jul 27 17:53:30.398023 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:30.398001    2573 scope.go:117] "RemoveContainer" containerID="5f777a901fa2b9acc851e30a3125614d62f17fd631e9037439da0f23fbd09466"
Jul 27 17:53:30.405282 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:30.405261    2573 scope.go:117] "RemoveContainer" containerID="2a5007ee04dd1c5bdb0aa4b6b1beee9823c2fe849dd66932a72bf7541051d656"
Jul 27 17:53:30.420896 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:30.420871    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ftbxp"]
Jul 27 17:53:30.429876 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:30.429851    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-ftbxp"]
Jul 27 17:53:30.470038 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:30.470007    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="379d8717-0ed3-4289-99eb-c613b9af3a7f" path="/var/lib/kubelet/pods/379d8717-0ed3-4289-99eb-c613b9af3a7f/volumes"
Jul 27 17:53:33.154371 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:33.154333    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-8kr42"]
Jul 27 17:53:33.154792 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:33.154666    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="379d8717-0ed3-4289-99eb-c613b9af3a7f" containerName="kserve-container"
Jul 27 17:53:33.154792 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:33.154678    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="379d8717-0ed3-4289-99eb-c613b9af3a7f" containerName="kserve-container"
Jul 27 17:53:33.154792 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:33.154695    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="379d8717-0ed3-4289-99eb-c613b9af3a7f" containerName="kube-rbac-proxy"
Jul 27 17:53:33.154792 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:33.154700    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="379d8717-0ed3-4289-99eb-c613b9af3a7f" containerName="kube-rbac-proxy"
Jul 27 17:53:33.154792 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:33.154708    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="379d8717-0ed3-4289-99eb-c613b9af3a7f" containerName="storage-initializer"
Jul 27 17:53:33.154792 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:33.154715    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="379d8717-0ed3-4289-99eb-c613b9af3a7f" containerName="storage-initializer"
Jul 27 17:53:33.154792 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:33.154760    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="379d8717-0ed3-4289-99eb-c613b9af3a7f" containerName="kserve-container"
Jul 27 17:53:33.154792 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:33.154770    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="379d8717-0ed3-4289-99eb-c613b9af3a7f" containerName="kube-rbac-proxy"
Jul 27 17:53:33.159720 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:33.159701    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-8kr42"
Jul 27 17:53:33.162648 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:33.162620    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Jul 27 17:53:33.162796 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:33.162652    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z6ld7\""
Jul 27 17:53:33.162796 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:33.162620    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 17:53:33.162796 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:33.162630    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 17:53:33.164566 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:33.164541    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 17:53:33.164692 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:33.164545    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 17:53:33.164692 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:33.164569    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Jul 27 17:53:33.167465 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:33.167442    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-8kr42"]
Jul 27 17:53:33.271103 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:33.271063    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a54757b5-9860-4690-ab46-b81661a55239-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-8kr42\" (UID: \"a54757b5-9860-4690-ab46-b81661a55239\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-8kr42"
Jul 27 17:53:33.271103 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:33.271110    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a54757b5-9860-4690-ab46-b81661a55239-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-8kr42\" (UID: \"a54757b5-9860-4690-ab46-b81661a55239\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-8kr42"
Jul 27 17:53:33.271375 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:33.271162    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkghk\" (UniqueName: \"kubernetes.io/projected/a54757b5-9860-4690-ab46-b81661a55239-kube-api-access-vkghk\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-8kr42\" (UID: \"a54757b5-9860-4690-ab46-b81661a55239\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-8kr42"
Jul 27 17:53:33.271375 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:33.271226    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a54757b5-9860-4690-ab46-b81661a55239-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-8kr42\" (UID: \"a54757b5-9860-4690-ab46-b81661a55239\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-8kr42"
Jul 27 17:53:33.271375 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:33.271289    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a54757b5-9860-4690-ab46-b81661a55239-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-8kr42\" (UID: \"a54757b5-9860-4690-ab46-b81661a55239\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-8kr42"
Jul 27 17:53:33.372689 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:33.372653    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a54757b5-9860-4690-ab46-b81661a55239-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-8kr42\" (UID: \"a54757b5-9860-4690-ab46-b81661a55239\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-8kr42"
Jul 27 17:53:33.372840 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:33.372698    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a54757b5-9860-4690-ab46-b81661a55239-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-8kr42\" (UID: \"a54757b5-9860-4690-ab46-b81661a55239\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-8kr42"
Jul 27 17:53:33.372840 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:33.372766    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a54757b5-9860-4690-ab46-b81661a55239-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-8kr42\" (UID: \"a54757b5-9860-4690-ab46-b81661a55239\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-8kr42"
Jul 27 17:53:33.372840 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:33.372802    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a54757b5-9860-4690-ab46-b81661a55239-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-8kr42\" (UID: \"a54757b5-9860-4690-ab46-b81661a55239\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-8kr42"
Jul 27 17:53:33.372978 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:33.372839    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkghk\" (UniqueName: \"kubernetes.io/projected/a54757b5-9860-4690-ab46-b81661a55239-kube-api-access-vkghk\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-8kr42\" (UID: \"a54757b5-9860-4690-ab46-b81661a55239\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-8kr42"
Jul 27 17:53:33.373202 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:33.373176    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a54757b5-9860-4690-ab46-b81661a55239-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-8kr42\" (UID: \"a54757b5-9860-4690-ab46-b81661a55239\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-8kr42"
Jul 27 17:53:33.373473 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:33.373454    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a54757b5-9860-4690-ab46-b81661a55239-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-8kr42\" (UID: \"a54757b5-9860-4690-ab46-b81661a55239\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-8kr42"
Jul 27 17:53:33.373571 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:33.373551    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a54757b5-9860-4690-ab46-b81661a55239-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-8kr42\" (UID: \"a54757b5-9860-4690-ab46-b81661a55239\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-8kr42"
Jul 27 17:53:33.375216 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:33.375195    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a54757b5-9860-4690-ab46-b81661a55239-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-8kr42\" (UID: \"a54757b5-9860-4690-ab46-b81661a55239\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-8kr42"
Jul 27 17:53:33.382323 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:33.382297    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkghk\" (UniqueName: \"kubernetes.io/projected/a54757b5-9860-4690-ab46-b81661a55239-kube-api-access-vkghk\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-8kr42\" (UID: \"a54757b5-9860-4690-ab46-b81661a55239\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-8kr42"
Jul 27 17:53:33.471115 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:33.471011    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-8kr42"
Jul 27 17:53:33.607744 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:33.607705    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-8kr42"]
Jul 27 17:53:33.610504 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:53:33.610474    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda54757b5_9860_4690_ab46_b81661a55239.slice/crio-b2981c9e497f88bff7458e8e723ec706bebd57c9008160ef31665bb06f8f8340 WatchSource:0}: Error finding container b2981c9e497f88bff7458e8e723ec706bebd57c9008160ef31665bb06f8f8340: Status 404 returned error can't find the container with id b2981c9e497f88bff7458e8e723ec706bebd57c9008160ef31665bb06f8f8340
Jul 27 17:53:34.314949 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:34.314918    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:53:34.397997 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:34.397965    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:53:34.410438 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:34.410406    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-8kr42" event={"ID":"a54757b5-9860-4690-ab46-b81661a55239","Type":"ContainerStarted","Data":"99a3a2474da543a55e4c181ec84dc67f61e6c57386dfa9753917becba6cb1354"}
Jul 27 17:53:34.410612 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:34.410465    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-8kr42" event={"ID":"a54757b5-9860-4690-ab46-b81661a55239","Type":"ContainerStarted","Data":"b2981c9e497f88bff7458e8e723ec706bebd57c9008160ef31665bb06f8f8340"}
Jul 27 17:53:41.441803 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:41.441709    2573 generic.go:356] "Generic (PLEG): container finished" podID="a54757b5-9860-4690-ab46-b81661a55239" containerID="99a3a2474da543a55e4c181ec84dc67f61e6c57386dfa9753917becba6cb1354" exitCode=0
Jul 27 17:53:41.441803 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:41.441790    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-8kr42" event={"ID":"a54757b5-9860-4690-ab46-b81661a55239","Type":"ContainerDied","Data":"99a3a2474da543a55e4c181ec84dc67f61e6c57386dfa9753917becba6cb1354"}
Jul 27 17:53:42.456299 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:42.456260    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-8kr42" event={"ID":"a54757b5-9860-4690-ab46-b81661a55239","Type":"ContainerStarted","Data":"3a3852a53b6ba60b1570fa9d05790b6e75dc3cd28c9c9a1f6dd7257467233eeb"}
Jul 27 17:53:42.456796 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:42.456364    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-8kr42" event={"ID":"a54757b5-9860-4690-ab46-b81661a55239","Type":"ContainerStarted","Data":"3944683adcf5668addd15c3c0efc503a17b0d5542ec3fa011d41a59afa277240"}
Jul 27 17:53:42.456796 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:42.456599    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-8kr42"
Jul 27 17:53:42.479042 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:42.478955    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-8kr42" podStartSLOduration=1.970983633 podStartE2EDuration="9.478918716s" totalImagesPullingTime="0s" totalInitContainerRuntime="7.507935083s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:53:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:53:42.476974499 +0000 UTC m=+1564.620376550" watchObservedRunningTime="2026-07-27 17:53:42.478918716 +0000 UTC m=+1564.622320768"
Jul 27 17:53:43.461638 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:43.461608    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-8kr42"
Jul 27 17:53:43.462934 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:43.462901    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-8kr42" podUID="a54757b5-9860-4690-ab46-b81661a55239" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 27 17:53:44.467475 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:44.467427    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-8kr42" podUID="a54757b5-9860-4690-ab46-b81661a55239" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 27 17:53:49.473109 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:49.473079    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-8kr42"
Jul 27 17:53:49.473668 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:49.473624    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-8kr42" podUID="a54757b5-9860-4690-ab46-b81661a55239" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 27 17:53:59.475143 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:53:59.475111    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-8kr42"
Jul 27 17:54:15.902205 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:15.902169    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-8kr42"]
Jul 27 17:54:15.902739 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:15.902487    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-8kr42" podUID="a54757b5-9860-4690-ab46-b81661a55239" containerName="kserve-container" containerID="cri-o://3944683adcf5668addd15c3c0efc503a17b0d5542ec3fa011d41a59afa277240" gracePeriod=30
Jul 27 17:54:15.902739 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:15.902541    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-8kr42" podUID="a54757b5-9860-4690-ab46-b81661a55239" containerName="kube-rbac-proxy" containerID="cri-o://3a3852a53b6ba60b1570fa9d05790b6e75dc3cd28c9c9a1f6dd7257467233eeb" gracePeriod=30
Jul 27 17:54:15.922170 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:15.922124    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:54:16.613919 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:16.613885    2573 generic.go:356] "Generic (PLEG): container finished" podID="a54757b5-9860-4690-ab46-b81661a55239" containerID="3a3852a53b6ba60b1570fa9d05790b6e75dc3cd28c9c9a1f6dd7257467233eeb" exitCode=2
Jul 27 17:54:16.614082 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:16.613965    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-8kr42" event={"ID":"a54757b5-9860-4690-ab46-b81661a55239","Type":"ContainerDied","Data":"3a3852a53b6ba60b1570fa9d05790b6e75dc3cd28c9c9a1f6dd7257467233eeb"}
Jul 27 17:54:17.621383 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:17.621353    2573 generic.go:356] "Generic (PLEG): container finished" podID="a54757b5-9860-4690-ab46-b81661a55239" containerID="3944683adcf5668addd15c3c0efc503a17b0d5542ec3fa011d41a59afa277240" exitCode=0
Jul 27 17:54:17.621749 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:17.621413    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-8kr42" event={"ID":"a54757b5-9860-4690-ab46-b81661a55239","Type":"ContainerDied","Data":"3944683adcf5668addd15c3c0efc503a17b0d5542ec3fa011d41a59afa277240"}
Jul 27 17:54:17.659626 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:17.659602    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-8kr42"
Jul 27 17:54:17.755232 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:17.755138    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a54757b5-9860-4690-ab46-b81661a55239-proxy-tls\") pod \"a54757b5-9860-4690-ab46-b81661a55239\" (UID: \"a54757b5-9860-4690-ab46-b81661a55239\") "
Jul 27 17:54:17.755232 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:17.755189    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a54757b5-9860-4690-ab46-b81661a55239-cabundle-cert\") pod \"a54757b5-9860-4690-ab46-b81661a55239\" (UID: \"a54757b5-9860-4690-ab46-b81661a55239\") "
Jul 27 17:54:17.755506 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:17.755243    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a54757b5-9860-4690-ab46-b81661a55239-kube-rbac-proxy-sar-config\") pod \"a54757b5-9860-4690-ab46-b81661a55239\" (UID: \"a54757b5-9860-4690-ab46-b81661a55239\") "
Jul 27 17:54:17.755506 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:17.755306    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a54757b5-9860-4690-ab46-b81661a55239-kserve-provision-location\") pod \"a54757b5-9860-4690-ab46-b81661a55239\" (UID: \"a54757b5-9860-4690-ab46-b81661a55239\") "
Jul 27 17:54:17.755506 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:17.755343    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vkghk\" (UniqueName: \"kubernetes.io/projected/a54757b5-9860-4690-ab46-b81661a55239-kube-api-access-vkghk\") pod \"a54757b5-9860-4690-ab46-b81661a55239\" (UID: \"a54757b5-9860-4690-ab46-b81661a55239\") "
Jul 27 17:54:17.755665 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:17.755637    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a54757b5-9860-4690-ab46-b81661a55239-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a54757b5-9860-4690-ab46-b81661a55239" (UID: "a54757b5-9860-4690-ab46-b81661a55239"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:54:17.755715 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:17.755648    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a54757b5-9860-4690-ab46-b81661a55239-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a54757b5-9860-4690-ab46-b81661a55239" (UID: "a54757b5-9860-4690-ab46-b81661a55239"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:54:17.756208 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:17.756181    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a54757b5-9860-4690-ab46-b81661a55239-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a54757b5-9860-4690-ab46-b81661a55239" (UID: "a54757b5-9860-4690-ab46-b81661a55239"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:54:17.757381 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:17.757359    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a54757b5-9860-4690-ab46-b81661a55239-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a54757b5-9860-4690-ab46-b81661a55239" (UID: "a54757b5-9860-4690-ab46-b81661a55239"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:54:17.757475 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:17.757390    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a54757b5-9860-4690-ab46-b81661a55239-kube-api-access-vkghk" (OuterVolumeSpecName: "kube-api-access-vkghk") pod "a54757b5-9860-4690-ab46-b81661a55239" (UID: "a54757b5-9860-4690-ab46-b81661a55239"). InnerVolumeSpecName "kube-api-access-vkghk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:54:17.856179 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:17.856143    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a54757b5-9860-4690-ab46-b81661a55239-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:54:17.856179 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:17.856171    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a54757b5-9860-4690-ab46-b81661a55239-kserve-provision-location\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:54:17.856179 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:17.856181    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkghk\" (UniqueName: \"kubernetes.io/projected/a54757b5-9860-4690-ab46-b81661a55239-kube-api-access-vkghk\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:54:17.856179 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:17.856190    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a54757b5-9860-4690-ab46-b81661a55239-proxy-tls\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:54:17.856477 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:17.856199    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a54757b5-9860-4690-ab46-b81661a55239-cabundle-cert\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:54:18.628443 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:18.628405    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-8kr42" event={"ID":"a54757b5-9860-4690-ab46-b81661a55239","Type":"ContainerDied","Data":"b2981c9e497f88bff7458e8e723ec706bebd57c9008160ef31665bb06f8f8340"}
Jul 27 17:54:18.628443 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:18.628442    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-8kr42"
Jul 27 17:54:18.628911 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:18.628503    2573 scope.go:117] "RemoveContainer" containerID="3a3852a53b6ba60b1570fa9d05790b6e75dc3cd28c9c9a1f6dd7257467233eeb"
Jul 27 17:54:18.636555 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:18.636532    2573 scope.go:117] "RemoveContainer" containerID="3944683adcf5668addd15c3c0efc503a17b0d5542ec3fa011d41a59afa277240"
Jul 27 17:54:18.644215 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:18.644193    2573 scope.go:117] "RemoveContainer" containerID="99a3a2474da543a55e4c181ec84dc67f61e6c57386dfa9753917becba6cb1354"
Jul 27 17:54:18.649425 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:18.649402    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-8kr42"]
Jul 27 17:54:18.653880 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:18.653645    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-8kr42"]
Jul 27 17:54:20.469699 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:20.469667    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a54757b5-9860-4690-ab46-b81661a55239" path="/var/lib/kubelet/pods/a54757b5-9860-4690-ab46-b81661a55239/volumes"
Jul 27 17:54:21.044145 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:21.044113    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8"]
Jul 27 17:54:21.044450 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:21.044438    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a54757b5-9860-4690-ab46-b81661a55239" containerName="kserve-container"
Jul 27 17:54:21.044450 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:21.044451    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a54757b5-9860-4690-ab46-b81661a55239" containerName="kserve-container"
Jul 27 17:54:21.044536 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:21.044465    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a54757b5-9860-4690-ab46-b81661a55239" containerName="storage-initializer"
Jul 27 17:54:21.044536 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:21.044471    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a54757b5-9860-4690-ab46-b81661a55239" containerName="storage-initializer"
Jul 27 17:54:21.044536 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:21.044481    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a54757b5-9860-4690-ab46-b81661a55239" containerName="kube-rbac-proxy"
Jul 27 17:54:21.044536 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:21.044487    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a54757b5-9860-4690-ab46-b81661a55239" containerName="kube-rbac-proxy"
Jul 27 17:54:21.044677 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:21.044540    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a54757b5-9860-4690-ab46-b81661a55239" containerName="kube-rbac-proxy"
Jul 27 17:54:21.044677 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:21.044550    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a54757b5-9860-4690-ab46-b81661a55239" containerName="kserve-container"
Jul 27 17:54:21.049209 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:21.049192    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8"
Jul 27 17:54:21.052025 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:21.052000    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-predictor-serving-cert\""
Jul 27 17:54:21.052204 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:21.052028    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-kube-rbac-proxy-sar-config\""
Jul 27 17:54:21.052290 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:21.052238    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z6ld7\""
Jul 27 17:54:21.052356 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:21.052334    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 17:54:21.052356 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:21.052335    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 17:54:21.053319 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:21.053305    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 17:54:21.053378 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:21.053364    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 17:54:21.057153 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:21.057131    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8"]
Jul 27 17:54:21.077162 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:21.077137    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:54:21.185526 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:21.185488    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/416e4a4e-94c3-4515-ad94-9fb495fcfe1b-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8\" (UID: \"416e4a4e-94c3-4515-ad94-9fb495fcfe1b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8"
Jul 27 17:54:21.185673 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:21.185549    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bssc\" (UniqueName: \"kubernetes.io/projected/416e4a4e-94c3-4515-ad94-9fb495fcfe1b-kube-api-access-4bssc\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8\" (UID: \"416e4a4e-94c3-4515-ad94-9fb495fcfe1b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8"
Jul 27 17:54:21.185673 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:21.185575    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/416e4a4e-94c3-4515-ad94-9fb495fcfe1b-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8\" (UID: \"416e4a4e-94c3-4515-ad94-9fb495fcfe1b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8"
Jul 27 17:54:21.185673 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:21.185602    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/416e4a4e-94c3-4515-ad94-9fb495fcfe1b-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8\" (UID: \"416e4a4e-94c3-4515-ad94-9fb495fcfe1b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8"
Jul 27 17:54:21.185673 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:21.185636    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/416e4a4e-94c3-4515-ad94-9fb495fcfe1b-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8\" (UID: \"416e4a4e-94c3-4515-ad94-9fb495fcfe1b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8"
Jul 27 17:54:21.286877 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:21.286848    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/416e4a4e-94c3-4515-ad94-9fb495fcfe1b-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8\" (UID: \"416e4a4e-94c3-4515-ad94-9fb495fcfe1b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8"
Jul 27 17:54:21.286961 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:21.286894    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/416e4a4e-94c3-4515-ad94-9fb495fcfe1b-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8\" (UID: \"416e4a4e-94c3-4515-ad94-9fb495fcfe1b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8"
Jul 27 17:54:21.286961 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:21.286927    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/416e4a4e-94c3-4515-ad94-9fb495fcfe1b-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8\" (UID: \"416e4a4e-94c3-4515-ad94-9fb495fcfe1b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8"
Jul 27 17:54:21.287026 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:21.286970    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bssc\" (UniqueName: \"kubernetes.io/projected/416e4a4e-94c3-4515-ad94-9fb495fcfe1b-kube-api-access-4bssc\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8\" (UID: \"416e4a4e-94c3-4515-ad94-9fb495fcfe1b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8"
Jul 27 17:54:21.287192 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:21.287160    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/416e4a4e-94c3-4515-ad94-9fb495fcfe1b-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8\" (UID: \"416e4a4e-94c3-4515-ad94-9fb495fcfe1b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8"
Jul 27 17:54:21.287294 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:21.287274    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/416e4a4e-94c3-4515-ad94-9fb495fcfe1b-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8\" (UID: \"416e4a4e-94c3-4515-ad94-9fb495fcfe1b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8"
Jul 27 17:54:21.287349 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:54:21.287331    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-serving-cert: secret "isvc-autogluon-ts-v1-noslash-predictor-serving-cert" not found
Jul 27 17:54:21.287414 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:54:21.287402    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/416e4a4e-94c3-4515-ad94-9fb495fcfe1b-proxy-tls podName:416e4a4e-94c3-4515-ad94-9fb495fcfe1b nodeName:}" failed. No retries permitted until 2026-07-27 17:54:21.787382224 +0000 UTC m=+1603.930784253 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/416e4a4e-94c3-4515-ad94-9fb495fcfe1b-proxy-tls") pod "isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8" (UID: "416e4a4e-94c3-4515-ad94-9fb495fcfe1b") : secret "isvc-autogluon-ts-v1-noslash-predictor-serving-cert" not found
Jul 27 17:54:21.287585 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:21.287566    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/416e4a4e-94c3-4515-ad94-9fb495fcfe1b-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8\" (UID: \"416e4a4e-94c3-4515-ad94-9fb495fcfe1b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8"
Jul 27 17:54:21.287717 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:21.287696    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/416e4a4e-94c3-4515-ad94-9fb495fcfe1b-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8\" (UID: \"416e4a4e-94c3-4515-ad94-9fb495fcfe1b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8"
Jul 27 17:54:21.295996 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:21.295936    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bssc\" (UniqueName: \"kubernetes.io/projected/416e4a4e-94c3-4515-ad94-9fb495fcfe1b-kube-api-access-4bssc\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8\" (UID: \"416e4a4e-94c3-4515-ad94-9fb495fcfe1b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8"
Jul 27 17:54:21.364002 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:21.363971    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:54:21.792441 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:21.792383    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/416e4a4e-94c3-4515-ad94-9fb495fcfe1b-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8\" (UID: \"416e4a4e-94c3-4515-ad94-9fb495fcfe1b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8"
Jul 27 17:54:21.794787 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:21.794765    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/416e4a4e-94c3-4515-ad94-9fb495fcfe1b-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8\" (UID: \"416e4a4e-94c3-4515-ad94-9fb495fcfe1b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8"
Jul 27 17:54:21.959830 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:21.959793    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8"
Jul 27 17:54:22.100183 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:22.100150    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8"]
Jul 27 17:54:22.103103 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:54:22.103077    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod416e4a4e_94c3_4515_ad94_9fb495fcfe1b.slice/crio-90959d9d7e3550bf31b10016e201e4654cd68eff9f7ed358f035d20e21b70c98 WatchSource:0}: Error finding container 90959d9d7e3550bf31b10016e201e4654cd68eff9f7ed358f035d20e21b70c98: Status 404 returned error can't find the container with id 90959d9d7e3550bf31b10016e201e4654cd68eff9f7ed358f035d20e21b70c98
Jul 27 17:54:22.648594 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:22.648551    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8" event={"ID":"416e4a4e-94c3-4515-ad94-9fb495fcfe1b","Type":"ContainerStarted","Data":"02d8c7cc2c8a601084bff87b338ed8f70d1b835e4e10f8bcc35d178e50c9bfd9"}
Jul 27 17:54:22.648808 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:22.648660    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8" event={"ID":"416e4a4e-94c3-4515-ad94-9fb495fcfe1b","Type":"ContainerStarted","Data":"90959d9d7e3550bf31b10016e201e4654cd68eff9f7ed358f035d20e21b70c98"}
Jul 27 17:54:26.666524 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:26.666486    2573 generic.go:356] "Generic (PLEG): container finished" podID="416e4a4e-94c3-4515-ad94-9fb495fcfe1b" containerID="02d8c7cc2c8a601084bff87b338ed8f70d1b835e4e10f8bcc35d178e50c9bfd9" exitCode=0
Jul 27 17:54:26.666916 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:26.666559    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8" event={"ID":"416e4a4e-94c3-4515-ad94-9fb495fcfe1b","Type":"ContainerDied","Data":"02d8c7cc2c8a601084bff87b338ed8f70d1b835e4e10f8bcc35d178e50c9bfd9"}
Jul 27 17:54:27.673229 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:27.673194    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8" event={"ID":"416e4a4e-94c3-4515-ad94-9fb495fcfe1b","Type":"ContainerStarted","Data":"de72c20a732994d9a16558ac862f714c80b3d7fec052aeb77db3b41eee5a1cb4"}
Jul 27 17:54:27.673626 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:27.673281    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8" event={"ID":"416e4a4e-94c3-4515-ad94-9fb495fcfe1b","Type":"ContainerStarted","Data":"77618a08237c6d680868435e4aa8c4d5be22df7321278112a0bf779ffb185838"}
Jul 27 17:54:27.673626 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:27.673611    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8"
Jul 27 17:54:27.673778 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:27.673761    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8"
Jul 27 17:54:27.675135 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:27.675105    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8" podUID="416e4a4e-94c3-4515-ad94-9fb495fcfe1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 27 17:54:27.698130 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:27.698077    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8" podStartSLOduration=2.225765242 podStartE2EDuration="6.698060841s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.472295599s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:54:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:54:27.696524586 +0000 UTC m=+1609.839926638" watchObservedRunningTime="2026-07-27 17:54:27.698060841 +0000 UTC m=+1609.841462891"
Jul 27 17:54:28.678917 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:28.678879    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8" podUID="416e4a4e-94c3-4515-ad94-9fb495fcfe1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 27 17:54:33.683836 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:33.683755    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8"
Jul 27 17:54:33.684298 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:33.684242    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8" podUID="416e4a4e-94c3-4515-ad94-9fb495fcfe1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 27 17:54:43.685400 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:43.685367    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8"
Jul 27 17:54:56.383475 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:56.383439    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8"]
Jul 27 17:54:56.384058 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:56.383809    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8" podUID="416e4a4e-94c3-4515-ad94-9fb495fcfe1b" containerName="kserve-container" containerID="cri-o://77618a08237c6d680868435e4aa8c4d5be22df7321278112a0bf779ffb185838" gracePeriod=30
Jul 27 17:54:56.384058 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:56.383814    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8" podUID="416e4a4e-94c3-4515-ad94-9fb495fcfe1b" containerName="kube-rbac-proxy" containerID="cri-o://de72c20a732994d9a16558ac862f714c80b3d7fec052aeb77db3b41eee5a1cb4" gracePeriod=30
Jul 27 17:54:56.397829 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:56.397803    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:54:56.804761 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:56.804728    2573 generic.go:356] "Generic (PLEG): container finished" podID="416e4a4e-94c3-4515-ad94-9fb495fcfe1b" containerID="de72c20a732994d9a16558ac862f714c80b3d7fec052aeb77db3b41eee5a1cb4" exitCode=2
Jul 27 17:54:56.804942 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:56.804797    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8" event={"ID":"416e4a4e-94c3-4515-ad94-9fb495fcfe1b","Type":"ContainerDied","Data":"de72c20a732994d9a16558ac862f714c80b3d7fec052aeb77db3b41eee5a1cb4"}
Jul 27 17:54:58.679001 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:54:58.678960    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8" podUID="416e4a4e-94c3-4515-ad94-9fb495fcfe1b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Jul 27 17:55:01.515695 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.515653    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:55:01.566842 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.566812    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kzxmm"]
Jul 27 17:55:01.570345 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.570326    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kzxmm"
Jul 27 17:55:01.573108 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.573084    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Jul 27 17:55:01.573108 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.573103    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Jul 27 17:55:01.580540 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.580507    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kzxmm"]
Jul 27 17:55:01.588836 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.588801    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:55:01.589684 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.589665    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr"]
Jul 27 17:55:01.593192 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.593167    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr"
Jul 27 17:55:01.597066 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.597043    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Jul 27 17:55:01.605156 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.605090    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/548b5a37-e500-45c6-b82f-13f2a74dbff8-proxy-tls\") pod \"isvc-canary-create-predictor-cb85cf55c-kzxmm\" (UID: \"548b5a37-e500-45c6-b82f-13f2a74dbff8\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kzxmm"
Jul 27 17:55:01.605292 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.605208    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/548b5a37-e500-45c6-b82f-13f2a74dbff8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-cb85cf55c-kzxmm\" (UID: \"548b5a37-e500-45c6-b82f-13f2a74dbff8\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kzxmm"
Jul 27 17:55:01.605292 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.605272    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/548b5a37-e500-45c6-b82f-13f2a74dbff8-kserve-provision-location\") pod \"isvc-canary-create-predictor-cb85cf55c-kzxmm\" (UID: \"548b5a37-e500-45c6-b82f-13f2a74dbff8\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kzxmm"
Jul 27 17:55:01.605424 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.605305    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8855\" (UniqueName: \"kubernetes.io/projected/548b5a37-e500-45c6-b82f-13f2a74dbff8-kube-api-access-q8855\") pod \"isvc-canary-create-predictor-cb85cf55c-kzxmm\" (UID: \"548b5a37-e500-45c6-b82f-13f2a74dbff8\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kzxmm"
Jul 27 17:55:01.606013 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.605975    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/548b5a37-e500-45c6-b82f-13f2a74dbff8-cabundle-cert\") pod \"isvc-canary-create-predictor-cb85cf55c-kzxmm\" (UID: \"548b5a37-e500-45c6-b82f-13f2a74dbff8\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kzxmm"
Jul 27 17:55:01.614298 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.612375    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr"]
Jul 27 17:55:01.702069 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.702035    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:55:01.707263 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.707224    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/548b5a37-e500-45c6-b82f-13f2a74dbff8-proxy-tls\") pod \"isvc-canary-create-predictor-cb85cf55c-kzxmm\" (UID: \"548b5a37-e500-45c6-b82f-13f2a74dbff8\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kzxmm"
Jul 27 17:55:01.707375 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.707293    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9775f83a-fe61-493f-bb43-38a1229ba941-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr\" (UID: \"9775f83a-fe61-493f-bb43-38a1229ba941\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr"
Jul 27 17:55:01.707375 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.707339    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/548b5a37-e500-45c6-b82f-13f2a74dbff8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-cb85cf55c-kzxmm\" (UID: \"548b5a37-e500-45c6-b82f-13f2a74dbff8\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kzxmm"
Jul 27 17:55:01.707489 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.707459    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9775f83a-fe61-493f-bb43-38a1229ba941-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr\" (UID: \"9775f83a-fe61-493f-bb43-38a1229ba941\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr"
Jul 27 17:55:01.707635 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.707518    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/548b5a37-e500-45c6-b82f-13f2a74dbff8-kserve-provision-location\") pod \"isvc-canary-create-predictor-cb85cf55c-kzxmm\" (UID: \"548b5a37-e500-45c6-b82f-13f2a74dbff8\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kzxmm"
Jul 27 17:55:01.707635 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.707554    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8855\" (UniqueName: \"kubernetes.io/projected/548b5a37-e500-45c6-b82f-13f2a74dbff8-kube-api-access-q8855\") pod \"isvc-canary-create-predictor-cb85cf55c-kzxmm\" (UID: \"548b5a37-e500-45c6-b82f-13f2a74dbff8\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kzxmm"
Jul 27 17:55:01.707635 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.707585    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9775f83a-fe61-493f-bb43-38a1229ba941-proxy-tls\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr\" (UID: \"9775f83a-fe61-493f-bb43-38a1229ba941\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr"
Jul 27 17:55:01.707806 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.707715    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9775f83a-fe61-493f-bb43-38a1229ba941-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr\" (UID: \"9775f83a-fe61-493f-bb43-38a1229ba941\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr"
Jul 27 17:55:01.707806 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.707774    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/548b5a37-e500-45c6-b82f-13f2a74dbff8-cabundle-cert\") pod \"isvc-canary-create-predictor-cb85cf55c-kzxmm\" (UID: \"548b5a37-e500-45c6-b82f-13f2a74dbff8\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kzxmm"
Jul 27 17:55:01.707937 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.707848    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7txpg\" (UniqueName: \"kubernetes.io/projected/9775f83a-fe61-493f-bb43-38a1229ba941-kube-api-access-7txpg\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr\" (UID: \"9775f83a-fe61-493f-bb43-38a1229ba941\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr"
Jul 27 17:55:01.707983 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.707942    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/548b5a37-e500-45c6-b82f-13f2a74dbff8-kserve-provision-location\") pod \"isvc-canary-create-predictor-cb85cf55c-kzxmm\" (UID: \"548b5a37-e500-45c6-b82f-13f2a74dbff8\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kzxmm"
Jul 27 17:55:01.708120 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.708097    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/548b5a37-e500-45c6-b82f-13f2a74dbff8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-cb85cf55c-kzxmm\" (UID: \"548b5a37-e500-45c6-b82f-13f2a74dbff8\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kzxmm"
Jul 27 17:55:01.708347 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.708328    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/548b5a37-e500-45c6-b82f-13f2a74dbff8-cabundle-cert\") pod \"isvc-canary-create-predictor-cb85cf55c-kzxmm\" (UID: \"548b5a37-e500-45c6-b82f-13f2a74dbff8\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kzxmm"
Jul 27 17:55:01.709771 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.709754    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/548b5a37-e500-45c6-b82f-13f2a74dbff8-proxy-tls\") pod \"isvc-canary-create-predictor-cb85cf55c-kzxmm\" (UID: \"548b5a37-e500-45c6-b82f-13f2a74dbff8\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kzxmm"
Jul 27 17:55:01.715717 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.715697    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8855\" (UniqueName: \"kubernetes.io/projected/548b5a37-e500-45c6-b82f-13f2a74dbff8-kube-api-access-q8855\") pod \"isvc-canary-create-predictor-cb85cf55c-kzxmm\" (UID: \"548b5a37-e500-45c6-b82f-13f2a74dbff8\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kzxmm"
Jul 27 17:55:01.747928 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.747904    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8"
Jul 27 17:55:01.808894 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.808792    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/416e4a4e-94c3-4515-ad94-9fb495fcfe1b-proxy-tls\") pod \"416e4a4e-94c3-4515-ad94-9fb495fcfe1b\" (UID: \"416e4a4e-94c3-4515-ad94-9fb495fcfe1b\") "
Jul 27 17:55:01.808894 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.808855    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/416e4a4e-94c3-4515-ad94-9fb495fcfe1b-kserve-provision-location\") pod \"416e4a4e-94c3-4515-ad94-9fb495fcfe1b\" (UID: \"416e4a4e-94c3-4515-ad94-9fb495fcfe1b\") "
Jul 27 17:55:01.808894 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.808886    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/416e4a4e-94c3-4515-ad94-9fb495fcfe1b-kube-rbac-proxy-sar-config\") pod \"416e4a4e-94c3-4515-ad94-9fb495fcfe1b\" (UID: \"416e4a4e-94c3-4515-ad94-9fb495fcfe1b\") "
Jul 27 17:55:01.809174 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.808935    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4bssc\" (UniqueName: \"kubernetes.io/projected/416e4a4e-94c3-4515-ad94-9fb495fcfe1b-kube-api-access-4bssc\") pod \"416e4a4e-94c3-4515-ad94-9fb495fcfe1b\" (UID: \"416e4a4e-94c3-4515-ad94-9fb495fcfe1b\") "
Jul 27 17:55:01.809174 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.808997    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/416e4a4e-94c3-4515-ad94-9fb495fcfe1b-cabundle-cert\") pod \"416e4a4e-94c3-4515-ad94-9fb495fcfe1b\" (UID: \"416e4a4e-94c3-4515-ad94-9fb495fcfe1b\") "
Jul 27 17:55:01.809174 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.809110    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9775f83a-fe61-493f-bb43-38a1229ba941-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr\" (UID: \"9775f83a-fe61-493f-bb43-38a1229ba941\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr"
Jul 27 17:55:01.809373 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.809192    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7txpg\" (UniqueName: \"kubernetes.io/projected/9775f83a-fe61-493f-bb43-38a1229ba941-kube-api-access-7txpg\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr\" (UID: \"9775f83a-fe61-493f-bb43-38a1229ba941\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr"
Jul 27 17:55:01.809373 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.809242    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9775f83a-fe61-493f-bb43-38a1229ba941-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr\" (UID: \"9775f83a-fe61-493f-bb43-38a1229ba941\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr"
Jul 27 17:55:01.809373 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.809311    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9775f83a-fe61-493f-bb43-38a1229ba941-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr\" (UID: \"9775f83a-fe61-493f-bb43-38a1229ba941\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr"
Jul 27 17:55:01.809373 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.809359    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9775f83a-fe61-493f-bb43-38a1229ba941-proxy-tls\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr\" (UID: \"9775f83a-fe61-493f-bb43-38a1229ba941\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr"
Jul 27 17:55:01.809573 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.809392    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/416e4a4e-94c3-4515-ad94-9fb495fcfe1b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "416e4a4e-94c3-4515-ad94-9fb495fcfe1b" (UID: "416e4a4e-94c3-4515-ad94-9fb495fcfe1b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:55:01.810086 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.810041    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9775f83a-fe61-493f-bb43-38a1229ba941-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr\" (UID: \"9775f83a-fe61-493f-bb43-38a1229ba941\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr"
Jul 27 17:55:01.810229 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.810102    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9775f83a-fe61-493f-bb43-38a1229ba941-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr\" (UID: \"9775f83a-fe61-493f-bb43-38a1229ba941\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr"
Jul 27 17:55:01.810229 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.810152    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/416e4a4e-94c3-4515-ad94-9fb495fcfe1b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "416e4a4e-94c3-4515-ad94-9fb495fcfe1b" (UID: "416e4a4e-94c3-4515-ad94-9fb495fcfe1b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:55:01.810560 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.810528    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:55:01.810727 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.810707    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9775f83a-fe61-493f-bb43-38a1229ba941-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr\" (UID: \"9775f83a-fe61-493f-bb43-38a1229ba941\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr"
Jul 27 17:55:01.810860 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.810840    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/416e4a4e-94c3-4515-ad94-9fb495fcfe1b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "416e4a4e-94c3-4515-ad94-9fb495fcfe1b" (UID: "416e4a4e-94c3-4515-ad94-9fb495fcfe1b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:55:01.811375 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.811356    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/416e4a4e-94c3-4515-ad94-9fb495fcfe1b-kube-api-access-4bssc" (OuterVolumeSpecName: "kube-api-access-4bssc") pod "416e4a4e-94c3-4515-ad94-9fb495fcfe1b" (UID: "416e4a4e-94c3-4515-ad94-9fb495fcfe1b"). InnerVolumeSpecName "kube-api-access-4bssc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:55:01.811535 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.811523    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/416e4a4e-94c3-4515-ad94-9fb495fcfe1b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "416e4a4e-94c3-4515-ad94-9fb495fcfe1b" (UID: "416e4a4e-94c3-4515-ad94-9fb495fcfe1b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:55:01.811976 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.811948    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9775f83a-fe61-493f-bb43-38a1229ba941-proxy-tls\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr\" (UID: \"9775f83a-fe61-493f-bb43-38a1229ba941\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr"
Jul 27 17:55:01.818469 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.818445    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7txpg\" (UniqueName: \"kubernetes.io/projected/9775f83a-fe61-493f-bb43-38a1229ba941-kube-api-access-7txpg\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr\" (UID: \"9775f83a-fe61-493f-bb43-38a1229ba941\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr"
Jul 27 17:55:01.826657 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.826631    2573 generic.go:356] "Generic (PLEG): container finished" podID="416e4a4e-94c3-4515-ad94-9fb495fcfe1b" containerID="77618a08237c6d680868435e4aa8c4d5be22df7321278112a0bf779ffb185838" exitCode=0
Jul 27 17:55:01.826771 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.826687    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8"
Jul 27 17:55:01.826771 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.826713    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8" event={"ID":"416e4a4e-94c3-4515-ad94-9fb495fcfe1b","Type":"ContainerDied","Data":"77618a08237c6d680868435e4aa8c4d5be22df7321278112a0bf779ffb185838"}
Jul 27 17:55:01.826848 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.826782    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8" event={"ID":"416e4a4e-94c3-4515-ad94-9fb495fcfe1b","Type":"ContainerDied","Data":"90959d9d7e3550bf31b10016e201e4654cd68eff9f7ed358f035d20e21b70c98"}
Jul 27 17:55:01.826848 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.826809    2573 scope.go:117] "RemoveContainer" containerID="de72c20a732994d9a16558ac862f714c80b3d7fec052aeb77db3b41eee5a1cb4"
Jul 27 17:55:01.836423 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.836403    2573 scope.go:117] "RemoveContainer" containerID="77618a08237c6d680868435e4aa8c4d5be22df7321278112a0bf779ffb185838"
Jul 27 17:55:01.844262 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.844233    2573 scope.go:117] "RemoveContainer" containerID="02d8c7cc2c8a601084bff87b338ed8f70d1b835e4e10f8bcc35d178e50c9bfd9"
Jul 27 17:55:01.851857 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.851833    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8"]
Jul 27 17:55:01.852455 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.852440    2573 scope.go:117] "RemoveContainer" containerID="de72c20a732994d9a16558ac862f714c80b3d7fec052aeb77db3b41eee5a1cb4"
Jul 27 17:55:01.852740 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:55:01.852714    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de72c20a732994d9a16558ac862f714c80b3d7fec052aeb77db3b41eee5a1cb4\": container with ID starting with de72c20a732994d9a16558ac862f714c80b3d7fec052aeb77db3b41eee5a1cb4 not found: ID does not exist" containerID="de72c20a732994d9a16558ac862f714c80b3d7fec052aeb77db3b41eee5a1cb4"
Jul 27 17:55:01.852841 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.852742    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de72c20a732994d9a16558ac862f714c80b3d7fec052aeb77db3b41eee5a1cb4"} err="failed to get container status \"de72c20a732994d9a16558ac862f714c80b3d7fec052aeb77db3b41eee5a1cb4\": rpc error: code = NotFound desc = could not find container \"de72c20a732994d9a16558ac862f714c80b3d7fec052aeb77db3b41eee5a1cb4\": container with ID starting with de72c20a732994d9a16558ac862f714c80b3d7fec052aeb77db3b41eee5a1cb4 not found: ID does not exist"
Jul 27 17:55:01.852841 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.852761    2573 scope.go:117] "RemoveContainer" containerID="77618a08237c6d680868435e4aa8c4d5be22df7321278112a0bf779ffb185838"
Jul 27 17:55:01.853018 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:55:01.853000    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77618a08237c6d680868435e4aa8c4d5be22df7321278112a0bf779ffb185838\": container with ID starting with 77618a08237c6d680868435e4aa8c4d5be22df7321278112a0bf779ffb185838 not found: ID does not exist" containerID="77618a08237c6d680868435e4aa8c4d5be22df7321278112a0bf779ffb185838"
Jul 27 17:55:01.853058 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.853023    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77618a08237c6d680868435e4aa8c4d5be22df7321278112a0bf779ffb185838"} err="failed to get container status \"77618a08237c6d680868435e4aa8c4d5be22df7321278112a0bf779ffb185838\": rpc error: code = NotFound desc = could not find container \"77618a08237c6d680868435e4aa8c4d5be22df7321278112a0bf779ffb185838\": container with ID starting with 77618a08237c6d680868435e4aa8c4d5be22df7321278112a0bf779ffb185838 not found: ID does not exist"
Jul 27 17:55:01.853058 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.853037    2573 scope.go:117] "RemoveContainer" containerID="02d8c7cc2c8a601084bff87b338ed8f70d1b835e4e10f8bcc35d178e50c9bfd9"
Jul 27 17:55:01.853329 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:55:01.853312    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02d8c7cc2c8a601084bff87b338ed8f70d1b835e4e10f8bcc35d178e50c9bfd9\": container with ID starting with 02d8c7cc2c8a601084bff87b338ed8f70d1b835e4e10f8bcc35d178e50c9bfd9 not found: ID does not exist" containerID="02d8c7cc2c8a601084bff87b338ed8f70d1b835e4e10f8bcc35d178e50c9bfd9"
Jul 27 17:55:01.853402 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.853337    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02d8c7cc2c8a601084bff87b338ed8f70d1b835e4e10f8bcc35d178e50c9bfd9"} err="failed to get container status \"02d8c7cc2c8a601084bff87b338ed8f70d1b835e4e10f8bcc35d178e50c9bfd9\": rpc error: code = NotFound desc = could not find container \"02d8c7cc2c8a601084bff87b338ed8f70d1b835e4e10f8bcc35d178e50c9bfd9\": container with ID starting with 02d8c7cc2c8a601084bff87b338ed8f70d1b835e4e10f8bcc35d178e50c9bfd9 not found: ID does not exist"
Jul 27 17:55:01.857150 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.857127    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-jt6s8"]
Jul 27 17:55:01.910568 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.910531    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bssc\" (UniqueName: \"kubernetes.io/projected/416e4a4e-94c3-4515-ad94-9fb495fcfe1b-kube-api-access-4bssc\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:55:01.910568 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.910562    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/416e4a4e-94c3-4515-ad94-9fb495fcfe1b-cabundle-cert\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:55:01.910568 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.910573    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/416e4a4e-94c3-4515-ad94-9fb495fcfe1b-proxy-tls\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:55:01.910790 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.910592    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/416e4a4e-94c3-4515-ad94-9fb495fcfe1b-kserve-provision-location\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:55:01.910790 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.910601    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/416e4a4e-94c3-4515-ad94-9fb495fcfe1b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:55:01.919430 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.919399    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kzxmm"
Jul 27 17:55:01.924456 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:01.924440    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr"
Jul 27 17:55:02.068880 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:02.068851    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kzxmm"]
Jul 27 17:55:02.071533 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:55:02.071504    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod548b5a37_e500_45c6_b82f_13f2a74dbff8.slice/crio-8a15e5641e79fdc9698528457c470ac97b8ddbc740aa8d9cbda83c06c8c5644d WatchSource:0}: Error finding container 8a15e5641e79fdc9698528457c470ac97b8ddbc740aa8d9cbda83c06c8c5644d: Status 404 returned error can't find the container with id 8a15e5641e79fdc9698528457c470ac97b8ddbc740aa8d9cbda83c06c8c5644d
Jul 27 17:55:02.092406 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:02.092377    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr"]
Jul 27 17:55:02.102356 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:55:02.102325    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9775f83a_fe61_493f_bb43_38a1229ba941.slice/crio-ea6a5f5129965e33cb6a14b41223d35d1de2d95cae98c896c57516a16f4375af WatchSource:0}: Error finding container ea6a5f5129965e33cb6a14b41223d35d1de2d95cae98c896c57516a16f4375af: Status 404 returned error can't find the container with id ea6a5f5129965e33cb6a14b41223d35d1de2d95cae98c896c57516a16f4375af
Jul 27 17:55:02.470546 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:02.470514    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="416e4a4e-94c3-4515-ad94-9fb495fcfe1b" path="/var/lib/kubelet/pods/416e4a4e-94c3-4515-ad94-9fb495fcfe1b/volumes"
Jul 27 17:55:02.833376 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:02.833284    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr" event={"ID":"9775f83a-fe61-493f-bb43-38a1229ba941","Type":"ContainerStarted","Data":"4ff834dfda126367ee58b1ed8843bdf288cfd091cacd02180132b7c7d2708ffa"}
Jul 27 17:55:02.833376 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:02.833367    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr" event={"ID":"9775f83a-fe61-493f-bb43-38a1229ba941","Type":"ContainerStarted","Data":"ea6a5f5129965e33cb6a14b41223d35d1de2d95cae98c896c57516a16f4375af"}
Jul 27 17:55:02.834941 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:02.834917    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kzxmm" event={"ID":"548b5a37-e500-45c6-b82f-13f2a74dbff8","Type":"ContainerStarted","Data":"9f928e878e2ed1cc8b47d9cf0e22e84819a8970f812fcc2d8d26da01635c21d9"}
Jul 27 17:55:02.835043 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:02.834977    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kzxmm" event={"ID":"548b5a37-e500-45c6-b82f-13f2a74dbff8","Type":"ContainerStarted","Data":"8a15e5641e79fdc9698528457c470ac97b8ddbc740aa8d9cbda83c06c8c5644d"}
Jul 27 17:55:06.855558 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:06.855517    2573 generic.go:356] "Generic (PLEG): container finished" podID="9775f83a-fe61-493f-bb43-38a1229ba941" containerID="4ff834dfda126367ee58b1ed8843bdf288cfd091cacd02180132b7c7d2708ffa" exitCode=0
Jul 27 17:55:06.856033 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:06.855593    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr" event={"ID":"9775f83a-fe61-493f-bb43-38a1229ba941","Type":"ContainerDied","Data":"4ff834dfda126367ee58b1ed8843bdf288cfd091cacd02180132b7c7d2708ffa"}
Jul 27 17:55:06.857226 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:06.857204    2573 generic.go:356] "Generic (PLEG): container finished" podID="548b5a37-e500-45c6-b82f-13f2a74dbff8" containerID="9f928e878e2ed1cc8b47d9cf0e22e84819a8970f812fcc2d8d26da01635c21d9" exitCode=0
Jul 27 17:55:06.857321 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:06.857270    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kzxmm" event={"ID":"548b5a37-e500-45c6-b82f-13f2a74dbff8","Type":"ContainerDied","Data":"9f928e878e2ed1cc8b47d9cf0e22e84819a8970f812fcc2d8d26da01635c21d9"}
Jul 27 17:55:07.864463 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:07.864422    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kzxmm" event={"ID":"548b5a37-e500-45c6-b82f-13f2a74dbff8","Type":"ContainerStarted","Data":"dd3845f1c546d33035560ab131b92cf2bfd83280d3271d45297f98fa150f4192"}
Jul 27 17:55:07.864949 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:07.864496    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kzxmm" event={"ID":"548b5a37-e500-45c6-b82f-13f2a74dbff8","Type":"ContainerStarted","Data":"b312e97e310664120f3cf656575d550413f00d579893157d157ccab9a04e1c5a"}
Jul 27 17:55:07.864949 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:07.864681    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kzxmm"
Jul 27 17:55:07.866422 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:07.866399    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr" event={"ID":"9775f83a-fe61-493f-bb43-38a1229ba941","Type":"ContainerStarted","Data":"43912300d54deaa5e57abcfb9a6e22075fa131fdde392cfdbc7773cc3e623a6a"}
Jul 27 17:55:07.866515 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:07.866453    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr" event={"ID":"9775f83a-fe61-493f-bb43-38a1229ba941","Type":"ContainerStarted","Data":"afcebf5b0d8b4cd236af166f6b456c69bebe19e9d2e6049aa2f7a05e9d7ecb8f"}
Jul 27 17:55:07.866646 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:07.866632    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr"
Jul 27 17:55:07.890807 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:07.890754    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kzxmm" podStartSLOduration=3.045302198 podStartE2EDuration="6.890737183s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.845434985s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:55:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:55:07.88779712 +0000 UTC m=+1650.031199172" watchObservedRunningTime="2026-07-27 17:55:07.890737183 +0000 UTC m=+1650.034139233"
Jul 27 17:55:07.912035 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:07.911986    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr" podStartSLOduration=3.134654419 podStartE2EDuration="6.911971336s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.777316917s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:55:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:55:07.911020543 +0000 UTC m=+1650.054422594" watchObservedRunningTime="2026-07-27 17:55:07.911971336 +0000 UTC m=+1650.055373386"
Jul 27 17:55:08.871539 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:08.871499    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kzxmm"
Jul 27 17:55:08.872028 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:08.871563    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr"
Jul 27 17:55:08.872556 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:08.872519    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kzxmm" podUID="548b5a37-e500-45c6-b82f-13f2a74dbff8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 27 17:55:08.872649 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:08.872620    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr" podUID="9775f83a-fe61-493f-bb43-38a1229ba941" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 27 17:55:09.875708 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:09.875649    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr" podUID="9775f83a-fe61-493f-bb43-38a1229ba941" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 27 17:55:09.876122 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:09.875728    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kzxmm" podUID="548b5a37-e500-45c6-b82f-13f2a74dbff8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 27 17:55:14.880696 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:14.880661    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kzxmm"
Jul 27 17:55:14.881152 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:14.880829    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr"
Jul 27 17:55:14.881152 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:14.881086    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kzxmm" podUID="548b5a37-e500-45c6-b82f-13f2a74dbff8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 27 17:55:14.881537 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:14.881516    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr" podUID="9775f83a-fe61-493f-bb43-38a1229ba941" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 27 17:55:24.881800 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:24.881762    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr" podUID="9775f83a-fe61-493f-bb43-38a1229ba941" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 27 17:55:24.882211 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:24.881764    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kzxmm" podUID="548b5a37-e500-45c6-b82f-13f2a74dbff8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 27 17:55:34.882028 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:34.881990    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr" podUID="9775f83a-fe61-493f-bb43-38a1229ba941" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 27 17:55:34.882488 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:34.881997    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kzxmm" podUID="548b5a37-e500-45c6-b82f-13f2a74dbff8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 27 17:55:44.883192 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:44.883140    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kzxmm" podUID="548b5a37-e500-45c6-b82f-13f2a74dbff8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 27 17:55:44.883718 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:44.883601    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr" podUID="9775f83a-fe61-493f-bb43-38a1229ba941" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 27 17:55:54.881165 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:54.881123    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kzxmm" podUID="548b5a37-e500-45c6-b82f-13f2a74dbff8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 27 17:55:54.881583 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:55:54.881458    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr" podUID="9775f83a-fe61-493f-bb43-38a1229ba941" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 27 17:56:04.881701 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:04.881658    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kzxmm" podUID="548b5a37-e500-45c6-b82f-13f2a74dbff8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 27 17:56:04.882229 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:04.881658    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr" podUID="9775f83a-fe61-493f-bb43-38a1229ba941" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 27 17:56:14.881168 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:14.881129    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kzxmm" podUID="548b5a37-e500-45c6-b82f-13f2a74dbff8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 27 17:56:14.881606 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:14.881422    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr" podUID="9775f83a-fe61-493f-bb43-38a1229ba941" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 27 17:56:22.468780 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:22.468750    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr"
Jul 27 17:56:24.881391 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:24.881364    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kzxmm"
Jul 27 17:56:31.656773 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:31.656738    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kzxmm"]
Jul 27 17:56:31.657282 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:31.656974    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kzxmm" podUID="548b5a37-e500-45c6-b82f-13f2a74dbff8" containerName="kserve-container" containerID="cri-o://b312e97e310664120f3cf656575d550413f00d579893157d157ccab9a04e1c5a" gracePeriod=30
Jul 27 17:56:31.657282 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:31.657019    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kzxmm" podUID="548b5a37-e500-45c6-b82f-13f2a74dbff8" containerName="kube-rbac-proxy" containerID="cri-o://dd3845f1c546d33035560ab131b92cf2bfd83280d3271d45297f98fa150f4192" gracePeriod=30
Jul 27 17:56:31.661730 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:31.661703    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr"]
Jul 27 17:56:31.662429 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:31.661963    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr" podUID="9775f83a-fe61-493f-bb43-38a1229ba941" containerName="kserve-container" containerID="cri-o://afcebf5b0d8b4cd236af166f6b456c69bebe19e9d2e6049aa2f7a05e9d7ecb8f" gracePeriod=30
Jul 27 17:56:31.662429 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:31.662042    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr" podUID="9775f83a-fe61-493f-bb43-38a1229ba941" containerName="kube-rbac-proxy" containerID="cri-o://43912300d54deaa5e57abcfb9a6e22075fa131fdde392cfdbc7773cc3e623a6a" gracePeriod=30
Jul 27 17:56:31.668206 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:31.668178    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:56:32.256177 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:32.256142    2573 generic.go:356] "Generic (PLEG): container finished" podID="9775f83a-fe61-493f-bb43-38a1229ba941" containerID="43912300d54deaa5e57abcfb9a6e22075fa131fdde392cfdbc7773cc3e623a6a" exitCode=2
Jul 27 17:56:32.256373 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:32.256213    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr" event={"ID":"9775f83a-fe61-493f-bb43-38a1229ba941","Type":"ContainerDied","Data":"43912300d54deaa5e57abcfb9a6e22075fa131fdde392cfdbc7773cc3e623a6a"}
Jul 27 17:56:32.258090 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:32.258064    2573 generic.go:356] "Generic (PLEG): container finished" podID="548b5a37-e500-45c6-b82f-13f2a74dbff8" containerID="dd3845f1c546d33035560ab131b92cf2bfd83280d3271d45297f98fa150f4192" exitCode=2
Jul 27 17:56:32.258209 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:32.258134    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kzxmm" event={"ID":"548b5a37-e500-45c6-b82f-13f2a74dbff8","Type":"ContainerDied","Data":"dd3845f1c546d33035560ab131b92cf2bfd83280d3271d45297f98fa150f4192"}
Jul 27 17:56:32.464783 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:32.464744    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr" podUID="9775f83a-fe61-493f-bb43-38a1229ba941" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 27 17:56:33.657013 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:33.656968    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:56:34.876436 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:34.876395    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kzxmm" podUID="548b5a37-e500-45c6-b82f-13f2a74dbff8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
Jul 27 17:56:34.876803 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:34.876399    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr" podUID="9775f83a-fe61-493f-bb43-38a1229ba941" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.39:8643/healthz\": dial tcp 10.133.0.39:8643: connect: connection refused"
Jul 27 17:56:34.881827 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:34.881802    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kzxmm" podUID="548b5a37-e500-45c6-b82f-13f2a74dbff8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 27 17:56:36.277125 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.277093    2573 generic.go:356] "Generic (PLEG): container finished" podID="548b5a37-e500-45c6-b82f-13f2a74dbff8" containerID="b312e97e310664120f3cf656575d550413f00d579893157d157ccab9a04e1c5a" exitCode=0
Jul 27 17:56:36.277630 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.277169    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kzxmm" event={"ID":"548b5a37-e500-45c6-b82f-13f2a74dbff8","Type":"ContainerDied","Data":"b312e97e310664120f3cf656575d550413f00d579893157d157ccab9a04e1c5a"}
Jul 27 17:56:36.420767 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.420742    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kzxmm"
Jul 27 17:56:36.524596 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.524573    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr"
Jul 27 17:56:36.525705 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.525684    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/548b5a37-e500-45c6-b82f-13f2a74dbff8-kserve-provision-location\") pod \"548b5a37-e500-45c6-b82f-13f2a74dbff8\" (UID: \"548b5a37-e500-45c6-b82f-13f2a74dbff8\") "
Jul 27 17:56:36.525795 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.525734    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8855\" (UniqueName: \"kubernetes.io/projected/548b5a37-e500-45c6-b82f-13f2a74dbff8-kube-api-access-q8855\") pod \"548b5a37-e500-45c6-b82f-13f2a74dbff8\" (UID: \"548b5a37-e500-45c6-b82f-13f2a74dbff8\") "
Jul 27 17:56:36.525795 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.525784    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/548b5a37-e500-45c6-b82f-13f2a74dbff8-proxy-tls\") pod \"548b5a37-e500-45c6-b82f-13f2a74dbff8\" (UID: \"548b5a37-e500-45c6-b82f-13f2a74dbff8\") "
Jul 27 17:56:36.525912 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.525837    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/548b5a37-e500-45c6-b82f-13f2a74dbff8-kube-rbac-proxy-sar-config\") pod \"548b5a37-e500-45c6-b82f-13f2a74dbff8\" (UID: \"548b5a37-e500-45c6-b82f-13f2a74dbff8\") "
Jul 27 17:56:36.525912 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.525879    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/548b5a37-e500-45c6-b82f-13f2a74dbff8-cabundle-cert\") pod \"548b5a37-e500-45c6-b82f-13f2a74dbff8\" (UID: \"548b5a37-e500-45c6-b82f-13f2a74dbff8\") "
Jul 27 17:56:36.526013 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.525983    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/548b5a37-e500-45c6-b82f-13f2a74dbff8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "548b5a37-e500-45c6-b82f-13f2a74dbff8" (UID: "548b5a37-e500-45c6-b82f-13f2a74dbff8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:56:36.526153 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.526137    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/548b5a37-e500-45c6-b82f-13f2a74dbff8-kserve-provision-location\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:56:36.526265 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.526222    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/548b5a37-e500-45c6-b82f-13f2a74dbff8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "548b5a37-e500-45c6-b82f-13f2a74dbff8" (UID: "548b5a37-e500-45c6-b82f-13f2a74dbff8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:56:36.526338 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.526262    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/548b5a37-e500-45c6-b82f-13f2a74dbff8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "548b5a37-e500-45c6-b82f-13f2a74dbff8" (UID: "548b5a37-e500-45c6-b82f-13f2a74dbff8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:56:36.527772 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.527753    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/548b5a37-e500-45c6-b82f-13f2a74dbff8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "548b5a37-e500-45c6-b82f-13f2a74dbff8" (UID: "548b5a37-e500-45c6-b82f-13f2a74dbff8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:56:36.527931 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.527914    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/548b5a37-e500-45c6-b82f-13f2a74dbff8-kube-api-access-q8855" (OuterVolumeSpecName: "kube-api-access-q8855") pod "548b5a37-e500-45c6-b82f-13f2a74dbff8" (UID: "548b5a37-e500-45c6-b82f-13f2a74dbff8"). InnerVolumeSpecName "kube-api-access-q8855". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:56:36.626924 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.626896    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9775f83a-fe61-493f-bb43-38a1229ba941-proxy-tls\") pod \"9775f83a-fe61-493f-bb43-38a1229ba941\" (UID: \"9775f83a-fe61-493f-bb43-38a1229ba941\") "
Jul 27 17:56:36.627094 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.626938    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9775f83a-fe61-493f-bb43-38a1229ba941-kserve-provision-location\") pod \"9775f83a-fe61-493f-bb43-38a1229ba941\" (UID: \"9775f83a-fe61-493f-bb43-38a1229ba941\") "
Jul 27 17:56:36.627094 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.626958    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7txpg\" (UniqueName: \"kubernetes.io/projected/9775f83a-fe61-493f-bb43-38a1229ba941-kube-api-access-7txpg\") pod \"9775f83a-fe61-493f-bb43-38a1229ba941\" (UID: \"9775f83a-fe61-493f-bb43-38a1229ba941\") "
Jul 27 17:56:36.627094 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.626980    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9775f83a-fe61-493f-bb43-38a1229ba941-cabundle-cert\") pod \"9775f83a-fe61-493f-bb43-38a1229ba941\" (UID: \"9775f83a-fe61-493f-bb43-38a1229ba941\") "
Jul 27 17:56:36.627094 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.627080    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9775f83a-fe61-493f-bb43-38a1229ba941-kube-rbac-proxy-sar-config\") pod \"9775f83a-fe61-493f-bb43-38a1229ba941\" (UID: \"9775f83a-fe61-493f-bb43-38a1229ba941\") "
Jul 27 17:56:36.627359 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.627332    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9775f83a-fe61-493f-bb43-38a1229ba941-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9775f83a-fe61-493f-bb43-38a1229ba941" (UID: "9775f83a-fe61-493f-bb43-38a1229ba941"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:56:36.627428 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.627359    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8855\" (UniqueName: \"kubernetes.io/projected/548b5a37-e500-45c6-b82f-13f2a74dbff8-kube-api-access-q8855\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:56:36.627428 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.627342    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9775f83a-fe61-493f-bb43-38a1229ba941-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9775f83a-fe61-493f-bb43-38a1229ba941" (UID: "9775f83a-fe61-493f-bb43-38a1229ba941"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:56:36.627428 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.627378    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/548b5a37-e500-45c6-b82f-13f2a74dbff8-proxy-tls\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:56:36.627428 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.627388    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/548b5a37-e500-45c6-b82f-13f2a74dbff8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:56:36.627428 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.627398    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/548b5a37-e500-45c6-b82f-13f2a74dbff8-cabundle-cert\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:56:36.627634 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.627429    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9775f83a-fe61-493f-bb43-38a1229ba941-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9775f83a-fe61-493f-bb43-38a1229ba941" (UID: "9775f83a-fe61-493f-bb43-38a1229ba941"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:56:36.628819 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.628792    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9775f83a-fe61-493f-bb43-38a1229ba941-kube-api-access-7txpg" (OuterVolumeSpecName: "kube-api-access-7txpg") pod "9775f83a-fe61-493f-bb43-38a1229ba941" (UID: "9775f83a-fe61-493f-bb43-38a1229ba941"). InnerVolumeSpecName "kube-api-access-7txpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:56:36.628904 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.628856    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9775f83a-fe61-493f-bb43-38a1229ba941-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9775f83a-fe61-493f-bb43-38a1229ba941" (UID: "9775f83a-fe61-493f-bb43-38a1229ba941"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:56:36.728017 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.727981    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9775f83a-fe61-493f-bb43-38a1229ba941-proxy-tls\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:56:36.728017 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.728012    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9775f83a-fe61-493f-bb43-38a1229ba941-kserve-provision-location\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:56:36.728275 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.728029    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7txpg\" (UniqueName: \"kubernetes.io/projected/9775f83a-fe61-493f-bb43-38a1229ba941-kube-api-access-7txpg\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:56:36.728275 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.728042    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9775f83a-fe61-493f-bb43-38a1229ba941-cabundle-cert\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:56:36.728275 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.728055    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9775f83a-fe61-493f-bb43-38a1229ba941-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:56:36.732110 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.732078    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:56:36.818461 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.818370    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:56:36.819457 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.819435    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-f492w"]
Jul 27 17:56:36.819752 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.819740    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="416e4a4e-94c3-4515-ad94-9fb495fcfe1b" containerName="storage-initializer"
Jul 27 17:56:36.819804 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.819754    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="416e4a4e-94c3-4515-ad94-9fb495fcfe1b" containerName="storage-initializer"
Jul 27 17:56:36.819804 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.819765    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="548b5a37-e500-45c6-b82f-13f2a74dbff8" containerName="kube-rbac-proxy"
Jul 27 17:56:36.819804 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.819771    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="548b5a37-e500-45c6-b82f-13f2a74dbff8" containerName="kube-rbac-proxy"
Jul 27 17:56:36.819804 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.819782    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="548b5a37-e500-45c6-b82f-13f2a74dbff8" containerName="storage-initializer"
Jul 27 17:56:36.819804 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.819787    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="548b5a37-e500-45c6-b82f-13f2a74dbff8" containerName="storage-initializer"
Jul 27 17:56:36.819804 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.819793    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="548b5a37-e500-45c6-b82f-13f2a74dbff8" containerName="kserve-container"
Jul 27 17:56:36.819804 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.819801    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="548b5a37-e500-45c6-b82f-13f2a74dbff8" containerName="kserve-container"
Jul 27 17:56:36.820048 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.819812    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9775f83a-fe61-493f-bb43-38a1229ba941" containerName="kube-rbac-proxy"
Jul 27 17:56:36.820048 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.819822    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9775f83a-fe61-493f-bb43-38a1229ba941" containerName="kube-rbac-proxy"
Jul 27 17:56:36.820048 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.819831    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9775f83a-fe61-493f-bb43-38a1229ba941" containerName="storage-initializer"
Jul 27 17:56:36.820048 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.819839    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9775f83a-fe61-493f-bb43-38a1229ba941" containerName="storage-initializer"
Jul 27 17:56:36.820048 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.819847    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9775f83a-fe61-493f-bb43-38a1229ba941" containerName="kserve-container"
Jul 27 17:56:36.820048 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.819855    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9775f83a-fe61-493f-bb43-38a1229ba941" containerName="kserve-container"
Jul 27 17:56:36.820048 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.819879    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="416e4a4e-94c3-4515-ad94-9fb495fcfe1b" containerName="kube-rbac-proxy"
Jul 27 17:56:36.820048 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.819888    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="416e4a4e-94c3-4515-ad94-9fb495fcfe1b" containerName="kube-rbac-proxy"
Jul 27 17:56:36.820048 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.819900    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="416e4a4e-94c3-4515-ad94-9fb495fcfe1b" containerName="kserve-container"
Jul 27 17:56:36.820048 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.819908    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="416e4a4e-94c3-4515-ad94-9fb495fcfe1b" containerName="kserve-container"
Jul 27 17:56:36.820048 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.819988    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="548b5a37-e500-45c6-b82f-13f2a74dbff8" containerName="kserve-container"
Jul 27 17:56:36.820048 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.820002    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9775f83a-fe61-493f-bb43-38a1229ba941" containerName="kserve-container"
Jul 27 17:56:36.820048 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.820011    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="548b5a37-e500-45c6-b82f-13f2a74dbff8" containerName="kube-rbac-proxy"
Jul 27 17:56:36.820048 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.820020    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="416e4a4e-94c3-4515-ad94-9fb495fcfe1b" containerName="kserve-container"
Jul 27 17:56:36.820048 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.820030    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9775f83a-fe61-493f-bb43-38a1229ba941" containerName="kube-rbac-proxy"
Jul 27 17:56:36.820048 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.820037    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="416e4a4e-94c3-4515-ad94-9fb495fcfe1b" containerName="kube-rbac-proxy"
Jul 27 17:56:36.824235 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.824218    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-f492w"
Jul 27 17:56:36.827214 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.827196    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Jul 27 17:56:36.827334 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.827202    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Jul 27 17:56:36.831186 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.831167    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-cq4vs"]
Jul 27 17:56:36.834573 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.834552    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-f492w"]
Jul 27 17:56:36.834691 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.834647    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-cq4vs"
Jul 27 17:56:36.837473 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.837455    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Jul 27 17:56:36.844885 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.844861    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-cq4vs"]
Jul 27 17:56:36.929502 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.929472    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdqhv\" (UniqueName: \"kubernetes.io/projected/aab97251-fd04-4e56-85d0-f8b76ddd901d-kube-api-access-fdqhv\") pod \"isvc-canary-promote-predictor-d9759ffd5-f492w\" (UID: \"aab97251-fd04-4e56-85d0-f8b76ddd901d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-f492w"
Jul 27 17:56:36.929638 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.929509    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f17000c-4324-4ca5-9f04-f41b2198fe50-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-86766c4958-cq4vs\" (UID: \"1f17000c-4324-4ca5-9f04-f41b2198fe50\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-cq4vs"
Jul 27 17:56:36.929638 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.929570    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f17000c-4324-4ca5-9f04-f41b2198fe50-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-86766c4958-cq4vs\" (UID: \"1f17000c-4324-4ca5-9f04-f41b2198fe50\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-cq4vs"
Jul 27 17:56:36.929638 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.929599    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aab97251-fd04-4e56-85d0-f8b76ddd901d-proxy-tls\") pod \"isvc-canary-promote-predictor-d9759ffd5-f492w\" (UID: \"aab97251-fd04-4e56-85d0-f8b76ddd901d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-f492w"
Jul 27 17:56:36.929638 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.929624    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7rkd\" (UniqueName: \"kubernetes.io/projected/1f17000c-4324-4ca5-9f04-f41b2198fe50-kube-api-access-q7rkd\") pod \"isvc-canary-promote-v2-predictor-86766c4958-cq4vs\" (UID: \"1f17000c-4324-4ca5-9f04-f41b2198fe50\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-cq4vs"
Jul 27 17:56:36.929810 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.929647    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aab97251-fd04-4e56-85d0-f8b76ddd901d-kserve-provision-location\") pod \"isvc-canary-promote-predictor-d9759ffd5-f492w\" (UID: \"aab97251-fd04-4e56-85d0-f8b76ddd901d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-f492w"
Jul 27 17:56:36.929810 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.929672    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aab97251-fd04-4e56-85d0-f8b76ddd901d-cabundle-cert\") pod \"isvc-canary-promote-predictor-d9759ffd5-f492w\" (UID: \"aab97251-fd04-4e56-85d0-f8b76ddd901d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-f492w"
Jul 27 17:56:36.929810 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.929693    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1f17000c-4324-4ca5-9f04-f41b2198fe50-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-86766c4958-cq4vs\" (UID: \"1f17000c-4324-4ca5-9f04-f41b2198fe50\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-cq4vs"
Jul 27 17:56:36.929810 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.929713    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aab97251-fd04-4e56-85d0-f8b76ddd901d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-d9759ffd5-f492w\" (UID: \"aab97251-fd04-4e56-85d0-f8b76ddd901d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-f492w"
Jul 27 17:56:36.929810 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.929728    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f17000c-4324-4ca5-9f04-f41b2198fe50-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-86766c4958-cq4vs\" (UID: \"1f17000c-4324-4ca5-9f04-f41b2198fe50\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-cq4vs"
Jul 27 17:56:36.992299 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:36.992270    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:56:37.031080 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:37.031056    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f17000c-4324-4ca5-9f04-f41b2198fe50-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-86766c4958-cq4vs\" (UID: \"1f17000c-4324-4ca5-9f04-f41b2198fe50\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-cq4vs"
Jul 27 17:56:37.031216 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:37.031089    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aab97251-fd04-4e56-85d0-f8b76ddd901d-proxy-tls\") pod \"isvc-canary-promote-predictor-d9759ffd5-f492w\" (UID: \"aab97251-fd04-4e56-85d0-f8b76ddd901d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-f492w"
Jul 27 17:56:37.031216 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:37.031107    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7rkd\" (UniqueName: \"kubernetes.io/projected/1f17000c-4324-4ca5-9f04-f41b2198fe50-kube-api-access-q7rkd\") pod \"isvc-canary-promote-v2-predictor-86766c4958-cq4vs\" (UID: \"1f17000c-4324-4ca5-9f04-f41b2198fe50\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-cq4vs"
Jul 27 17:56:37.031216 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:37.031136    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aab97251-fd04-4e56-85d0-f8b76ddd901d-kserve-provision-location\") pod \"isvc-canary-promote-predictor-d9759ffd5-f492w\" (UID: \"aab97251-fd04-4e56-85d0-f8b76ddd901d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-f492w"
Jul 27 17:56:37.031216 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:37.031169    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aab97251-fd04-4e56-85d0-f8b76ddd901d-cabundle-cert\") pod \"isvc-canary-promote-predictor-d9759ffd5-f492w\" (UID: \"aab97251-fd04-4e56-85d0-f8b76ddd901d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-f492w"
Jul 27 17:56:37.031216 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:37.031193    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1f17000c-4324-4ca5-9f04-f41b2198fe50-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-86766c4958-cq4vs\" (UID: \"1f17000c-4324-4ca5-9f04-f41b2198fe50\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-cq4vs"
Jul 27 17:56:37.031528 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:56:37.031225    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found
Jul 27 17:56:37.031528 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:56:37.031308    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aab97251-fd04-4e56-85d0-f8b76ddd901d-proxy-tls podName:aab97251-fd04-4e56-85d0-f8b76ddd901d nodeName:}" failed. No retries permitted until 2026-07-27 17:56:37.531287782 +0000 UTC m=+1739.674689812 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/aab97251-fd04-4e56-85d0-f8b76ddd901d-proxy-tls") pod "isvc-canary-promote-predictor-d9759ffd5-f492w" (UID: "aab97251-fd04-4e56-85d0-f8b76ddd901d") : secret "isvc-canary-promote-predictor-serving-cert" not found
Jul 27 17:56:37.031528 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:37.031224    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aab97251-fd04-4e56-85d0-f8b76ddd901d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-d9759ffd5-f492w\" (UID: \"aab97251-fd04-4e56-85d0-f8b76ddd901d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-f492w"
Jul 27 17:56:37.031528 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:37.031351    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f17000c-4324-4ca5-9f04-f41b2198fe50-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-86766c4958-cq4vs\" (UID: \"1f17000c-4324-4ca5-9f04-f41b2198fe50\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-cq4vs"
Jul 27 17:56:37.031528 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:37.031406    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdqhv\" (UniqueName: \"kubernetes.io/projected/aab97251-fd04-4e56-85d0-f8b76ddd901d-kube-api-access-fdqhv\") pod \"isvc-canary-promote-predictor-d9759ffd5-f492w\" (UID: \"aab97251-fd04-4e56-85d0-f8b76ddd901d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-f492w"
Jul 27 17:56:37.031528 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:37.031430    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f17000c-4324-4ca5-9f04-f41b2198fe50-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-86766c4958-cq4vs\" (UID: \"1f17000c-4324-4ca5-9f04-f41b2198fe50\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-cq4vs"
Jul 27 17:56:37.031528 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:37.031494    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f17000c-4324-4ca5-9f04-f41b2198fe50-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-86766c4958-cq4vs\" (UID: \"1f17000c-4324-4ca5-9f04-f41b2198fe50\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-cq4vs"
Jul 27 17:56:37.031849 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:37.031752    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aab97251-fd04-4e56-85d0-f8b76ddd901d-kserve-provision-location\") pod \"isvc-canary-promote-predictor-d9759ffd5-f492w\" (UID: \"aab97251-fd04-4e56-85d0-f8b76ddd901d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-f492w"
Jul 27 17:56:37.031996 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:37.031978    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1f17000c-4324-4ca5-9f04-f41b2198fe50-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-86766c4958-cq4vs\" (UID: \"1f17000c-4324-4ca5-9f04-f41b2198fe50\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-cq4vs"
Jul 27 17:56:37.032046 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:37.031990    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f17000c-4324-4ca5-9f04-f41b2198fe50-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-86766c4958-cq4vs\" (UID: \"1f17000c-4324-4ca5-9f04-f41b2198fe50\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-cq4vs"
Jul 27 17:56:37.032046 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:37.032008    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aab97251-fd04-4e56-85d0-f8b76ddd901d-cabundle-cert\") pod \"isvc-canary-promote-predictor-d9759ffd5-f492w\" (UID: \"aab97251-fd04-4e56-85d0-f8b76ddd901d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-f492w"
Jul 27 17:56:37.032151 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:37.032135    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aab97251-fd04-4e56-85d0-f8b76ddd901d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-d9759ffd5-f492w\" (UID: \"aab97251-fd04-4e56-85d0-f8b76ddd901d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-f492w"
Jul 27 17:56:37.033830 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:37.033809    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f17000c-4324-4ca5-9f04-f41b2198fe50-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-86766c4958-cq4vs\" (UID: \"1f17000c-4324-4ca5-9f04-f41b2198fe50\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-cq4vs"
Jul 27 17:56:37.040491 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:37.040472    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdqhv\" (UniqueName: \"kubernetes.io/projected/aab97251-fd04-4e56-85d0-f8b76ddd901d-kube-api-access-fdqhv\") pod \"isvc-canary-promote-predictor-d9759ffd5-f492w\" (UID: \"aab97251-fd04-4e56-85d0-f8b76ddd901d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-f492w"
Jul 27 17:56:37.040578 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:37.040509    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7rkd\" (UniqueName: \"kubernetes.io/projected/1f17000c-4324-4ca5-9f04-f41b2198fe50-kube-api-access-q7rkd\") pod \"isvc-canary-promote-v2-predictor-86766c4958-cq4vs\" (UID: \"1f17000c-4324-4ca5-9f04-f41b2198fe50\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-cq4vs"
Jul 27 17:56:37.146605 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:37.146573    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-cq4vs"
Jul 27 17:56:37.284228 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:37.284183    2573 generic.go:356] "Generic (PLEG): container finished" podID="9775f83a-fe61-493f-bb43-38a1229ba941" containerID="afcebf5b0d8b4cd236af166f6b456c69bebe19e9d2e6049aa2f7a05e9d7ecb8f" exitCode=0
Jul 27 17:56:37.284670 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:37.284306    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr"
Jul 27 17:56:37.284670 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:37.284315    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr" event={"ID":"9775f83a-fe61-493f-bb43-38a1229ba941","Type":"ContainerDied","Data":"afcebf5b0d8b4cd236af166f6b456c69bebe19e9d2e6049aa2f7a05e9d7ecb8f"}
Jul 27 17:56:37.284670 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:37.284389    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr" event={"ID":"9775f83a-fe61-493f-bb43-38a1229ba941","Type":"ContainerDied","Data":"ea6a5f5129965e33cb6a14b41223d35d1de2d95cae98c896c57516a16f4375af"}
Jul 27 17:56:37.284670 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:37.284427    2573 scope.go:117] "RemoveContainer" containerID="43912300d54deaa5e57abcfb9a6e22075fa131fdde392cfdbc7773cc3e623a6a"
Jul 27 17:56:37.286449 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:37.286413    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-cq4vs"]
Jul 27 17:56:37.286815 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:37.286771    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kzxmm"
Jul 27 17:56:37.286815 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:37.286786    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kzxmm" event={"ID":"548b5a37-e500-45c6-b82f-13f2a74dbff8","Type":"ContainerDied","Data":"8a15e5641e79fdc9698528457c470ac97b8ddbc740aa8d9cbda83c06c8c5644d"}
Jul 27 17:56:37.287152 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:37.287129    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:56:37.288873 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:56:37.288832    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f17000c_4324_4ca5_9f04_f41b2198fe50.slice/crio-3b164de25e6c02e7399d8d146e83552347def8c6a7412578b660016ecc2576da WatchSource:0}: Error finding container 3b164de25e6c02e7399d8d146e83552347def8c6a7412578b660016ecc2576da: Status 404 returned error can't find the container with id 3b164de25e6c02e7399d8d146e83552347def8c6a7412578b660016ecc2576da
Jul 27 17:56:37.303130 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:37.303109    2573 scope.go:117] "RemoveContainer" containerID="afcebf5b0d8b4cd236af166f6b456c69bebe19e9d2e6049aa2f7a05e9d7ecb8f"
Jul 27 17:56:37.315544 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:37.315529    2573 scope.go:117] "RemoveContainer" containerID="4ff834dfda126367ee58b1ed8843bdf288cfd091cacd02180132b7c7d2708ffa"
Jul 27 17:56:37.315648 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:37.315629    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr"]
Jul 27 17:56:37.319666 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:37.319645    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-xtrlr"]
Jul 27 17:56:37.329780 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:37.329764    2573 scope.go:117] "RemoveContainer" containerID="43912300d54deaa5e57abcfb9a6e22075fa131fdde392cfdbc7773cc3e623a6a"
Jul 27 17:56:37.330078 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:56:37.330052    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43912300d54deaa5e57abcfb9a6e22075fa131fdde392cfdbc7773cc3e623a6a\": container with ID starting with 43912300d54deaa5e57abcfb9a6e22075fa131fdde392cfdbc7773cc3e623a6a not found: ID does not exist" containerID="43912300d54deaa5e57abcfb9a6e22075fa131fdde392cfdbc7773cc3e623a6a"
Jul 27 17:56:37.330172 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:37.330089    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43912300d54deaa5e57abcfb9a6e22075fa131fdde392cfdbc7773cc3e623a6a"} err="failed to get container status \"43912300d54deaa5e57abcfb9a6e22075fa131fdde392cfdbc7773cc3e623a6a\": rpc error: code = NotFound desc = could not find container \"43912300d54deaa5e57abcfb9a6e22075fa131fdde392cfdbc7773cc3e623a6a\": container with ID starting with 43912300d54deaa5e57abcfb9a6e22075fa131fdde392cfdbc7773cc3e623a6a not found: ID does not exist"
Jul 27 17:56:37.330172 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:37.330112    2573 scope.go:117] "RemoveContainer" containerID="afcebf5b0d8b4cd236af166f6b456c69bebe19e9d2e6049aa2f7a05e9d7ecb8f"
Jul 27 17:56:37.330562 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:56:37.330538    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afcebf5b0d8b4cd236af166f6b456c69bebe19e9d2e6049aa2f7a05e9d7ecb8f\": container with ID starting with afcebf5b0d8b4cd236af166f6b456c69bebe19e9d2e6049aa2f7a05e9d7ecb8f not found: ID does not exist" containerID="afcebf5b0d8b4cd236af166f6b456c69bebe19e9d2e6049aa2f7a05e9d7ecb8f"
Jul 27 17:56:37.330659 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:37.330571    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afcebf5b0d8b4cd236af166f6b456c69bebe19e9d2e6049aa2f7a05e9d7ecb8f"} err="failed to get container status \"afcebf5b0d8b4cd236af166f6b456c69bebe19e9d2e6049aa2f7a05e9d7ecb8f\": rpc error: code = NotFound desc = could not find container \"afcebf5b0d8b4cd236af166f6b456c69bebe19e9d2e6049aa2f7a05e9d7ecb8f\": container with ID starting with afcebf5b0d8b4cd236af166f6b456c69bebe19e9d2e6049aa2f7a05e9d7ecb8f not found: ID does not exist"
Jul 27 17:56:37.330659 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:37.330592    2573 scope.go:117] "RemoveContainer" containerID="4ff834dfda126367ee58b1ed8843bdf288cfd091cacd02180132b7c7d2708ffa"
Jul 27 17:56:37.330932 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:56:37.330913    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ff834dfda126367ee58b1ed8843bdf288cfd091cacd02180132b7c7d2708ffa\": container with ID starting with 4ff834dfda126367ee58b1ed8843bdf288cfd091cacd02180132b7c7d2708ffa not found: ID does not exist" containerID="4ff834dfda126367ee58b1ed8843bdf288cfd091cacd02180132b7c7d2708ffa"
Jul 27 17:56:37.330995 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:37.330939    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ff834dfda126367ee58b1ed8843bdf288cfd091cacd02180132b7c7d2708ffa"} err="failed to get container status \"4ff834dfda126367ee58b1ed8843bdf288cfd091cacd02180132b7c7d2708ffa\": rpc error: code = NotFound desc = could not find container \"4ff834dfda126367ee58b1ed8843bdf288cfd091cacd02180132b7c7d2708ffa\": container with ID starting with 4ff834dfda126367ee58b1ed8843bdf288cfd091cacd02180132b7c7d2708ffa not found: ID does not exist"
Jul 27 17:56:37.330995 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:37.330960    2573 scope.go:117] "RemoveContainer" containerID="dd3845f1c546d33035560ab131b92cf2bfd83280d3271d45297f98fa150f4192"
Jul 27 17:56:37.333544 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:37.333526    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kzxmm"]
Jul 27 17:56:37.335363 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:37.335341    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-kzxmm"]
Jul 27 17:56:37.339011 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:37.338995    2573 scope.go:117] "RemoveContainer" containerID="b312e97e310664120f3cf656575d550413f00d579893157d157ccab9a04e1c5a"
Jul 27 17:56:37.346153 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:37.346135    2573 scope.go:117] "RemoveContainer" containerID="9f928e878e2ed1cc8b47d9cf0e22e84819a8970f812fcc2d8d26da01635c21d9"
Jul 27 17:56:37.535991 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:37.535899    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aab97251-fd04-4e56-85d0-f8b76ddd901d-proxy-tls\") pod \"isvc-canary-promote-predictor-d9759ffd5-f492w\" (UID: \"aab97251-fd04-4e56-85d0-f8b76ddd901d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-f492w"
Jul 27 17:56:37.538239 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:37.538204    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aab97251-fd04-4e56-85d0-f8b76ddd901d-proxy-tls\") pod \"isvc-canary-promote-predictor-d9759ffd5-f492w\" (UID: \"aab97251-fd04-4e56-85d0-f8b76ddd901d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-f492w"
Jul 27 17:56:37.735387 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:37.735346    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-f492w"
Jul 27 17:56:37.871226 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:37.871194    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-f492w"]
Jul 27 17:56:37.873486 ip-10-0-140-0 kubenswrapper[2573]: W0727 17:56:37.873461    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaab97251_fd04_4e56_85d0_f8b76ddd901d.slice/crio-7fa5d5c1e7d6b6315d17e7450782f08b306cc872bb1f6d1ef8b82518a8d149b2 WatchSource:0}: Error finding container 7fa5d5c1e7d6b6315d17e7450782f08b306cc872bb1f6d1ef8b82518a8d149b2: Status 404 returned error can't find the container with id 7fa5d5c1e7d6b6315d17e7450782f08b306cc872bb1f6d1ef8b82518a8d149b2
Jul 27 17:56:38.293868 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:38.293832    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-f492w" event={"ID":"aab97251-fd04-4e56-85d0-f8b76ddd901d","Type":"ContainerStarted","Data":"ae05510e18c350f3eef6c85ffa017eb63601b702230d7082b820c2c3f31eccb2"}
Jul 27 17:56:38.294287 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:38.293897    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-f492w" event={"ID":"aab97251-fd04-4e56-85d0-f8b76ddd901d","Type":"ContainerStarted","Data":"7fa5d5c1e7d6b6315d17e7450782f08b306cc872bb1f6d1ef8b82518a8d149b2"}
Jul 27 17:56:38.295416 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:38.295391    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-cq4vs" event={"ID":"1f17000c-4324-4ca5-9f04-f41b2198fe50","Type":"ContainerStarted","Data":"e2a9e59fe8e88d4e64caee7c6eab9b5812ea44b8e1ca2523e5d359d1a22a45d0"}
Jul 27 17:56:38.295541 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:38.295451    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-cq4vs" event={"ID":"1f17000c-4324-4ca5-9f04-f41b2198fe50","Type":"ContainerStarted","Data":"3b164de25e6c02e7399d8d146e83552347def8c6a7412578b660016ecc2576da"}
Jul 27 17:56:38.469635 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:38.469603    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="548b5a37-e500-45c6-b82f-13f2a74dbff8" path="/var/lib/kubelet/pods/548b5a37-e500-45c6-b82f-13f2a74dbff8/volumes"
Jul 27 17:56:38.470075 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:38.470062    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9775f83a-fe61-493f-bb43-38a1229ba941" path="/var/lib/kubelet/pods/9775f83a-fe61-493f-bb43-38a1229ba941/volumes"
Jul 27 17:56:41.315255 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:41.315212    2573 generic.go:356] "Generic (PLEG): container finished" podID="1f17000c-4324-4ca5-9f04-f41b2198fe50" containerID="e2a9e59fe8e88d4e64caee7c6eab9b5812ea44b8e1ca2523e5d359d1a22a45d0" exitCode=0
Jul 27 17:56:41.315603 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:41.315288    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-cq4vs" event={"ID":"1f17000c-4324-4ca5-9f04-f41b2198fe50","Type":"ContainerDied","Data":"e2a9e59fe8e88d4e64caee7c6eab9b5812ea44b8e1ca2523e5d359d1a22a45d0"}
Jul 27 17:56:42.322257 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:42.322225    2573 generic.go:356] "Generic (PLEG): container finished" podID="aab97251-fd04-4e56-85d0-f8b76ddd901d" containerID="ae05510e18c350f3eef6c85ffa017eb63601b702230d7082b820c2c3f31eccb2" exitCode=0
Jul 27 17:56:42.322728 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:42.322313    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-f492w" event={"ID":"aab97251-fd04-4e56-85d0-f8b76ddd901d","Type":"ContainerDied","Data":"ae05510e18c350f3eef6c85ffa017eb63601b702230d7082b820c2c3f31eccb2"}
Jul 27 17:56:42.324466 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:42.324443    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-cq4vs" event={"ID":"1f17000c-4324-4ca5-9f04-f41b2198fe50","Type":"ContainerStarted","Data":"2f0a14e3f2cf42ca346b994a400750e867de98e0a6bec30e893f72de2a260af9"}
Jul 27 17:56:42.324557 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:42.324493    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-cq4vs" event={"ID":"1f17000c-4324-4ca5-9f04-f41b2198fe50","Type":"ContainerStarted","Data":"f555a8162ffa4e7716908952553875e65f08234d585d15339cf35db53b3dcb9a"}
Jul 27 17:56:42.324699 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:42.324683    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-cq4vs"
Jul 27 17:56:42.364851 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:42.364811    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-cq4vs" podStartSLOduration=2.641162275 podStartE2EDuration="6.36479513s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.723632855s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:56:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:56:42.363124764 +0000 UTC m=+1744.506526814" watchObservedRunningTime="2026-07-27 17:56:42.36479513 +0000 UTC m=+1744.508197213"
Jul 27 17:56:43.331283 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:43.331228    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-f492w" event={"ID":"aab97251-fd04-4e56-85d0-f8b76ddd901d","Type":"ContainerStarted","Data":"ba0cb6cdd9a66f49ed1417bf17858a2ed4afed53168e08250a4ee06ebb4de4d7"}
Jul 27 17:56:43.331669 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:43.331333    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-f492w" event={"ID":"aab97251-fd04-4e56-85d0-f8b76ddd901d","Type":"ContainerStarted","Data":"efba697fdcfb0b4ab6ca983a7729a096c7dc4899258e8862776cbab447dace18"}
Jul 27 17:56:43.331669 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:43.331578    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-cq4vs"
Jul 27 17:56:43.331813 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:43.331794    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-f492w"
Jul 27 17:56:43.331873 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:43.331841    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-f492w"
Jul 27 17:56:43.332913 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:43.332888    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-cq4vs" podUID="1f17000c-4324-4ca5-9f04-f41b2198fe50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 27 17:56:43.333030 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:43.332913    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-f492w" podUID="aab97251-fd04-4e56-85d0-f8b76ddd901d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 27 17:56:43.353106 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:43.353052    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-f492w" podStartSLOduration=3.876906936 podStartE2EDuration="7.353036867s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.476129931s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:56:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:56:43.352214162 +0000 UTC m=+1745.495616212" watchObservedRunningTime="2026-07-27 17:56:43.353036867 +0000 UTC m=+1745.496438991"
Jul 27 17:56:44.336037 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:44.335989    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-cq4vs" podUID="1f17000c-4324-4ca5-9f04-f41b2198fe50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 27 17:56:44.336426 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:44.336119    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-f492w" podUID="aab97251-fd04-4e56-85d0-f8b76ddd901d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 27 17:56:49.340583 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:49.340550    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-cq4vs"
Jul 27 17:56:49.340968 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:49.340638    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-f492w"
Jul 27 17:56:49.341073 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:49.341045    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-f492w" podUID="aab97251-fd04-4e56-85d0-f8b76ddd901d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 27 17:56:49.341125 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:49.341089    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-cq4vs" podUID="1f17000c-4324-4ca5-9f04-f41b2198fe50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 27 17:56:59.341421 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:59.341378    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-cq4vs" podUID="1f17000c-4324-4ca5-9f04-f41b2198fe50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 27 17:56:59.341898 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:56:59.341378    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-f492w" podUID="aab97251-fd04-4e56-85d0-f8b76ddd901d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 27 17:57:09.341937 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:57:09.341893    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-cq4vs" podUID="1f17000c-4324-4ca5-9f04-f41b2198fe50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 27 17:57:09.342471 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:57:09.341890    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-f492w" podUID="aab97251-fd04-4e56-85d0-f8b76ddd901d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 27 17:57:19.341353 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:57:19.341315    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-f492w" podUID="aab97251-fd04-4e56-85d0-f8b76ddd901d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 27 17:57:19.341832 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:57:19.341316    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-cq4vs" podUID="1f17000c-4324-4ca5-9f04-f41b2198fe50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 27 17:57:29.341620 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:57:29.341577    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-cq4vs" podUID="1f17000c-4324-4ca5-9f04-f41b2198fe50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 27 17:57:29.342081 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:57:29.341577    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-f492w" podUID="aab97251-fd04-4e56-85d0-f8b76ddd901d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 27 17:57:38.387302 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:57:38.387241    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vc2tz_87dac797-f8d2-4d36-999f-f8ce855dbeff/ovn-acl-logging/0.log"
Jul 27 17:57:38.389767 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:57:38.389747    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vc2tz_87dac797-f8d2-4d36-999f-f8ce855dbeff/ovn-acl-logging/0.log"
Jul 27 17:57:39.341438 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:57:39.341403    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-f492w" podUID="aab97251-fd04-4e56-85d0-f8b76ddd901d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 27 17:57:39.341638 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:57:39.341403    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-cq4vs" podUID="1f17000c-4324-4ca5-9f04-f41b2198fe50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 27 17:57:46.212380 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:57:46.212346    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:57:49.341003 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:57:49.340965    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-f492w" podUID="aab97251-fd04-4e56-85d0-f8b76ddd901d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 27 17:57:49.341397 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:57:49.340965    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-cq4vs" podUID="1f17000c-4324-4ca5-9f04-f41b2198fe50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 27 17:57:50.464184 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:57:50.464139    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-cq4vs" podUID="1f17000c-4324-4ca5-9f04-f41b2198fe50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 27 17:57:59.342150 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:57:59.342118    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-f492w"
Jul 27 17:58:00.469066 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:58:00.469037    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-cq4vs"
Jul 27 17:58:06.873276 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:58:06.873227    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-f492w"]
Jul 27 17:58:06.873681 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:58:06.873527    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-f492w" podUID="aab97251-fd04-4e56-85d0-f8b76ddd901d" containerName="kserve-container" containerID="cri-o://efba697fdcfb0b4ab6ca983a7729a096c7dc4899258e8862776cbab447dace18" gracePeriod=30
Jul 27 17:58:06.873681 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:58:06.873547    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-f492w" podUID="aab97251-fd04-4e56-85d0-f8b76ddd901d" containerName="kube-rbac-proxy" containerID="cri-o://ba0cb6cdd9a66f49ed1417bf17858a2ed4afed53168e08250a4ee06ebb4de4d7" gracePeriod=30
Jul 27 17:58:06.889373 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:58:06.889344    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:58:07.695066 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:58:07.695032    2573 generic.go:356] "Generic (PLEG): container finished" podID="aab97251-fd04-4e56-85d0-f8b76ddd901d" containerID="ba0cb6cdd9a66f49ed1417bf17858a2ed4afed53168e08250a4ee06ebb4de4d7" exitCode=2
Jul 27 17:58:07.695243 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:58:07.695109    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-f492w" event={"ID":"aab97251-fd04-4e56-85d0-f8b76ddd901d","Type":"ContainerDied","Data":"ba0cb6cdd9a66f49ed1417bf17858a2ed4afed53168e08250a4ee06ebb4de4d7"}
Jul 27 17:58:09.336401 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:58:09.336356    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-f492w" podUID="aab97251-fd04-4e56-85d0-f8b76ddd901d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Jul 27 17:58:09.341824 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:58:09.341795    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-f492w" podUID="aab97251-fd04-4e56-85d0-f8b76ddd901d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 27 17:58:11.620330 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:58:11.620299    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-f492w"
Jul 27 17:58:11.649665 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:58:11.649588    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aab97251-fd04-4e56-85d0-f8b76ddd901d-kube-rbac-proxy-sar-config\") pod \"aab97251-fd04-4e56-85d0-f8b76ddd901d\" (UID: \"aab97251-fd04-4e56-85d0-f8b76ddd901d\") "
Jul 27 17:58:11.649665 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:58:11.649632    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aab97251-fd04-4e56-85d0-f8b76ddd901d-cabundle-cert\") pod \"aab97251-fd04-4e56-85d0-f8b76ddd901d\" (UID: \"aab97251-fd04-4e56-85d0-f8b76ddd901d\") "
Jul 27 17:58:11.649900 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:58:11.649682    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aab97251-fd04-4e56-85d0-f8b76ddd901d-kserve-provision-location\") pod \"aab97251-fd04-4e56-85d0-f8b76ddd901d\" (UID: \"aab97251-fd04-4e56-85d0-f8b76ddd901d\") "
Jul 27 17:58:11.649900 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:58:11.649727    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fdqhv\" (UniqueName: \"kubernetes.io/projected/aab97251-fd04-4e56-85d0-f8b76ddd901d-kube-api-access-fdqhv\") pod \"aab97251-fd04-4e56-85d0-f8b76ddd901d\" (UID: \"aab97251-fd04-4e56-85d0-f8b76ddd901d\") "
Jul 27 17:58:11.649900 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:58:11.649788    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aab97251-fd04-4e56-85d0-f8b76ddd901d-proxy-tls\") pod \"aab97251-fd04-4e56-85d0-f8b76ddd901d\" (UID: \"aab97251-fd04-4e56-85d0-f8b76ddd901d\") "
Jul 27 17:58:11.650064 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:58:11.650035    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aab97251-fd04-4e56-85d0-f8b76ddd901d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "aab97251-fd04-4e56-85d0-f8b76ddd901d" (UID: "aab97251-fd04-4e56-85d0-f8b76ddd901d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:58:11.650064 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:58:11.650040    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aab97251-fd04-4e56-85d0-f8b76ddd901d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "aab97251-fd04-4e56-85d0-f8b76ddd901d" (UID: "aab97251-fd04-4e56-85d0-f8b76ddd901d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:58:11.650155 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:58:11.650053    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aab97251-fd04-4e56-85d0-f8b76ddd901d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "aab97251-fd04-4e56-85d0-f8b76ddd901d" (UID: "aab97251-fd04-4e56-85d0-f8b76ddd901d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:58:11.651902 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:58:11.651876    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aab97251-fd04-4e56-85d0-f8b76ddd901d-kube-api-access-fdqhv" (OuterVolumeSpecName: "kube-api-access-fdqhv") pod "aab97251-fd04-4e56-85d0-f8b76ddd901d" (UID: "aab97251-fd04-4e56-85d0-f8b76ddd901d"). InnerVolumeSpecName "kube-api-access-fdqhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:58:11.651999 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:58:11.651915    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aab97251-fd04-4e56-85d0-f8b76ddd901d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "aab97251-fd04-4e56-85d0-f8b76ddd901d" (UID: "aab97251-fd04-4e56-85d0-f8b76ddd901d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:58:11.714715 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:58:11.714681    2573 generic.go:356] "Generic (PLEG): container finished" podID="aab97251-fd04-4e56-85d0-f8b76ddd901d" containerID="efba697fdcfb0b4ab6ca983a7729a096c7dc4899258e8862776cbab447dace18" exitCode=0
Jul 27 17:58:11.714891 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:58:11.714748    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-f492w"
Jul 27 17:58:11.714891 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:58:11.714767    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-f492w" event={"ID":"aab97251-fd04-4e56-85d0-f8b76ddd901d","Type":"ContainerDied","Data":"efba697fdcfb0b4ab6ca983a7729a096c7dc4899258e8862776cbab447dace18"}
Jul 27 17:58:11.714891 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:58:11.714841    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-f492w" event={"ID":"aab97251-fd04-4e56-85d0-f8b76ddd901d","Type":"ContainerDied","Data":"7fa5d5c1e7d6b6315d17e7450782f08b306cc872bb1f6d1ef8b82518a8d149b2"}
Jul 27 17:58:11.714891 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:58:11.714883    2573 scope.go:117] "RemoveContainer" containerID="ba0cb6cdd9a66f49ed1417bf17858a2ed4afed53168e08250a4ee06ebb4de4d7"
Jul 27 17:58:11.723656 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:58:11.723635    2573 scope.go:117] "RemoveContainer" containerID="efba697fdcfb0b4ab6ca983a7729a096c7dc4899258e8862776cbab447dace18"
Jul 27 17:58:11.731062 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:58:11.731045    2573 scope.go:117] "RemoveContainer" containerID="ae05510e18c350f3eef6c85ffa017eb63601b702230d7082b820c2c3f31eccb2"
Jul 27 17:58:11.738963 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:58:11.738936    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-f492w"]
Jul 27 17:58:11.740101 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:58:11.740080    2573 scope.go:117] "RemoveContainer" containerID="ba0cb6cdd9a66f49ed1417bf17858a2ed4afed53168e08250a4ee06ebb4de4d7"
Jul 27 17:58:11.740417 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:58:11.740399    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba0cb6cdd9a66f49ed1417bf17858a2ed4afed53168e08250a4ee06ebb4de4d7\": container with ID starting with ba0cb6cdd9a66f49ed1417bf17858a2ed4afed53168e08250a4ee06ebb4de4d7 not found: ID does not exist" containerID="ba0cb6cdd9a66f49ed1417bf17858a2ed4afed53168e08250a4ee06ebb4de4d7"
Jul 27 17:58:11.740494 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:58:11.740423    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba0cb6cdd9a66f49ed1417bf17858a2ed4afed53168e08250a4ee06ebb4de4d7"} err="failed to get container status \"ba0cb6cdd9a66f49ed1417bf17858a2ed4afed53168e08250a4ee06ebb4de4d7\": rpc error: code = NotFound desc = could not find container \"ba0cb6cdd9a66f49ed1417bf17858a2ed4afed53168e08250a4ee06ebb4de4d7\": container with ID starting with ba0cb6cdd9a66f49ed1417bf17858a2ed4afed53168e08250a4ee06ebb4de4d7 not found: ID does not exist"
Jul 27 17:58:11.740494 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:58:11.740441    2573 scope.go:117] "RemoveContainer" containerID="efba697fdcfb0b4ab6ca983a7729a096c7dc4899258e8862776cbab447dace18"
Jul 27 17:58:11.740708 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:58:11.740690    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efba697fdcfb0b4ab6ca983a7729a096c7dc4899258e8862776cbab447dace18\": container with ID starting with efba697fdcfb0b4ab6ca983a7729a096c7dc4899258e8862776cbab447dace18 not found: ID does not exist" containerID="efba697fdcfb0b4ab6ca983a7729a096c7dc4899258e8862776cbab447dace18"
Jul 27 17:58:11.740753 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:58:11.740714    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efba697fdcfb0b4ab6ca983a7729a096c7dc4899258e8862776cbab447dace18"} err="failed to get container status \"efba697fdcfb0b4ab6ca983a7729a096c7dc4899258e8862776cbab447dace18\": rpc error: code = NotFound desc = could not find container \"efba697fdcfb0b4ab6ca983a7729a096c7dc4899258e8862776cbab447dace18\": container with ID starting with efba697fdcfb0b4ab6ca983a7729a096c7dc4899258e8862776cbab447dace18 not found: ID does not exist"
Jul 27 17:58:11.740753 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:58:11.740730    2573 scope.go:117] "RemoveContainer" containerID="ae05510e18c350f3eef6c85ffa017eb63601b702230d7082b820c2c3f31eccb2"
Jul 27 17:58:11.740986 ip-10-0-140-0 kubenswrapper[2573]: E0727 17:58:11.740966    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae05510e18c350f3eef6c85ffa017eb63601b702230d7082b820c2c3f31eccb2\": container with ID starting with ae05510e18c350f3eef6c85ffa017eb63601b702230d7082b820c2c3f31eccb2 not found: ID does not exist" containerID="ae05510e18c350f3eef6c85ffa017eb63601b702230d7082b820c2c3f31eccb2"
Jul 27 17:58:11.741047 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:58:11.740993    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae05510e18c350f3eef6c85ffa017eb63601b702230d7082b820c2c3f31eccb2"} err="failed to get container status \"ae05510e18c350f3eef6c85ffa017eb63601b702230d7082b820c2c3f31eccb2\": rpc error: code = NotFound desc = could not find container \"ae05510e18c350f3eef6c85ffa017eb63601b702230d7082b820c2c3f31eccb2\": container with ID starting with ae05510e18c350f3eef6c85ffa017eb63601b702230d7082b820c2c3f31eccb2 not found: ID does not exist"
Jul 27 17:58:11.743996 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:58:11.743975    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-f492w"]
Jul 27 17:58:11.750428 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:58:11.750410    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aab97251-fd04-4e56-85d0-f8b76ddd901d-proxy-tls\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:58:11.750516 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:58:11.750434    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aab97251-fd04-4e56-85d0-f8b76ddd901d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:58:11.750516 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:58:11.750448    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aab97251-fd04-4e56-85d0-f8b76ddd901d-cabundle-cert\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:58:11.750516 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:58:11.750462    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aab97251-fd04-4e56-85d0-f8b76ddd901d-kserve-provision-location\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:58:11.750516 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:58:11.750475    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdqhv\" (UniqueName: \"kubernetes.io/projected/aab97251-fd04-4e56-85d0-f8b76ddd901d-kube-api-access-fdqhv\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 17:58:12.470063 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:58:12.470030    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aab97251-fd04-4e56-85d0-f8b76ddd901d" path="/var/lib/kubelet/pods/aab97251-fd04-4e56-85d0-f8b76ddd901d/volumes"
Jul 27 17:59:57.153810 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:59:57.153769    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-cq4vs"]
Jul 27 17:59:57.154415 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:59:57.154078    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-cq4vs" podUID="1f17000c-4324-4ca5-9f04-f41b2198fe50" containerName="kserve-container" containerID="cri-o://f555a8162ffa4e7716908952553875e65f08234d585d15339cf35db53b3dcb9a" gracePeriod=30
Jul 27 17:59:57.154415 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:59:57.154146    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-cq4vs" podUID="1f17000c-4324-4ca5-9f04-f41b2198fe50" containerName="kube-rbac-proxy" containerID="cri-o://2f0a14e3f2cf42ca346b994a400750e867de98e0a6bec30e893f72de2a260af9" gracePeriod=30
Jul 27 17:59:57.206319 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:59:57.206281    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 17:59:58.178266 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:59:58.178222    2573 generic.go:356] "Generic (PLEG): container finished" podID="1f17000c-4324-4ca5-9f04-f41b2198fe50" containerID="2f0a14e3f2cf42ca346b994a400750e867de98e0a6bec30e893f72de2a260af9" exitCode=2
Jul 27 17:59:58.178655 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:59:58.178285    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-cq4vs" event={"ID":"1f17000c-4324-4ca5-9f04-f41b2198fe50","Type":"ContainerDied","Data":"2f0a14e3f2cf42ca346b994a400750e867de98e0a6bec30e893f72de2a260af9"}
Jul 27 17:59:59.336822 ip-10-0-140-0 kubenswrapper[2573]: I0727 17:59:59.336780    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-cq4vs" podUID="1f17000c-4324-4ca5-9f04-f41b2198fe50" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused"
Jul 27 18:00:00.464980 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:00.464933    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-cq4vs" podUID="1f17000c-4324-4ca5-9f04-f41b2198fe50" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 27 18:00:01.807597 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:01.807574    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-cq4vs"
Jul 27 18:00:01.928919 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:01.928887    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1f17000c-4324-4ca5-9f04-f41b2198fe50-cabundle-cert\") pod \"1f17000c-4324-4ca5-9f04-f41b2198fe50\" (UID: \"1f17000c-4324-4ca5-9f04-f41b2198fe50\") "
Jul 27 18:00:01.929082 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:01.928927    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7rkd\" (UniqueName: \"kubernetes.io/projected/1f17000c-4324-4ca5-9f04-f41b2198fe50-kube-api-access-q7rkd\") pod \"1f17000c-4324-4ca5-9f04-f41b2198fe50\" (UID: \"1f17000c-4324-4ca5-9f04-f41b2198fe50\") "
Jul 27 18:00:01.929082 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:01.928954    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f17000c-4324-4ca5-9f04-f41b2198fe50-proxy-tls\") pod \"1f17000c-4324-4ca5-9f04-f41b2198fe50\" (UID: \"1f17000c-4324-4ca5-9f04-f41b2198fe50\") "
Jul 27 18:00:01.929082 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:01.929017    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f17000c-4324-4ca5-9f04-f41b2198fe50-kserve-provision-location\") pod \"1f17000c-4324-4ca5-9f04-f41b2198fe50\" (UID: \"1f17000c-4324-4ca5-9f04-f41b2198fe50\") "
Jul 27 18:00:01.929082 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:01.929033    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f17000c-4324-4ca5-9f04-f41b2198fe50-kube-rbac-proxy-sar-config\") pod \"1f17000c-4324-4ca5-9f04-f41b2198fe50\" (UID: \"1f17000c-4324-4ca5-9f04-f41b2198fe50\") "
Jul 27 18:00:01.929394 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:01.929359    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f17000c-4324-4ca5-9f04-f41b2198fe50-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1f17000c-4324-4ca5-9f04-f41b2198fe50" (UID: "1f17000c-4324-4ca5-9f04-f41b2198fe50"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:00:01.929454 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:01.929376    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f17000c-4324-4ca5-9f04-f41b2198fe50-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1f17000c-4324-4ca5-9f04-f41b2198fe50" (UID: "1f17000c-4324-4ca5-9f04-f41b2198fe50"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:00:01.929454 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:01.929445    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f17000c-4324-4ca5-9f04-f41b2198fe50-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1f17000c-4324-4ca5-9f04-f41b2198fe50" (UID: "1f17000c-4324-4ca5-9f04-f41b2198fe50"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:00:01.931154 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:01.931130    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f17000c-4324-4ca5-9f04-f41b2198fe50-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1f17000c-4324-4ca5-9f04-f41b2198fe50" (UID: "1f17000c-4324-4ca5-9f04-f41b2198fe50"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:00:01.931241 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:01.931171    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f17000c-4324-4ca5-9f04-f41b2198fe50-kube-api-access-q7rkd" (OuterVolumeSpecName: "kube-api-access-q7rkd") pod "1f17000c-4324-4ca5-9f04-f41b2198fe50" (UID: "1f17000c-4324-4ca5-9f04-f41b2198fe50"). InnerVolumeSpecName "kube-api-access-q7rkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:00:02.029754 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.029707    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1f17000c-4324-4ca5-9f04-f41b2198fe50-cabundle-cert\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:00:02.029754 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.029747    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7rkd\" (UniqueName: \"kubernetes.io/projected/1f17000c-4324-4ca5-9f04-f41b2198fe50-kube-api-access-q7rkd\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:00:02.029754 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.029759    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f17000c-4324-4ca5-9f04-f41b2198fe50-proxy-tls\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:00:02.029754 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.029768    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1f17000c-4324-4ca5-9f04-f41b2198fe50-kserve-provision-location\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:00:02.029754 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.029777    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f17000c-4324-4ca5-9f04-f41b2198fe50-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:00:02.197817 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.197730    2573 generic.go:356] "Generic (PLEG): container finished" podID="1f17000c-4324-4ca5-9f04-f41b2198fe50" containerID="f555a8162ffa4e7716908952553875e65f08234d585d15339cf35db53b3dcb9a" exitCode=0
Jul 27 18:00:02.197817 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.197809    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-cq4vs"
Jul 27 18:00:02.198043 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.197805    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-cq4vs" event={"ID":"1f17000c-4324-4ca5-9f04-f41b2198fe50","Type":"ContainerDied","Data":"f555a8162ffa4e7716908952553875e65f08234d585d15339cf35db53b3dcb9a"}
Jul 27 18:00:02.198043 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.197964    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-cq4vs" event={"ID":"1f17000c-4324-4ca5-9f04-f41b2198fe50","Type":"ContainerDied","Data":"3b164de25e6c02e7399d8d146e83552347def8c6a7412578b660016ecc2576da"}
Jul 27 18:00:02.198043 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.198039    2573 scope.go:117] "RemoveContainer" containerID="2f0a14e3f2cf42ca346b994a400750e867de98e0a6bec30e893f72de2a260af9"
Jul 27 18:00:02.206494 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.206477    2573 scope.go:117] "RemoveContainer" containerID="f555a8162ffa4e7716908952553875e65f08234d585d15339cf35db53b3dcb9a"
Jul 27 18:00:02.214210 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.214189    2573 scope.go:117] "RemoveContainer" containerID="e2a9e59fe8e88d4e64caee7c6eab9b5812ea44b8e1ca2523e5d359d1a22a45d0"
Jul 27 18:00:02.222652 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.222618    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-cq4vs"]
Jul 27 18:00:02.223586 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.223566    2573 scope.go:117] "RemoveContainer" containerID="2f0a14e3f2cf42ca346b994a400750e867de98e0a6bec30e893f72de2a260af9"
Jul 27 18:00:02.223844 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:00:02.223825    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f0a14e3f2cf42ca346b994a400750e867de98e0a6bec30e893f72de2a260af9\": container with ID starting with 2f0a14e3f2cf42ca346b994a400750e867de98e0a6bec30e893f72de2a260af9 not found: ID does not exist" containerID="2f0a14e3f2cf42ca346b994a400750e867de98e0a6bec30e893f72de2a260af9"
Jul 27 18:00:02.223925 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.223855    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f0a14e3f2cf42ca346b994a400750e867de98e0a6bec30e893f72de2a260af9"} err="failed to get container status \"2f0a14e3f2cf42ca346b994a400750e867de98e0a6bec30e893f72de2a260af9\": rpc error: code = NotFound desc = could not find container \"2f0a14e3f2cf42ca346b994a400750e867de98e0a6bec30e893f72de2a260af9\": container with ID starting with 2f0a14e3f2cf42ca346b994a400750e867de98e0a6bec30e893f72de2a260af9 not found: ID does not exist"
Jul 27 18:00:02.223925 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.223879    2573 scope.go:117] "RemoveContainer" containerID="f555a8162ffa4e7716908952553875e65f08234d585d15339cf35db53b3dcb9a"
Jul 27 18:00:02.224168 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:00:02.224151    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f555a8162ffa4e7716908952553875e65f08234d585d15339cf35db53b3dcb9a\": container with ID starting with f555a8162ffa4e7716908952553875e65f08234d585d15339cf35db53b3dcb9a not found: ID does not exist" containerID="f555a8162ffa4e7716908952553875e65f08234d585d15339cf35db53b3dcb9a"
Jul 27 18:00:02.224243 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.224176    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f555a8162ffa4e7716908952553875e65f08234d585d15339cf35db53b3dcb9a"} err="failed to get container status \"f555a8162ffa4e7716908952553875e65f08234d585d15339cf35db53b3dcb9a\": rpc error: code = NotFound desc = could not find container \"f555a8162ffa4e7716908952553875e65f08234d585d15339cf35db53b3dcb9a\": container with ID starting with f555a8162ffa4e7716908952553875e65f08234d585d15339cf35db53b3dcb9a not found: ID does not exist"
Jul 27 18:00:02.224243 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.224195    2573 scope.go:117] "RemoveContainer" containerID="e2a9e59fe8e88d4e64caee7c6eab9b5812ea44b8e1ca2523e5d359d1a22a45d0"
Jul 27 18:00:02.224496 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:00:02.224472    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2a9e59fe8e88d4e64caee7c6eab9b5812ea44b8e1ca2523e5d359d1a22a45d0\": container with ID starting with e2a9e59fe8e88d4e64caee7c6eab9b5812ea44b8e1ca2523e5d359d1a22a45d0 not found: ID does not exist" containerID="e2a9e59fe8e88d4e64caee7c6eab9b5812ea44b8e1ca2523e5d359d1a22a45d0"
Jul 27 18:00:02.224537 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.224504    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2a9e59fe8e88d4e64caee7c6eab9b5812ea44b8e1ca2523e5d359d1a22a45d0"} err="failed to get container status \"e2a9e59fe8e88d4e64caee7c6eab9b5812ea44b8e1ca2523e5d359d1a22a45d0\": rpc error: code = NotFound desc = could not find container \"e2a9e59fe8e88d4e64caee7c6eab9b5812ea44b8e1ca2523e5d359d1a22a45d0\": container with ID starting with e2a9e59fe8e88d4e64caee7c6eab9b5812ea44b8e1ca2523e5d359d1a22a45d0 not found: ID does not exist"
Jul 27 18:00:02.227998 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.227972    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-cq4vs"]
Jul 27 18:00:02.286335 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.286303    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-424c9"]
Jul 27 18:00:02.286615 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.286603    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aab97251-fd04-4e56-85d0-f8b76ddd901d" containerName="storage-initializer"
Jul 27 18:00:02.286682 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.286618    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="aab97251-fd04-4e56-85d0-f8b76ddd901d" containerName="storage-initializer"
Jul 27 18:00:02.286682 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.286645    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aab97251-fd04-4e56-85d0-f8b76ddd901d" containerName="kube-rbac-proxy"
Jul 27 18:00:02.286682 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.286654    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="aab97251-fd04-4e56-85d0-f8b76ddd901d" containerName="kube-rbac-proxy"
Jul 27 18:00:02.286682 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.286666    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aab97251-fd04-4e56-85d0-f8b76ddd901d" containerName="kserve-container"
Jul 27 18:00:02.286682 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.286676    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="aab97251-fd04-4e56-85d0-f8b76ddd901d" containerName="kserve-container"
Jul 27 18:00:02.286834 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.286689    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f17000c-4324-4ca5-9f04-f41b2198fe50" containerName="kserve-container"
Jul 27 18:00:02.286834 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.286696    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f17000c-4324-4ca5-9f04-f41b2198fe50" containerName="kserve-container"
Jul 27 18:00:02.286834 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.286705    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f17000c-4324-4ca5-9f04-f41b2198fe50" containerName="kube-rbac-proxy"
Jul 27 18:00:02.286834 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.286710    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f17000c-4324-4ca5-9f04-f41b2198fe50" containerName="kube-rbac-proxy"
Jul 27 18:00:02.286834 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.286719    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f17000c-4324-4ca5-9f04-f41b2198fe50" containerName="storage-initializer"
Jul 27 18:00:02.286834 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.286724    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f17000c-4324-4ca5-9f04-f41b2198fe50" containerName="storage-initializer"
Jul 27 18:00:02.286834 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.286779    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f17000c-4324-4ca5-9f04-f41b2198fe50" containerName="kserve-container"
Jul 27 18:00:02.286834 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.286788    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f17000c-4324-4ca5-9f04-f41b2198fe50" containerName="kube-rbac-proxy"
Jul 27 18:00:02.286834 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.286795    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="aab97251-fd04-4e56-85d0-f8b76ddd901d" containerName="kube-rbac-proxy"
Jul 27 18:00:02.286834 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.286802    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="aab97251-fd04-4e56-85d0-f8b76ddd901d" containerName="kserve-container"
Jul 27 18:00:02.290943 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.290926    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-424c9"
Jul 27 18:00:02.293815 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.293790    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Jul 27 18:00:02.294045 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.294028    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 18:00:02.294112 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.294068    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 18:00:02.294175 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.294064    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Jul 27 18:00:02.294311 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.294296    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 18:00:02.294417 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.294398    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z6ld7\""
Jul 27 18:00:02.294484 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.294420    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 18:00:02.298592 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.298570    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xjw87"]
Jul 27 18:00:02.302111 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.302091    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-424c9"]
Jul 27 18:00:02.302222 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.302205    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xjw87"
Jul 27 18:00:02.304995 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.304975    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Jul 27 18:00:02.312117 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.312099    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xjw87"]
Jul 27 18:00:02.332938 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.332901    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fdd8df10-64ad-4c9c-bad7-3782e38c6e68-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-424c9\" (UID: \"fdd8df10-64ad-4c9c-bad7-3782e38c6e68\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-424c9"
Jul 27 18:00:02.333105 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.332943    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dk5l\" (UniqueName: \"kubernetes.io/projected/fdd8df10-64ad-4c9c-bad7-3782e38c6e68-kube-api-access-9dk5l\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-424c9\" (UID: \"fdd8df10-64ad-4c9c-bad7-3782e38c6e68\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-424c9"
Jul 27 18:00:02.333105 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.332979    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fdd8df10-64ad-4c9c-bad7-3782e38c6e68-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-424c9\" (UID: \"fdd8df10-64ad-4c9c-bad7-3782e38c6e68\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-424c9"
Jul 27 18:00:02.333105 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.333016    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fdd8df10-64ad-4c9c-bad7-3782e38c6e68-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-424c9\" (UID: \"fdd8df10-64ad-4c9c-bad7-3782e38c6e68\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-424c9"
Jul 27 18:00:02.333105 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.333069    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdd8df10-64ad-4c9c-bad7-3782e38c6e68-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-424c9\" (UID: \"fdd8df10-64ad-4c9c-bad7-3782e38c6e68\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-424c9"
Jul 27 18:00:02.341231 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.341192    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:00:02.400830 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.400797    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:00:02.433521 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.433493    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fdd8df10-64ad-4c9c-bad7-3782e38c6e68-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-424c9\" (UID: \"fdd8df10-64ad-4c9c-bad7-3782e38c6e68\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-424c9"
Jul 27 18:00:02.433622 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.433530    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14096185-74a8-4e25-a035-7fd0bb5e44c9-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-xjw87\" (UID: \"14096185-74a8-4e25-a035-7fd0bb5e44c9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xjw87"
Jul 27 18:00:02.433622 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.433552    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dk5l\" (UniqueName: \"kubernetes.io/projected/fdd8df10-64ad-4c9c-bad7-3782e38c6e68-kube-api-access-9dk5l\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-424c9\" (UID: \"fdd8df10-64ad-4c9c-bad7-3782e38c6e68\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-424c9"
Jul 27 18:00:02.433622 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.433575    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fdd8df10-64ad-4c9c-bad7-3782e38c6e68-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-424c9\" (UID: \"fdd8df10-64ad-4c9c-bad7-3782e38c6e68\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-424c9"
Jul 27 18:00:02.433622 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.433592    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/14096185-74a8-4e25-a035-7fd0bb5e44c9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-xjw87\" (UID: \"14096185-74a8-4e25-a035-7fd0bb5e44c9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xjw87"
Jul 27 18:00:02.433829 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.433722    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fdd8df10-64ad-4c9c-bad7-3782e38c6e68-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-424c9\" (UID: \"fdd8df10-64ad-4c9c-bad7-3782e38c6e68\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-424c9"
Jul 27 18:00:02.433829 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.433775    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14096185-74a8-4e25-a035-7fd0bb5e44c9-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-xjw87\" (UID: \"14096185-74a8-4e25-a035-7fd0bb5e44c9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xjw87"
Jul 27 18:00:02.433946 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.433827    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdd8df10-64ad-4c9c-bad7-3782e38c6e68-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-424c9\" (UID: \"fdd8df10-64ad-4c9c-bad7-3782e38c6e68\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-424c9"
Jul 27 18:00:02.433946 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.433874    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74sk7\" (UniqueName: \"kubernetes.io/projected/14096185-74a8-4e25-a035-7fd0bb5e44c9-kube-api-access-74sk7\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-xjw87\" (UID: \"14096185-74a8-4e25-a035-7fd0bb5e44c9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xjw87"
Jul 27 18:00:02.433946 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.433903    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14096185-74a8-4e25-a035-7fd0bb5e44c9-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-xjw87\" (UID: \"14096185-74a8-4e25-a035-7fd0bb5e44c9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xjw87"
Jul 27 18:00:02.434170 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.434151    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdd8df10-64ad-4c9c-bad7-3782e38c6e68-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-424c9\" (UID: \"fdd8df10-64ad-4c9c-bad7-3782e38c6e68\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-424c9"
Jul 27 18:00:02.434213 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.434190    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fdd8df10-64ad-4c9c-bad7-3782e38c6e68-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-424c9\" (UID: \"fdd8df10-64ad-4c9c-bad7-3782e38c6e68\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-424c9"
Jul 27 18:00:02.434303 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.434287    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fdd8df10-64ad-4c9c-bad7-3782e38c6e68-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-424c9\" (UID: \"fdd8df10-64ad-4c9c-bad7-3782e38c6e68\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-424c9"
Jul 27 18:00:02.436208 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.436192    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fdd8df10-64ad-4c9c-bad7-3782e38c6e68-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-424c9\" (UID: \"fdd8df10-64ad-4c9c-bad7-3782e38c6e68\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-424c9"
Jul 27 18:00:02.441771 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.441750    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dk5l\" (UniqueName: \"kubernetes.io/projected/fdd8df10-64ad-4c9c-bad7-3782e38c6e68-kube-api-access-9dk5l\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-424c9\" (UID: \"fdd8df10-64ad-4c9c-bad7-3782e38c6e68\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-424c9"
Jul 27 18:00:02.468921 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.468861    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f17000c-4324-4ca5-9f04-f41b2198fe50" path="/var/lib/kubelet/pods/1f17000c-4324-4ca5-9f04-f41b2198fe50/volumes"
Jul 27 18:00:02.517935 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.517901    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:00:02.535148 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.535121    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14096185-74a8-4e25-a035-7fd0bb5e44c9-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-xjw87\" (UID: \"14096185-74a8-4e25-a035-7fd0bb5e44c9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xjw87"
Jul 27 18:00:02.535310 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.535170    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74sk7\" (UniqueName: \"kubernetes.io/projected/14096185-74a8-4e25-a035-7fd0bb5e44c9-kube-api-access-74sk7\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-xjw87\" (UID: \"14096185-74a8-4e25-a035-7fd0bb5e44c9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xjw87"
Jul 27 18:00:02.535310 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.535191    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14096185-74a8-4e25-a035-7fd0bb5e44c9-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-xjw87\" (UID: \"14096185-74a8-4e25-a035-7fd0bb5e44c9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xjw87"
Jul 27 18:00:02.535310 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.535237    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14096185-74a8-4e25-a035-7fd0bb5e44c9-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-xjw87\" (UID: \"14096185-74a8-4e25-a035-7fd0bb5e44c9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xjw87"
Jul 27 18:00:02.535310 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.535301    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/14096185-74a8-4e25-a035-7fd0bb5e44c9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-xjw87\" (UID: \"14096185-74a8-4e25-a035-7fd0bb5e44c9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xjw87"
Jul 27 18:00:02.535591 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.535566    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14096185-74a8-4e25-a035-7fd0bb5e44c9-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-xjw87\" (UID: \"14096185-74a8-4e25-a035-7fd0bb5e44c9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xjw87"
Jul 27 18:00:02.535849 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.535830    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14096185-74a8-4e25-a035-7fd0bb5e44c9-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-xjw87\" (UID: \"14096185-74a8-4e25-a035-7fd0bb5e44c9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xjw87"
Jul 27 18:00:02.535911 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.535840    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/14096185-74a8-4e25-a035-7fd0bb5e44c9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-xjw87\" (UID: \"14096185-74a8-4e25-a035-7fd0bb5e44c9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xjw87"
Jul 27 18:00:02.537756 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.537733    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14096185-74a8-4e25-a035-7fd0bb5e44c9-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-xjw87\" (UID: \"14096185-74a8-4e25-a035-7fd0bb5e44c9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xjw87"
Jul 27 18:00:02.544326 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.544305    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74sk7\" (UniqueName: \"kubernetes.io/projected/14096185-74a8-4e25-a035-7fd0bb5e44c9-kube-api-access-74sk7\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-xjw87\" (UID: \"14096185-74a8-4e25-a035-7fd0bb5e44c9\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xjw87"
Jul 27 18:00:02.601304 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.601277    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-424c9"
Jul 27 18:00:02.612321 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.612274    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xjw87"
Jul 27 18:00:02.744212 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.744182    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-424c9"]
Jul 27 18:00:02.745872 ip-10-0-140-0 kubenswrapper[2573]: W0727 18:00:02.745843    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdd8df10_64ad_4c9c_bad7_3782e38c6e68.slice/crio-3fe537bc017d7c975c4225573b0714055c62d0e43467722d5fdce9411841c501 WatchSource:0}: Error finding container 3fe537bc017d7c975c4225573b0714055c62d0e43467722d5fdce9411841c501: Status 404 returned error can't find the container with id 3fe537bc017d7c975c4225573b0714055c62d0e43467722d5fdce9411841c501
Jul 27 18:00:02.767054 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:02.767030    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xjw87"]
Jul 27 18:00:02.775421 ip-10-0-140-0 kubenswrapper[2573]: W0727 18:00:02.775392    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14096185_74a8_4e25_a035_7fd0bb5e44c9.slice/crio-2d8255a3ce03a58cf4ecaf8ecd2747abba56c5d8aeab52506135d9cd9ecc5c47 WatchSource:0}: Error finding container 2d8255a3ce03a58cf4ecaf8ecd2747abba56c5d8aeab52506135d9cd9ecc5c47: Status 404 returned error can't find the container with id 2d8255a3ce03a58cf4ecaf8ecd2747abba56c5d8aeab52506135d9cd9ecc5c47
Jul 27 18:00:03.203913 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:03.203871    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xjw87" event={"ID":"14096185-74a8-4e25-a035-7fd0bb5e44c9","Type":"ContainerStarted","Data":"9dba4ae4da2a83ad99d223e8f85277565aaceab98fa65a9ad61944f1553d37f8"}
Jul 27 18:00:03.204370 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:03.203957    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xjw87" event={"ID":"14096185-74a8-4e25-a035-7fd0bb5e44c9","Type":"ContainerStarted","Data":"2d8255a3ce03a58cf4ecaf8ecd2747abba56c5d8aeab52506135d9cd9ecc5c47"}
Jul 27 18:00:03.205692 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:03.205660    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-424c9" event={"ID":"fdd8df10-64ad-4c9c-bad7-3782e38c6e68","Type":"ContainerStarted","Data":"6a6b4bb490126bf1947c8626742605d199be62ee0d0392d07cd676ed6ff07962"}
Jul 27 18:00:03.205819 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:03.205717    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-424c9" event={"ID":"fdd8df10-64ad-4c9c-bad7-3782e38c6e68","Type":"ContainerStarted","Data":"3fe537bc017d7c975c4225573b0714055c62d0e43467722d5fdce9411841c501"}
Jul 27 18:00:07.225261 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:07.225212    2573 generic.go:356] "Generic (PLEG): container finished" podID="14096185-74a8-4e25-a035-7fd0bb5e44c9" containerID="9dba4ae4da2a83ad99d223e8f85277565aaceab98fa65a9ad61944f1553d37f8" exitCode=0
Jul 27 18:00:07.225707 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:07.225286    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xjw87" event={"ID":"14096185-74a8-4e25-a035-7fd0bb5e44c9","Type":"ContainerDied","Data":"9dba4ae4da2a83ad99d223e8f85277565aaceab98fa65a9ad61944f1553d37f8"}
Jul 27 18:00:07.226898 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:07.226878    2573 generic.go:356] "Generic (PLEG): container finished" podID="fdd8df10-64ad-4c9c-bad7-3782e38c6e68" containerID="6a6b4bb490126bf1947c8626742605d199be62ee0d0392d07cd676ed6ff07962" exitCode=0
Jul 27 18:00:07.226985 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:07.226952    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-424c9" event={"ID":"fdd8df10-64ad-4c9c-bad7-3782e38c6e68","Type":"ContainerDied","Data":"6a6b4bb490126bf1947c8626742605d199be62ee0d0392d07cd676ed6ff07962"}
Jul 27 18:00:08.233709 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:08.233676    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xjw87" event={"ID":"14096185-74a8-4e25-a035-7fd0bb5e44c9","Type":"ContainerStarted","Data":"5301424f3fe6c1519d1b2cdf75adb66838dcdd6c82bd4501c4da9fe61388a3d6"}
Jul 27 18:00:08.234125 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:08.233751    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xjw87" event={"ID":"14096185-74a8-4e25-a035-7fd0bb5e44c9","Type":"ContainerStarted","Data":"f3d374e40b0ac27465f1faa3f55471b34c62b2a727bd79734e1bceacc350321a"}
Jul 27 18:00:08.234125 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:08.233988    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xjw87"
Jul 27 18:00:08.235968 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:08.235943    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-424c9" event={"ID":"fdd8df10-64ad-4c9c-bad7-3782e38c6e68","Type":"ContainerStarted","Data":"4189bee25127481114052277360bb12c93d2e608cfb1992194d8b06f01ba95c0"}
Jul 27 18:00:08.236016 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:08.236007    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-424c9" event={"ID":"fdd8df10-64ad-4c9c-bad7-3782e38c6e68","Type":"ContainerStarted","Data":"866546a58d91a9c8c1acd9b536109a3c848f390ffb3ce2f879d5af97d320fced"}
Jul 27 18:00:08.236292 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:08.236276    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-424c9"
Jul 27 18:00:08.236341 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:08.236318    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-424c9"
Jul 27 18:00:08.237773 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:08.237747    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-424c9" podUID="fdd8df10-64ad-4c9c-bad7-3782e38c6e68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 27 18:00:08.261343 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:08.261300    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xjw87" podStartSLOduration=2.3861108030000002 podStartE2EDuration="6.261286514s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.875175711s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:00:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:00:08.259277873 +0000 UTC m=+1950.402679923" watchObservedRunningTime="2026-07-27 18:00:08.261286514 +0000 UTC m=+1950.404688565"
Jul 27 18:00:08.303970 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:08.303922    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-424c9" podStartSLOduration=2.486609905 podStartE2EDuration="6.303905601s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.817295696s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:00:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:00:08.303504933 +0000 UTC m=+1950.446906984" watchObservedRunningTime="2026-07-27 18:00:08.303905601 +0000 UTC m=+1950.447307651"
Jul 27 18:00:09.242952 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:09.242923    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xjw87"
Jul 27 18:00:09.243423 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:09.242951    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-424c9" podUID="fdd8df10-64ad-4c9c-bad7-3782e38c6e68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 27 18:00:09.243986 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:09.243960    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xjw87" podUID="14096185-74a8-4e25-a035-7fd0bb5e44c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 27 18:00:10.247766 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:10.247716    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xjw87" podUID="14096185-74a8-4e25-a035-7fd0bb5e44c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 27 18:00:14.248156 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:14.248125    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-424c9"
Jul 27 18:00:14.248517 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:14.248476    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-424c9" podUID="fdd8df10-64ad-4c9c-bad7-3782e38c6e68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 27 18:00:15.251830 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:15.251804    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xjw87"
Jul 27 18:00:15.252352 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:15.252325    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xjw87" podUID="14096185-74a8-4e25-a035-7fd0bb5e44c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 27 18:00:24.249456 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:24.249416    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-424c9" podUID="fdd8df10-64ad-4c9c-bad7-3782e38c6e68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 27 18:00:25.252303 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:25.252267    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xjw87" podUID="14096185-74a8-4e25-a035-7fd0bb5e44c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 27 18:00:34.249537 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:34.249450    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-424c9" podUID="fdd8df10-64ad-4c9c-bad7-3782e38c6e68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 27 18:00:35.252277 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:35.252225    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xjw87" podUID="14096185-74a8-4e25-a035-7fd0bb5e44c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 27 18:00:44.248531 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:44.248489    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-424c9" podUID="fdd8df10-64ad-4c9c-bad7-3782e38c6e68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 27 18:00:45.253060 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:45.253022    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xjw87" podUID="14096185-74a8-4e25-a035-7fd0bb5e44c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 27 18:00:54.248989 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:54.248952    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-424c9" podUID="fdd8df10-64ad-4c9c-bad7-3782e38c6e68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 27 18:00:55.252202 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:00:55.252161    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xjw87" podUID="14096185-74a8-4e25-a035-7fd0bb5e44c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 27 18:01:04.248629 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:01:04.248591    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-424c9" podUID="fdd8df10-64ad-4c9c-bad7-3782e38c6e68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 27 18:01:05.252999 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:01:05.252961    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xjw87" podUID="14096185-74a8-4e25-a035-7fd0bb5e44c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 27 18:01:14.249264 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:01:14.249210    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-424c9" podUID="fdd8df10-64ad-4c9c-bad7-3782e38c6e68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 27 18:01:15.252265 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:01:15.252211    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xjw87" podUID="14096185-74a8-4e25-a035-7fd0bb5e44c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 27 18:01:24.249427 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:01:24.249392    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-424c9"
Jul 27 18:01:25.253321 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:01:25.253291    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xjw87"
Jul 27 18:01:32.309660 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:01:32.309620    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xjw87"]
Jul 27 18:01:32.310055 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:01:32.309898    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xjw87" podUID="14096185-74a8-4e25-a035-7fd0bb5e44c9" containerName="kserve-container" containerID="cri-o://f3d374e40b0ac27465f1faa3f55471b34c62b2a727bd79734e1bceacc350321a" gracePeriod=30
Jul 27 18:01:32.310055 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:01:32.309963    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xjw87" podUID="14096185-74a8-4e25-a035-7fd0bb5e44c9" containerName="kube-rbac-proxy" containerID="cri-o://5301424f3fe6c1519d1b2cdf75adb66838dcdd6c82bd4501c4da9fe61388a3d6" gracePeriod=30
Jul 27 18:01:32.339835 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:01:32.339811    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:01:32.615023 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:01:32.614935    2573 generic.go:356] "Generic (PLEG): container finished" podID="14096185-74a8-4e25-a035-7fd0bb5e44c9" containerID="5301424f3fe6c1519d1b2cdf75adb66838dcdd6c82bd4501c4da9fe61388a3d6" exitCode=2
Jul 27 18:01:32.615023 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:01:32.615006    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xjw87" event={"ID":"14096185-74a8-4e25-a035-7fd0bb5e44c9","Type":"ContainerDied","Data":"5301424f3fe6c1519d1b2cdf75adb66838dcdd6c82bd4501c4da9fe61388a3d6"}
Jul 27 18:01:35.248182 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:01:35.248141    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xjw87" podUID="14096185-74a8-4e25-a035-7fd0bb5e44c9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.43:8643/healthz\": dial tcp 10.133.0.43:8643: connect: connection refused"
Jul 27 18:01:35.252481 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:01:35.252453    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xjw87" podUID="14096185-74a8-4e25-a035-7fd0bb5e44c9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 27 18:01:36.959233 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:01:36.959203    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xjw87"
Jul 27 18:01:37.031486 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:01:37.031449    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-74sk7\" (UniqueName: \"kubernetes.io/projected/14096185-74a8-4e25-a035-7fd0bb5e44c9-kube-api-access-74sk7\") pod \"14096185-74a8-4e25-a035-7fd0bb5e44c9\" (UID: \"14096185-74a8-4e25-a035-7fd0bb5e44c9\") "
Jul 27 18:01:37.031652 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:01:37.031510    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/14096185-74a8-4e25-a035-7fd0bb5e44c9-kube-rbac-proxy-sar-config\") pod \"14096185-74a8-4e25-a035-7fd0bb5e44c9\" (UID: \"14096185-74a8-4e25-a035-7fd0bb5e44c9\") "
Jul 27 18:01:37.031652 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:01:37.031623    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14096185-74a8-4e25-a035-7fd0bb5e44c9-kserve-provision-location\") pod \"14096185-74a8-4e25-a035-7fd0bb5e44c9\" (UID: \"14096185-74a8-4e25-a035-7fd0bb5e44c9\") "
Jul 27 18:01:37.031745 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:01:37.031661    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14096185-74a8-4e25-a035-7fd0bb5e44c9-cabundle-cert\") pod \"14096185-74a8-4e25-a035-7fd0bb5e44c9\" (UID: \"14096185-74a8-4e25-a035-7fd0bb5e44c9\") "
Jul 27 18:01:37.031745 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:01:37.031716    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14096185-74a8-4e25-a035-7fd0bb5e44c9-proxy-tls\") pod \"14096185-74a8-4e25-a035-7fd0bb5e44c9\" (UID: \"14096185-74a8-4e25-a035-7fd0bb5e44c9\") "
Jul 27 18:01:37.031895 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:01:37.031871    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14096185-74a8-4e25-a035-7fd0bb5e44c9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "14096185-74a8-4e25-a035-7fd0bb5e44c9" (UID: "14096185-74a8-4e25-a035-7fd0bb5e44c9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:01:37.032004 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:01:37.031911    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14096185-74a8-4e25-a035-7fd0bb5e44c9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "14096185-74a8-4e25-a035-7fd0bb5e44c9" (UID: "14096185-74a8-4e25-a035-7fd0bb5e44c9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:01:37.032099 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:01:37.032074    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14096185-74a8-4e25-a035-7fd0bb5e44c9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "14096185-74a8-4e25-a035-7fd0bb5e44c9" (UID: "14096185-74a8-4e25-a035-7fd0bb5e44c9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:01:37.033695 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:01:37.033671    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14096185-74a8-4e25-a035-7fd0bb5e44c9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "14096185-74a8-4e25-a035-7fd0bb5e44c9" (UID: "14096185-74a8-4e25-a035-7fd0bb5e44c9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:01:37.033787 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:01:37.033710    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14096185-74a8-4e25-a035-7fd0bb5e44c9-kube-api-access-74sk7" (OuterVolumeSpecName: "kube-api-access-74sk7") pod "14096185-74a8-4e25-a035-7fd0bb5e44c9" (UID: "14096185-74a8-4e25-a035-7fd0bb5e44c9"). InnerVolumeSpecName "kube-api-access-74sk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:01:37.132890 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:01:37.132813    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/14096185-74a8-4e25-a035-7fd0bb5e44c9-kserve-provision-location\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:01:37.132890 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:01:37.132839    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/14096185-74a8-4e25-a035-7fd0bb5e44c9-cabundle-cert\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:01:37.132890 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:01:37.132848    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/14096185-74a8-4e25-a035-7fd0bb5e44c9-proxy-tls\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:01:37.132890 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:01:37.132857    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74sk7\" (UniqueName: \"kubernetes.io/projected/14096185-74a8-4e25-a035-7fd0bb5e44c9-kube-api-access-74sk7\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:01:37.132890 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:01:37.132865    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/14096185-74a8-4e25-a035-7fd0bb5e44c9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:01:37.639097 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:01:37.639065    2573 generic.go:356] "Generic (PLEG): container finished" podID="14096185-74a8-4e25-a035-7fd0bb5e44c9" containerID="f3d374e40b0ac27465f1faa3f55471b34c62b2a727bd79734e1bceacc350321a" exitCode=0
Jul 27 18:01:37.639290 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:01:37.639131    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xjw87"
Jul 27 18:01:37.639290 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:01:37.639144    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xjw87" event={"ID":"14096185-74a8-4e25-a035-7fd0bb5e44c9","Type":"ContainerDied","Data":"f3d374e40b0ac27465f1faa3f55471b34c62b2a727bd79734e1bceacc350321a"}
Jul 27 18:01:37.639290 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:01:37.639216    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xjw87" event={"ID":"14096185-74a8-4e25-a035-7fd0bb5e44c9","Type":"ContainerDied","Data":"2d8255a3ce03a58cf4ecaf8ecd2747abba56c5d8aeab52506135d9cd9ecc5c47"}
Jul 27 18:01:37.639290 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:01:37.639268    2573 scope.go:117] "RemoveContainer" containerID="5301424f3fe6c1519d1b2cdf75adb66838dcdd6c82bd4501c4da9fe61388a3d6"
Jul 27 18:01:37.650232 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:01:37.650217    2573 scope.go:117] "RemoveContainer" containerID="f3d374e40b0ac27465f1faa3f55471b34c62b2a727bd79734e1bceacc350321a"
Jul 27 18:01:37.657595 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:01:37.657574    2573 scope.go:117] "RemoveContainer" containerID="9dba4ae4da2a83ad99d223e8f85277565aaceab98fa65a9ad61944f1553d37f8"
Jul 27 18:01:37.664462 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:01:37.664440    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xjw87"]
Jul 27 18:01:37.665199 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:01:37.665186    2573 scope.go:117] "RemoveContainer" containerID="5301424f3fe6c1519d1b2cdf75adb66838dcdd6c82bd4501c4da9fe61388a3d6"
Jul 27 18:01:37.665502 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:01:37.665483    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5301424f3fe6c1519d1b2cdf75adb66838dcdd6c82bd4501c4da9fe61388a3d6\": container with ID starting with 5301424f3fe6c1519d1b2cdf75adb66838dcdd6c82bd4501c4da9fe61388a3d6 not found: ID does not exist" containerID="5301424f3fe6c1519d1b2cdf75adb66838dcdd6c82bd4501c4da9fe61388a3d6"
Jul 27 18:01:37.665560 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:01:37.665513    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5301424f3fe6c1519d1b2cdf75adb66838dcdd6c82bd4501c4da9fe61388a3d6"} err="failed to get container status \"5301424f3fe6c1519d1b2cdf75adb66838dcdd6c82bd4501c4da9fe61388a3d6\": rpc error: code = NotFound desc = could not find container \"5301424f3fe6c1519d1b2cdf75adb66838dcdd6c82bd4501c4da9fe61388a3d6\": container with ID starting with 5301424f3fe6c1519d1b2cdf75adb66838dcdd6c82bd4501c4da9fe61388a3d6 not found: ID does not exist"
Jul 27 18:01:37.665560 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:01:37.665531    2573 scope.go:117] "RemoveContainer" containerID="f3d374e40b0ac27465f1faa3f55471b34c62b2a727bd79734e1bceacc350321a"
Jul 27 18:01:37.665770 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:01:37.665754    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3d374e40b0ac27465f1faa3f55471b34c62b2a727bd79734e1bceacc350321a\": container with ID starting with f3d374e40b0ac27465f1faa3f55471b34c62b2a727bd79734e1bceacc350321a not found: ID does not exist" containerID="f3d374e40b0ac27465f1faa3f55471b34c62b2a727bd79734e1bceacc350321a"
Jul 27 18:01:37.665811 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:01:37.665774    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3d374e40b0ac27465f1faa3f55471b34c62b2a727bd79734e1bceacc350321a"} err="failed to get container status \"f3d374e40b0ac27465f1faa3f55471b34c62b2a727bd79734e1bceacc350321a\": rpc error: code = NotFound desc = could not find container \"f3d374e40b0ac27465f1faa3f55471b34c62b2a727bd79734e1bceacc350321a\": container with ID starting with f3d374e40b0ac27465f1faa3f55471b34c62b2a727bd79734e1bceacc350321a not found: ID does not exist"
Jul 27 18:01:37.665811 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:01:37.665791    2573 scope.go:117] "RemoveContainer" containerID="9dba4ae4da2a83ad99d223e8f85277565aaceab98fa65a9ad61944f1553d37f8"
Jul 27 18:01:37.665998 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:01:37.665979    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dba4ae4da2a83ad99d223e8f85277565aaceab98fa65a9ad61944f1553d37f8\": container with ID starting with 9dba4ae4da2a83ad99d223e8f85277565aaceab98fa65a9ad61944f1553d37f8 not found: ID does not exist" containerID="9dba4ae4da2a83ad99d223e8f85277565aaceab98fa65a9ad61944f1553d37f8"
Jul 27 18:01:37.666059 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:01:37.666006    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dba4ae4da2a83ad99d223e8f85277565aaceab98fa65a9ad61944f1553d37f8"} err="failed to get container status \"9dba4ae4da2a83ad99d223e8f85277565aaceab98fa65a9ad61944f1553d37f8\": rpc error: code = NotFound desc = could not find container \"9dba4ae4da2a83ad99d223e8f85277565aaceab98fa65a9ad61944f1553d37f8\": container with ID starting with 9dba4ae4da2a83ad99d223e8f85277565aaceab98fa65a9ad61944f1553d37f8 not found: ID does not exist"
Jul 27 18:01:37.671743 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:01:37.671721    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-xjw87"]
Jul 27 18:01:38.469289 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:01:38.469243    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14096185-74a8-4e25-a035-7fd0bb5e44c9" path="/var/lib/kubelet/pods/14096185-74a8-4e25-a035-7fd0bb5e44c9/volumes"
Jul 27 18:01:42.805063 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:01:42.805031    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:02:38.412823 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:02:38.412792    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vc2tz_87dac797-f8d2-4d36-999f-f8ce855dbeff/ovn-acl-logging/0.log"
Jul 27 18:02:38.418301 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:02:38.418273    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vc2tz_87dac797-f8d2-4d36-999f-f8ce855dbeff/ovn-acl-logging/0.log"
Jul 27 18:03:02.491537 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:02.491507    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-424c9"]
Jul 27 18:03:02.492067 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:02.491754    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-424c9" podUID="fdd8df10-64ad-4c9c-bad7-3782e38c6e68" containerName="kserve-container" containerID="cri-o://866546a58d91a9c8c1acd9b536109a3c848f390ffb3ce2f879d5af97d320fced" gracePeriod=30
Jul 27 18:03:02.492067 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:02.491791    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-424c9" podUID="fdd8df10-64ad-4c9c-bad7-3782e38c6e68" containerName="kube-rbac-proxy" containerID="cri-o://4189bee25127481114052277360bb12c93d2e608cfb1992194d8b06f01ba95c0" gracePeriod=30
Jul 27 18:03:02.508896 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:02.508839    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:03:03.013705 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:03.013672    2573 generic.go:356] "Generic (PLEG): container finished" podID="fdd8df10-64ad-4c9c-bad7-3782e38c6e68" containerID="4189bee25127481114052277360bb12c93d2e608cfb1992194d8b06f01ba95c0" exitCode=2
Jul 27 18:03:03.013705 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:03.013708    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-424c9" event={"ID":"fdd8df10-64ad-4c9c-bad7-3782e38c6e68","Type":"ContainerDied","Data":"4189bee25127481114052277360bb12c93d2e608cfb1992194d8b06f01ba95c0"}
Jul 27 18:03:04.243543 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:04.243501    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-424c9" podUID="fdd8df10-64ad-4c9c-bad7-3782e38c6e68" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused"
Jul 27 18:03:04.248891 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:04.248868    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-424c9" podUID="fdd8df10-64ad-4c9c-bad7-3782e38c6e68" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 27 18:03:07.032982 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.032948    2573 generic.go:356] "Generic (PLEG): container finished" podID="fdd8df10-64ad-4c9c-bad7-3782e38c6e68" containerID="866546a58d91a9c8c1acd9b536109a3c848f390ffb3ce2f879d5af97d320fced" exitCode=0
Jul 27 18:03:07.033314 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.033023    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-424c9" event={"ID":"fdd8df10-64ad-4c9c-bad7-3782e38c6e68","Type":"ContainerDied","Data":"866546a58d91a9c8c1acd9b536109a3c848f390ffb3ce2f879d5af97d320fced"}
Jul 27 18:03:07.033314 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.033124    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-424c9" event={"ID":"fdd8df10-64ad-4c9c-bad7-3782e38c6e68","Type":"ContainerDied","Data":"3fe537bc017d7c975c4225573b0714055c62d0e43467722d5fdce9411841c501"}
Jul 27 18:03:07.033314 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.033148    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3fe537bc017d7c975c4225573b0714055c62d0e43467722d5fdce9411841c501"
Jul 27 18:03:07.044669 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.044648    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-424c9"
Jul 27 18:03:07.095739 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.095711    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fdd8df10-64ad-4c9c-bad7-3782e38c6e68-kube-rbac-proxy-sar-config\") pod \"fdd8df10-64ad-4c9c-bad7-3782e38c6e68\" (UID: \"fdd8df10-64ad-4c9c-bad7-3782e38c6e68\") "
Jul 27 18:03:07.095903 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.095759    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdd8df10-64ad-4c9c-bad7-3782e38c6e68-kserve-provision-location\") pod \"fdd8df10-64ad-4c9c-bad7-3782e38c6e68\" (UID: \"fdd8df10-64ad-4c9c-bad7-3782e38c6e68\") "
Jul 27 18:03:07.095903 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.095798    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fdd8df10-64ad-4c9c-bad7-3782e38c6e68-proxy-tls\") pod \"fdd8df10-64ad-4c9c-bad7-3782e38c6e68\" (UID: \"fdd8df10-64ad-4c9c-bad7-3782e38c6e68\") "
Jul 27 18:03:07.095903 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.095854    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fdd8df10-64ad-4c9c-bad7-3782e38c6e68-cabundle-cert\") pod \"fdd8df10-64ad-4c9c-bad7-3782e38c6e68\" (UID: \"fdd8df10-64ad-4c9c-bad7-3782e38c6e68\") "
Jul 27 18:03:07.095903 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.095880    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9dk5l\" (UniqueName: \"kubernetes.io/projected/fdd8df10-64ad-4c9c-bad7-3782e38c6e68-kube-api-access-9dk5l\") pod \"fdd8df10-64ad-4c9c-bad7-3782e38c6e68\" (UID: \"fdd8df10-64ad-4c9c-bad7-3782e38c6e68\") "
Jul 27 18:03:07.096194 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.096128    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdd8df10-64ad-4c9c-bad7-3782e38c6e68-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fdd8df10-64ad-4c9c-bad7-3782e38c6e68" (UID: "fdd8df10-64ad-4c9c-bad7-3782e38c6e68"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:03:07.096194 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.096167    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdd8df10-64ad-4c9c-bad7-3782e38c6e68-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fdd8df10-64ad-4c9c-bad7-3782e38c6e68" (UID: "fdd8df10-64ad-4c9c-bad7-3782e38c6e68"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:03:07.096322 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.096169    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdd8df10-64ad-4c9c-bad7-3782e38c6e68-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fdd8df10-64ad-4c9c-bad7-3782e38c6e68" (UID: "fdd8df10-64ad-4c9c-bad7-3782e38c6e68"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:03:07.098049 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.098025    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdd8df10-64ad-4c9c-bad7-3782e38c6e68-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fdd8df10-64ad-4c9c-bad7-3782e38c6e68" (UID: "fdd8df10-64ad-4c9c-bad7-3782e38c6e68"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:03:07.098121 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.098029    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdd8df10-64ad-4c9c-bad7-3782e38c6e68-kube-api-access-9dk5l" (OuterVolumeSpecName: "kube-api-access-9dk5l") pod "fdd8df10-64ad-4c9c-bad7-3782e38c6e68" (UID: "fdd8df10-64ad-4c9c-bad7-3782e38c6e68"). InnerVolumeSpecName "kube-api-access-9dk5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:03:07.196690 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.196616    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fdd8df10-64ad-4c9c-bad7-3782e38c6e68-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:03:07.196690 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.196644    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdd8df10-64ad-4c9c-bad7-3782e38c6e68-kserve-provision-location\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:03:07.196690 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.196655    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fdd8df10-64ad-4c9c-bad7-3782e38c6e68-proxy-tls\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:03:07.196690 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.196667    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fdd8df10-64ad-4c9c-bad7-3782e38c6e68-cabundle-cert\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:03:07.196690 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.196676    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dk5l\" (UniqueName: \"kubernetes.io/projected/fdd8df10-64ad-4c9c-bad7-3782e38c6e68-kube-api-access-9dk5l\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:03:07.603238 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.603146    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:03:07.664378 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.664345    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-xd7r4"]
Jul 27 18:03:07.664701 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.664687    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14096185-74a8-4e25-a035-7fd0bb5e44c9" containerName="kserve-container"
Jul 27 18:03:07.664764 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.664703    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="14096185-74a8-4e25-a035-7fd0bb5e44c9" containerName="kserve-container"
Jul 27 18:03:07.664764 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.664712    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdd8df10-64ad-4c9c-bad7-3782e38c6e68" containerName="kserve-container"
Jul 27 18:03:07.664764 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.664717    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdd8df10-64ad-4c9c-bad7-3782e38c6e68" containerName="kserve-container"
Jul 27 18:03:07.664764 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.664723    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14096185-74a8-4e25-a035-7fd0bb5e44c9" containerName="kube-rbac-proxy"
Jul 27 18:03:07.664764 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.664729    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="14096185-74a8-4e25-a035-7fd0bb5e44c9" containerName="kube-rbac-proxy"
Jul 27 18:03:07.664764 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.664743    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14096185-74a8-4e25-a035-7fd0bb5e44c9" containerName="storage-initializer"
Jul 27 18:03:07.664764 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.664751    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="14096185-74a8-4e25-a035-7fd0bb5e44c9" containerName="storage-initializer"
Jul 27 18:03:07.664764 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.664760    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdd8df10-64ad-4c9c-bad7-3782e38c6e68" containerName="storage-initializer"
Jul 27 18:03:07.664764 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.664767    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdd8df10-64ad-4c9c-bad7-3782e38c6e68" containerName="storage-initializer"
Jul 27 18:03:07.665052 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.664773    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdd8df10-64ad-4c9c-bad7-3782e38c6e68" containerName="kube-rbac-proxy"
Jul 27 18:03:07.665052 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.664778    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdd8df10-64ad-4c9c-bad7-3782e38c6e68" containerName="kube-rbac-proxy"
Jul 27 18:03:07.665052 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.664841    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdd8df10-64ad-4c9c-bad7-3782e38c6e68" containerName="kube-rbac-proxy"
Jul 27 18:03:07.665052 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.664850    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="14096185-74a8-4e25-a035-7fd0bb5e44c9" containerName="kube-rbac-proxy"
Jul 27 18:03:07.665052 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.664858    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdd8df10-64ad-4c9c-bad7-3782e38c6e68" containerName="kserve-container"
Jul 27 18:03:07.665052 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.664867    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="14096185-74a8-4e25-a035-7fd0bb5e44c9" containerName="kserve-container"
Jul 27 18:03:07.668207 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.668190    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-xd7r4"
Jul 27 18:03:07.670847 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.670824    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Jul 27 18:03:07.670967 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.670822    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Jul 27 18:03:07.675382 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.675362    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-pq9hv"]
Jul 27 18:03:07.678967 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.678947    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-xd7r4"]
Jul 27 18:03:07.679053 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.679037    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-pq9hv"
Jul 27 18:03:07.681672 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.681656    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Jul 27 18:03:07.690216 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.690193    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-pq9hv"]
Jul 27 18:03:07.697414 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.697394    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:03:07.699810 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.699784    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ea186d7-b2b1-4d9d-bf52-970af0da953f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-77f778d864-pq9hv\" (UID: \"6ea186d7-b2b1-4d9d-bf52-970af0da953f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-pq9hv"
Jul 27 18:03:07.699888 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.699833    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/410dfec0-91d8-4333-bc35-e7b19a1e314b-proxy-tls\") pod \"isvc-canary-stop-predictor-86dc9fdb65-xd7r4\" (UID: \"410dfec0-91d8-4333-bc35-e7b19a1e314b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-xd7r4"
Jul 27 18:03:07.699888 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.699862    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/410dfec0-91d8-4333-bc35-e7b19a1e314b-kserve-provision-location\") pod \"isvc-canary-stop-predictor-86dc9fdb65-xd7r4\" (UID: \"410dfec0-91d8-4333-bc35-e7b19a1e314b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-xd7r4"
Jul 27 18:03:07.699961 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.699905    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/410dfec0-91d8-4333-bc35-e7b19a1e314b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-86dc9fdb65-xd7r4\" (UID: \"410dfec0-91d8-4333-bc35-e7b19a1e314b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-xd7r4"
Jul 27 18:03:07.699994 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.699976    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzgs5\" (UniqueName: \"kubernetes.io/projected/410dfec0-91d8-4333-bc35-e7b19a1e314b-kube-api-access-bzgs5\") pod \"isvc-canary-stop-predictor-86dc9fdb65-xd7r4\" (UID: \"410dfec0-91d8-4333-bc35-e7b19a1e314b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-xd7r4"
Jul 27 18:03:07.700031 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.700017    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/410dfec0-91d8-4333-bc35-e7b19a1e314b-cabundle-cert\") pod \"isvc-canary-stop-predictor-86dc9fdb65-xd7r4\" (UID: \"410dfec0-91d8-4333-bc35-e7b19a1e314b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-xd7r4"
Jul 27 18:03:07.700078 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.700062    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ea186d7-b2b1-4d9d-bf52-970af0da953f-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-77f778d864-pq9hv\" (UID: \"6ea186d7-b2b1-4d9d-bf52-970af0da953f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-pq9hv"
Jul 27 18:03:07.700138 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.700121    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ea186d7-b2b1-4d9d-bf52-970af0da953f-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-77f778d864-pq9hv\" (UID: \"6ea186d7-b2b1-4d9d-bf52-970af0da953f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-pq9hv"
Jul 27 18:03:07.700174 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.700157    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ea186d7-b2b1-4d9d-bf52-970af0da953f-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-77f778d864-pq9hv\" (UID: \"6ea186d7-b2b1-4d9d-bf52-970af0da953f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-pq9hv"
Jul 27 18:03:07.700213 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.700180    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2c58\" (UniqueName: \"kubernetes.io/projected/6ea186d7-b2b1-4d9d-bf52-970af0da953f-kube-api-access-d2c58\") pod \"isvc-canary-stop-v2-predictor-77f778d864-pq9hv\" (UID: \"6ea186d7-b2b1-4d9d-bf52-970af0da953f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-pq9hv"
Jul 27 18:03:07.801329 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.801300    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ea186d7-b2b1-4d9d-bf52-970af0da953f-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-77f778d864-pq9hv\" (UID: \"6ea186d7-b2b1-4d9d-bf52-970af0da953f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-pq9hv"
Jul 27 18:03:07.801477 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.801338    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ea186d7-b2b1-4d9d-bf52-970af0da953f-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-77f778d864-pq9hv\" (UID: \"6ea186d7-b2b1-4d9d-bf52-970af0da953f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-pq9hv"
Jul 27 18:03:07.801477 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.801359    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2c58\" (UniqueName: \"kubernetes.io/projected/6ea186d7-b2b1-4d9d-bf52-970af0da953f-kube-api-access-d2c58\") pod \"isvc-canary-stop-v2-predictor-77f778d864-pq9hv\" (UID: \"6ea186d7-b2b1-4d9d-bf52-970af0da953f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-pq9hv"
Jul 27 18:03:07.801477 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.801391    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ea186d7-b2b1-4d9d-bf52-970af0da953f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-77f778d864-pq9hv\" (UID: \"6ea186d7-b2b1-4d9d-bf52-970af0da953f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-pq9hv"
Jul 27 18:03:07.801477 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.801421    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/410dfec0-91d8-4333-bc35-e7b19a1e314b-proxy-tls\") pod \"isvc-canary-stop-predictor-86dc9fdb65-xd7r4\" (UID: \"410dfec0-91d8-4333-bc35-e7b19a1e314b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-xd7r4"
Jul 27 18:03:07.801477 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.801446    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/410dfec0-91d8-4333-bc35-e7b19a1e314b-kserve-provision-location\") pod \"isvc-canary-stop-predictor-86dc9fdb65-xd7r4\" (UID: \"410dfec0-91d8-4333-bc35-e7b19a1e314b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-xd7r4"
Jul 27 18:03:07.801743 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.801486    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/410dfec0-91d8-4333-bc35-e7b19a1e314b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-86dc9fdb65-xd7r4\" (UID: \"410dfec0-91d8-4333-bc35-e7b19a1e314b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-xd7r4"
Jul 27 18:03:07.801743 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.801539    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzgs5\" (UniqueName: \"kubernetes.io/projected/410dfec0-91d8-4333-bc35-e7b19a1e314b-kube-api-access-bzgs5\") pod \"isvc-canary-stop-predictor-86dc9fdb65-xd7r4\" (UID: \"410dfec0-91d8-4333-bc35-e7b19a1e314b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-xd7r4"
Jul 27 18:03:07.801743 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.801568    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/410dfec0-91d8-4333-bc35-e7b19a1e314b-cabundle-cert\") pod \"isvc-canary-stop-predictor-86dc9fdb65-xd7r4\" (UID: \"410dfec0-91d8-4333-bc35-e7b19a1e314b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-xd7r4"
Jul 27 18:03:07.801743 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:03:07.801595    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-predictor-serving-cert: secret "isvc-canary-stop-predictor-serving-cert" not found
Jul 27 18:03:07.801743 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:03:07.801690    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-serving-cert: secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Jul 27 18:03:07.801743 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:03:07.801697    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/410dfec0-91d8-4333-bc35-e7b19a1e314b-proxy-tls podName:410dfec0-91d8-4333-bc35-e7b19a1e314b nodeName:}" failed. No retries permitted until 2026-07-27 18:03:08.301674932 +0000 UTC m=+2130.445076962 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/410dfec0-91d8-4333-bc35-e7b19a1e314b-proxy-tls") pod "isvc-canary-stop-predictor-86dc9fdb65-xd7r4" (UID: "410dfec0-91d8-4333-bc35-e7b19a1e314b") : secret "isvc-canary-stop-predictor-serving-cert" not found
Jul 27 18:03:07.801743 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.801734    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ea186d7-b2b1-4d9d-bf52-970af0da953f-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-77f778d864-pq9hv\" (UID: \"6ea186d7-b2b1-4d9d-bf52-970af0da953f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-pq9hv"
Jul 27 18:03:07.802127 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:03:07.801771    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ea186d7-b2b1-4d9d-bf52-970af0da953f-proxy-tls podName:6ea186d7-b2b1-4d9d-bf52-970af0da953f nodeName:}" failed. No retries permitted until 2026-07-27 18:03:08.301753604 +0000 UTC m=+2130.445155647 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6ea186d7-b2b1-4d9d-bf52-970af0da953f-proxy-tls") pod "isvc-canary-stop-v2-predictor-77f778d864-pq9hv" (UID: "6ea186d7-b2b1-4d9d-bf52-970af0da953f") : secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Jul 27 18:03:07.802127 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.801601    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ea186d7-b2b1-4d9d-bf52-970af0da953f-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-77f778d864-pq9hv\" (UID: \"6ea186d7-b2b1-4d9d-bf52-970af0da953f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-pq9hv"
Jul 27 18:03:07.802127 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.801845    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/410dfec0-91d8-4333-bc35-e7b19a1e314b-kserve-provision-location\") pod \"isvc-canary-stop-predictor-86dc9fdb65-xd7r4\" (UID: \"410dfec0-91d8-4333-bc35-e7b19a1e314b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-xd7r4"
Jul 27 18:03:07.802287 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.802160    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ea186d7-b2b1-4d9d-bf52-970af0da953f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-77f778d864-pq9hv\" (UID: \"6ea186d7-b2b1-4d9d-bf52-970af0da953f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-pq9hv"
Jul 27 18:03:07.802287 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.802159    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ea186d7-b2b1-4d9d-bf52-970af0da953f-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-77f778d864-pq9hv\" (UID: \"6ea186d7-b2b1-4d9d-bf52-970af0da953f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-pq9hv"
Jul 27 18:03:07.802434 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.802413    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/410dfec0-91d8-4333-bc35-e7b19a1e314b-cabundle-cert\") pod \"isvc-canary-stop-predictor-86dc9fdb65-xd7r4\" (UID: \"410dfec0-91d8-4333-bc35-e7b19a1e314b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-xd7r4"
Jul 27 18:03:07.802471 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.802457    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/410dfec0-91d8-4333-bc35-e7b19a1e314b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-86dc9fdb65-xd7r4\" (UID: \"410dfec0-91d8-4333-bc35-e7b19a1e314b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-xd7r4"
Jul 27 18:03:07.812131 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.812105    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2c58\" (UniqueName: \"kubernetes.io/projected/6ea186d7-b2b1-4d9d-bf52-970af0da953f-kube-api-access-d2c58\") pod \"isvc-canary-stop-v2-predictor-77f778d864-pq9hv\" (UID: \"6ea186d7-b2b1-4d9d-bf52-970af0da953f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-pq9hv"
Jul 27 18:03:07.812225 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.812105    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzgs5\" (UniqueName: \"kubernetes.io/projected/410dfec0-91d8-4333-bc35-e7b19a1e314b-kube-api-access-bzgs5\") pod \"isvc-canary-stop-predictor-86dc9fdb65-xd7r4\" (UID: \"410dfec0-91d8-4333-bc35-e7b19a1e314b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-xd7r4"
Jul 27 18:03:07.863530 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:07.863448    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:03:08.037575 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:08.037545    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-424c9"
Jul 27 18:03:08.059399 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:08.059376    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-424c9"]
Jul 27 18:03:08.066030 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:08.066007    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-424c9"]
Jul 27 18:03:08.115848 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:08.115784    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:03:08.306297 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:08.306268    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/410dfec0-91d8-4333-bc35-e7b19a1e314b-proxy-tls\") pod \"isvc-canary-stop-predictor-86dc9fdb65-xd7r4\" (UID: \"410dfec0-91d8-4333-bc35-e7b19a1e314b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-xd7r4"
Jul 27 18:03:08.306435 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:08.306326    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ea186d7-b2b1-4d9d-bf52-970af0da953f-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-77f778d864-pq9hv\" (UID: \"6ea186d7-b2b1-4d9d-bf52-970af0da953f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-pq9hv"
Jul 27 18:03:08.308761 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:08.308734    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/410dfec0-91d8-4333-bc35-e7b19a1e314b-proxy-tls\") pod \"isvc-canary-stop-predictor-86dc9fdb65-xd7r4\" (UID: \"410dfec0-91d8-4333-bc35-e7b19a1e314b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-xd7r4"
Jul 27 18:03:08.308856 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:08.308823    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ea186d7-b2b1-4d9d-bf52-970af0da953f-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-77f778d864-pq9hv\" (UID: \"6ea186d7-b2b1-4d9d-bf52-970af0da953f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-pq9hv"
Jul 27 18:03:08.469341 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:08.469306    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdd8df10-64ad-4c9c-bad7-3782e38c6e68" path="/var/lib/kubelet/pods/fdd8df10-64ad-4c9c-bad7-3782e38c6e68/volumes"
Jul 27 18:03:08.579498 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:08.579459    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-xd7r4"
Jul 27 18:03:08.588908 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:08.588882    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-pq9hv"
Jul 27 18:03:08.721311 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:08.721283    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-xd7r4"]
Jul 27 18:03:08.723282 ip-10-0-140-0 kubenswrapper[2573]: W0727 18:03:08.723216    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod410dfec0_91d8_4333_bc35_e7b19a1e314b.slice/crio-b2d7da22b9af6bf6aad72813fbdf906bf0b43878d6be2d2dc05c1d0471b63164 WatchSource:0}: Error finding container b2d7da22b9af6bf6aad72813fbdf906bf0b43878d6be2d2dc05c1d0471b63164: Status 404 returned error can't find the container with id b2d7da22b9af6bf6aad72813fbdf906bf0b43878d6be2d2dc05c1d0471b63164
Jul 27 18:03:08.739934 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:08.739914    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-pq9hv"]
Jul 27 18:03:08.741967 ip-10-0-140-0 kubenswrapper[2573]: W0727 18:03:08.741943    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ea186d7_b2b1_4d9d_bf52_970af0da953f.slice/crio-0d2a0a810b4a941733ae440748eaed4f6f3d645c725f24448eed2d554083d82a WatchSource:0}: Error finding container 0d2a0a810b4a941733ae440748eaed4f6f3d645c725f24448eed2d554083d82a: Status 404 returned error can't find the container with id 0d2a0a810b4a941733ae440748eaed4f6f3d645c725f24448eed2d554083d82a
Jul 27 18:03:09.043921 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:09.043822    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-xd7r4" event={"ID":"410dfec0-91d8-4333-bc35-e7b19a1e314b","Type":"ContainerStarted","Data":"3092a1a28157e5cb16c1c869c8e8a72fb6b1f9ab21c4b7eb06aa3bf5cf9701b9"}
Jul 27 18:03:09.043921 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:09.043897    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-xd7r4" event={"ID":"410dfec0-91d8-4333-bc35-e7b19a1e314b","Type":"ContainerStarted","Data":"b2d7da22b9af6bf6aad72813fbdf906bf0b43878d6be2d2dc05c1d0471b63164"}
Jul 27 18:03:09.045554 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:09.045516    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-pq9hv" event={"ID":"6ea186d7-b2b1-4d9d-bf52-970af0da953f","Type":"ContainerStarted","Data":"2360c0ac4a5daaf50f7652fbd7b89fd926486b91e49a88f3f040dc6041cf3bed"}
Jul 27 18:03:09.045676 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:09.045584    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-pq9hv" event={"ID":"6ea186d7-b2b1-4d9d-bf52-970af0da953f","Type":"ContainerStarted","Data":"0d2a0a810b4a941733ae440748eaed4f6f3d645c725f24448eed2d554083d82a"}
Jul 27 18:03:13.067696 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:13.067662    2573 generic.go:356] "Generic (PLEG): container finished" podID="6ea186d7-b2b1-4d9d-bf52-970af0da953f" containerID="2360c0ac4a5daaf50f7652fbd7b89fd926486b91e49a88f3f040dc6041cf3bed" exitCode=0
Jul 27 18:03:13.068143 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:13.067738    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-pq9hv" event={"ID":"6ea186d7-b2b1-4d9d-bf52-970af0da953f","Type":"ContainerDied","Data":"2360c0ac4a5daaf50f7652fbd7b89fd926486b91e49a88f3f040dc6041cf3bed"}
Jul 27 18:03:13.069419 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:13.069392    2573 generic.go:356] "Generic (PLEG): container finished" podID="410dfec0-91d8-4333-bc35-e7b19a1e314b" containerID="3092a1a28157e5cb16c1c869c8e8a72fb6b1f9ab21c4b7eb06aa3bf5cf9701b9" exitCode=0
Jul 27 18:03:13.069534 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:13.069444    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-xd7r4" event={"ID":"410dfec0-91d8-4333-bc35-e7b19a1e314b","Type":"ContainerDied","Data":"3092a1a28157e5cb16c1c869c8e8a72fb6b1f9ab21c4b7eb06aa3bf5cf9701b9"}
Jul 27 18:03:14.076164 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:14.076127    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-xd7r4" event={"ID":"410dfec0-91d8-4333-bc35-e7b19a1e314b","Type":"ContainerStarted","Data":"319718db98ba35e65b7cf685c8135c958c34525d60c9af7547249a2666fed37b"}
Jul 27 18:03:14.076612 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:14.076203    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-xd7r4" event={"ID":"410dfec0-91d8-4333-bc35-e7b19a1e314b","Type":"ContainerStarted","Data":"870f3e9c167f4a0423363ef4ac7d90ddce591e8e6127e09af969f320c9ed23a7"}
Jul 27 18:03:14.076612 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:14.076508    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-xd7r4"
Jul 27 18:03:14.076764 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:14.076663    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-xd7r4"
Jul 27 18:03:14.077875 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:14.077839    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-xd7r4" podUID="410dfec0-91d8-4333-bc35-e7b19a1e314b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 27 18:03:14.078307 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:14.078286    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-pq9hv" event={"ID":"6ea186d7-b2b1-4d9d-bf52-970af0da953f","Type":"ContainerStarted","Data":"9c722ea6202c93abd99596bba6b82a28f8bf4163de1860ccd043f2c84597a9b7"}
Jul 27 18:03:14.078375 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:14.078331    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-pq9hv" event={"ID":"6ea186d7-b2b1-4d9d-bf52-970af0da953f","Type":"ContainerStarted","Data":"8b920b150df382abe9d651d0b9e92d95dc8158a34d387cc36a504ef9daeff798"}
Jul 27 18:03:14.078520 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:14.078502    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-pq9hv"
Jul 27 18:03:14.078558 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:14.078544    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-pq9hv"
Jul 27 18:03:14.079488 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:14.079464    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-pq9hv" podUID="6ea186d7-b2b1-4d9d-bf52-970af0da953f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 27 18:03:14.096151 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:14.096106    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-xd7r4" podStartSLOduration=3.57382518 podStartE2EDuration="7.096090336s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.522265156s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:03:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:03:14.095557619 +0000 UTC m=+2136.238959671" watchObservedRunningTime="2026-07-27 18:03:14.096090336 +0000 UTC m=+2136.239492386"
Jul 27 18:03:14.114833 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:14.114784    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-pq9hv" podStartSLOduration=3.375882829 podStartE2EDuration="7.114769115s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.738886286s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:03:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:03:14.113795724 +0000 UTC m=+2136.257197777" watchObservedRunningTime="2026-07-27 18:03:14.114769115 +0000 UTC m=+2136.258171167"
Jul 27 18:03:15.083705 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:15.083666    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-pq9hv" podUID="6ea186d7-b2b1-4d9d-bf52-970af0da953f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 27 18:03:15.084149 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:15.083735    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-xd7r4" podUID="410dfec0-91d8-4333-bc35-e7b19a1e314b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 27 18:03:20.089486 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:20.089455    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-pq9hv"
Jul 27 18:03:20.089968 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:20.089732    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-xd7r4"
Jul 27 18:03:20.090081 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:20.090055    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-pq9hv" podUID="6ea186d7-b2b1-4d9d-bf52-970af0da953f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 27 18:03:20.090406 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:20.090380    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-xd7r4" podUID="410dfec0-91d8-4333-bc35-e7b19a1e314b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 27 18:03:30.091157 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:30.091060    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-xd7r4" podUID="410dfec0-91d8-4333-bc35-e7b19a1e314b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 27 18:03:30.091619 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:30.091060    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-pq9hv" podUID="6ea186d7-b2b1-4d9d-bf52-970af0da953f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 27 18:03:40.090839 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:40.090795    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-xd7r4" podUID="410dfec0-91d8-4333-bc35-e7b19a1e314b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 27 18:03:40.090839 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:40.090836    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-pq9hv" podUID="6ea186d7-b2b1-4d9d-bf52-970af0da953f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 27 18:03:50.090410 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:50.090364    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-xd7r4" podUID="410dfec0-91d8-4333-bc35-e7b19a1e314b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 27 18:03:50.090832 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:03:50.090372    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-pq9hv" podUID="6ea186d7-b2b1-4d9d-bf52-970af0da953f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 27 18:04:00.090445 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:00.090403    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-pq9hv" podUID="6ea186d7-b2b1-4d9d-bf52-970af0da953f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 27 18:04:00.090979 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:00.090416    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-xd7r4" podUID="410dfec0-91d8-4333-bc35-e7b19a1e314b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 27 18:04:05.813082 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:05.813018    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:04:05.887944 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:05.887898    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:04:10.090864 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:10.090823    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-xd7r4" podUID="410dfec0-91d8-4333-bc35-e7b19a1e314b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 27 18:04:10.091290 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:10.090895    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-pq9hv" podUID="6ea186d7-b2b1-4d9d-bf52-970af0da953f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 27 18:04:20.090348 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:20.090300    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-pq9hv" podUID="6ea186d7-b2b1-4d9d-bf52-970af0da953f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 27 18:04:20.090836 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:20.090364    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-xd7r4" podUID="410dfec0-91d8-4333-bc35-e7b19a1e314b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 27 18:04:23.463935 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:23.463896    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-xd7r4" podUID="410dfec0-91d8-4333-bc35-e7b19a1e314b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 27 18:04:30.090676 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:30.090647    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-pq9hv"
Jul 27 18:04:33.464429 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:33.464400    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-xd7r4"
Jul 27 18:04:37.670652 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:37.670610    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-xd7r4"]
Jul 27 18:04:37.671179 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:37.670921    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-xd7r4" podUID="410dfec0-91d8-4333-bc35-e7b19a1e314b" containerName="kserve-container" containerID="cri-o://870f3e9c167f4a0423363ef4ac7d90ddce591e8e6127e09af969f320c9ed23a7" gracePeriod=30
Jul 27 18:04:37.671179 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:37.670977    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-xd7r4" podUID="410dfec0-91d8-4333-bc35-e7b19a1e314b" containerName="kube-rbac-proxy" containerID="cri-o://319718db98ba35e65b7cf685c8135c958c34525d60c9af7547249a2666fed37b" gracePeriod=30
Jul 27 18:04:37.688041 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:37.688010    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:04:37.700755 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:37.700731    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-pq9hv"]
Jul 27 18:04:37.701009 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:37.700978    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-pq9hv" podUID="6ea186d7-b2b1-4d9d-bf52-970af0da953f" containerName="kserve-container" containerID="cri-o://8b920b150df382abe9d651d0b9e92d95dc8158a34d387cc36a504ef9daeff798" gracePeriod=30
Jul 27 18:04:37.701113 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:37.700988    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-pq9hv" podUID="6ea186d7-b2b1-4d9d-bf52-970af0da953f" containerName="kube-rbac-proxy" containerID="cri-o://9c722ea6202c93abd99596bba6b82a28f8bf4163de1860ccd043f2c84597a9b7" gracePeriod=30
Jul 27 18:04:37.746468 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:37.746439    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:04:38.475446 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:38.475416    2573 generic.go:356] "Generic (PLEG): container finished" podID="410dfec0-91d8-4333-bc35-e7b19a1e314b" containerID="319718db98ba35e65b7cf685c8135c958c34525d60c9af7547249a2666fed37b" exitCode=2
Jul 27 18:04:38.475630 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:38.475488    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-xd7r4" event={"ID":"410dfec0-91d8-4333-bc35-e7b19a1e314b","Type":"ContainerDied","Data":"319718db98ba35e65b7cf685c8135c958c34525d60c9af7547249a2666fed37b"}
Jul 27 18:04:38.477530 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:38.477509    2573 generic.go:356] "Generic (PLEG): container finished" podID="6ea186d7-b2b1-4d9d-bf52-970af0da953f" containerID="9c722ea6202c93abd99596bba6b82a28f8bf4163de1860ccd043f2c84597a9b7" exitCode=2
Jul 27 18:04:38.477651 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:38.477551    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-pq9hv" event={"ID":"6ea186d7-b2b1-4d9d-bf52-970af0da953f","Type":"ContainerDied","Data":"9c722ea6202c93abd99596bba6b82a28f8bf4163de1860ccd043f2c84597a9b7"}
Jul 27 18:04:40.084813 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:40.084777    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-xd7r4" podUID="410dfec0-91d8-4333-bc35-e7b19a1e314b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": dial tcp 10.133.0.44:8643: connect: connection refused"
Jul 27 18:04:40.085218 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:40.084780    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-pq9hv" podUID="6ea186d7-b2b1-4d9d-bf52-970af0da953f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.45:8643/healthz\": dial tcp 10.133.0.45:8643: connect: connection refused"
Jul 27 18:04:40.090884 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:40.090850    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-pq9hv" podUID="6ea186d7-b2b1-4d9d-bf52-970af0da953f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 27 18:04:42.495119 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:42.495087    2573 generic.go:356] "Generic (PLEG): container finished" podID="410dfec0-91d8-4333-bc35-e7b19a1e314b" containerID="870f3e9c167f4a0423363ef4ac7d90ddce591e8e6127e09af969f320c9ed23a7" exitCode=0
Jul 27 18:04:42.495505 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:42.495169    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-xd7r4" event={"ID":"410dfec0-91d8-4333-bc35-e7b19a1e314b","Type":"ContainerDied","Data":"870f3e9c167f4a0423363ef4ac7d90ddce591e8e6127e09af969f320c9ed23a7"}
Jul 27 18:04:42.495505 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:42.495274    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-xd7r4" event={"ID":"410dfec0-91d8-4333-bc35-e7b19a1e314b","Type":"ContainerDied","Data":"b2d7da22b9af6bf6aad72813fbdf906bf0b43878d6be2d2dc05c1d0471b63164"}
Jul 27 18:04:42.495505 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:42.495309    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2d7da22b9af6bf6aad72813fbdf906bf0b43878d6be2d2dc05c1d0471b63164"
Jul 27 18:04:42.563149 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:42.563125    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-xd7r4"
Jul 27 18:04:42.655423 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:42.655400    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-pq9hv"
Jul 27 18:04:42.696217 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:42.696191    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/410dfec0-91d8-4333-bc35-e7b19a1e314b-kube-rbac-proxy-sar-config\") pod \"410dfec0-91d8-4333-bc35-e7b19a1e314b\" (UID: \"410dfec0-91d8-4333-bc35-e7b19a1e314b\") "
Jul 27 18:04:42.696217 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:42.696223    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/410dfec0-91d8-4333-bc35-e7b19a1e314b-cabundle-cert\") pod \"410dfec0-91d8-4333-bc35-e7b19a1e314b\" (UID: \"410dfec0-91d8-4333-bc35-e7b19a1e314b\") "
Jul 27 18:04:42.696217 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:42.696243    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/410dfec0-91d8-4333-bc35-e7b19a1e314b-kserve-provision-location\") pod \"410dfec0-91d8-4333-bc35-e7b19a1e314b\" (UID: \"410dfec0-91d8-4333-bc35-e7b19a1e314b\") "
Jul 27 18:04:42.696567 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:42.696322    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/410dfec0-91d8-4333-bc35-e7b19a1e314b-proxy-tls\") pod \"410dfec0-91d8-4333-bc35-e7b19a1e314b\" (UID: \"410dfec0-91d8-4333-bc35-e7b19a1e314b\") "
Jul 27 18:04:42.696567 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:42.696342    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bzgs5\" (UniqueName: \"kubernetes.io/projected/410dfec0-91d8-4333-bc35-e7b19a1e314b-kube-api-access-bzgs5\") pod \"410dfec0-91d8-4333-bc35-e7b19a1e314b\" (UID: \"410dfec0-91d8-4333-bc35-e7b19a1e314b\") "
Jul 27 18:04:42.696688 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:42.696627    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/410dfec0-91d8-4333-bc35-e7b19a1e314b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "410dfec0-91d8-4333-bc35-e7b19a1e314b" (UID: "410dfec0-91d8-4333-bc35-e7b19a1e314b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:04:42.696688 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:42.696632    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/410dfec0-91d8-4333-bc35-e7b19a1e314b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "410dfec0-91d8-4333-bc35-e7b19a1e314b" (UID: "410dfec0-91d8-4333-bc35-e7b19a1e314b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:04:42.696688 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:42.696665    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/410dfec0-91d8-4333-bc35-e7b19a1e314b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "410dfec0-91d8-4333-bc35-e7b19a1e314b" (UID: "410dfec0-91d8-4333-bc35-e7b19a1e314b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:04:42.698514 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:42.698488    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/410dfec0-91d8-4333-bc35-e7b19a1e314b-kube-api-access-bzgs5" (OuterVolumeSpecName: "kube-api-access-bzgs5") pod "410dfec0-91d8-4333-bc35-e7b19a1e314b" (UID: "410dfec0-91d8-4333-bc35-e7b19a1e314b"). InnerVolumeSpecName "kube-api-access-bzgs5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:04:42.698619 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:42.698533    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/410dfec0-91d8-4333-bc35-e7b19a1e314b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "410dfec0-91d8-4333-bc35-e7b19a1e314b" (UID: "410dfec0-91d8-4333-bc35-e7b19a1e314b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:04:42.797537 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:42.797454    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d2c58\" (UniqueName: \"kubernetes.io/projected/6ea186d7-b2b1-4d9d-bf52-970af0da953f-kube-api-access-d2c58\") pod \"6ea186d7-b2b1-4d9d-bf52-970af0da953f\" (UID: \"6ea186d7-b2b1-4d9d-bf52-970af0da953f\") "
Jul 27 18:04:42.797537 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:42.797497    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ea186d7-b2b1-4d9d-bf52-970af0da953f-proxy-tls\") pod \"6ea186d7-b2b1-4d9d-bf52-970af0da953f\" (UID: \"6ea186d7-b2b1-4d9d-bf52-970af0da953f\") "
Jul 27 18:04:42.797537 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:42.797532    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ea186d7-b2b1-4d9d-bf52-970af0da953f-kserve-provision-location\") pod \"6ea186d7-b2b1-4d9d-bf52-970af0da953f\" (UID: \"6ea186d7-b2b1-4d9d-bf52-970af0da953f\") "
Jul 27 18:04:42.797806 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:42.797567    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ea186d7-b2b1-4d9d-bf52-970af0da953f-cabundle-cert\") pod \"6ea186d7-b2b1-4d9d-bf52-970af0da953f\" (UID: \"6ea186d7-b2b1-4d9d-bf52-970af0da953f\") "
Jul 27 18:04:42.797806 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:42.797612    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ea186d7-b2b1-4d9d-bf52-970af0da953f-kube-rbac-proxy-sar-config\") pod \"6ea186d7-b2b1-4d9d-bf52-970af0da953f\" (UID: \"6ea186d7-b2b1-4d9d-bf52-970af0da953f\") "
Jul 27 18:04:42.797806 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:42.797755    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/410dfec0-91d8-4333-bc35-e7b19a1e314b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:04:42.797806 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:42.797766    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/410dfec0-91d8-4333-bc35-e7b19a1e314b-cabundle-cert\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:04:42.797806 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:42.797780    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/410dfec0-91d8-4333-bc35-e7b19a1e314b-kserve-provision-location\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:04:42.797806 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:42.797797    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/410dfec0-91d8-4333-bc35-e7b19a1e314b-proxy-tls\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:04:42.798021 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:42.797810    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzgs5\" (UniqueName: \"kubernetes.io/projected/410dfec0-91d8-4333-bc35-e7b19a1e314b-kube-api-access-bzgs5\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:04:42.798021 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:42.797905    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea186d7-b2b1-4d9d-bf52-970af0da953f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6ea186d7-b2b1-4d9d-bf52-970af0da953f" (UID: "6ea186d7-b2b1-4d9d-bf52-970af0da953f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:04:42.798021 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:42.797910    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ea186d7-b2b1-4d9d-bf52-970af0da953f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6ea186d7-b2b1-4d9d-bf52-970af0da953f" (UID: "6ea186d7-b2b1-4d9d-bf52-970af0da953f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:04:42.798021 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:42.797967    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ea186d7-b2b1-4d9d-bf52-970af0da953f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6ea186d7-b2b1-4d9d-bf52-970af0da953f" (UID: "6ea186d7-b2b1-4d9d-bf52-970af0da953f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:04:42.799697 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:42.799675    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ea186d7-b2b1-4d9d-bf52-970af0da953f-kube-api-access-d2c58" (OuterVolumeSpecName: "kube-api-access-d2c58") pod "6ea186d7-b2b1-4d9d-bf52-970af0da953f" (UID: "6ea186d7-b2b1-4d9d-bf52-970af0da953f"). InnerVolumeSpecName "kube-api-access-d2c58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:04:42.799770 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:42.799675    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ea186d7-b2b1-4d9d-bf52-970af0da953f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6ea186d7-b2b1-4d9d-bf52-970af0da953f" (UID: "6ea186d7-b2b1-4d9d-bf52-970af0da953f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:04:42.898867 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:42.898827    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ea186d7-b2b1-4d9d-bf52-970af0da953f-proxy-tls\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:04:42.898867 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:42.898860    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ea186d7-b2b1-4d9d-bf52-970af0da953f-kserve-provision-location\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:04:42.898867 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:42.898877    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ea186d7-b2b1-4d9d-bf52-970af0da953f-cabundle-cert\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:04:42.899117 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:42.898889    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ea186d7-b2b1-4d9d-bf52-970af0da953f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:04:42.899117 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:42.898900    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2c58\" (UniqueName: \"kubernetes.io/projected/6ea186d7-b2b1-4d9d-bf52-970af0da953f-kube-api-access-d2c58\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:04:43.501109 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:43.501013    2573 generic.go:356] "Generic (PLEG): container finished" podID="6ea186d7-b2b1-4d9d-bf52-970af0da953f" containerID="8b920b150df382abe9d651d0b9e92d95dc8158a34d387cc36a504ef9daeff798" exitCode=0
Jul 27 18:04:43.501109 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:43.501079    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-pq9hv"
Jul 27 18:04:43.501657 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:43.501106    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-pq9hv" event={"ID":"6ea186d7-b2b1-4d9d-bf52-970af0da953f","Type":"ContainerDied","Data":"8b920b150df382abe9d651d0b9e92d95dc8158a34d387cc36a504ef9daeff798"}
Jul 27 18:04:43.501657 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:43.501167    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-pq9hv" event={"ID":"6ea186d7-b2b1-4d9d-bf52-970af0da953f","Type":"ContainerDied","Data":"0d2a0a810b4a941733ae440748eaed4f6f3d645c725f24448eed2d554083d82a"}
Jul 27 18:04:43.501657 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:43.501194    2573 scope.go:117] "RemoveContainer" containerID="9c722ea6202c93abd99596bba6b82a28f8bf4163de1860ccd043f2c84597a9b7"
Jul 27 18:04:43.501657 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:43.501403    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-xd7r4"
Jul 27 18:04:43.511093 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:43.511074    2573 scope.go:117] "RemoveContainer" containerID="8b920b150df382abe9d651d0b9e92d95dc8158a34d387cc36a504ef9daeff798"
Jul 27 18:04:43.519113 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:43.519096    2573 scope.go:117] "RemoveContainer" containerID="2360c0ac4a5daaf50f7652fbd7b89fd926486b91e49a88f3f040dc6041cf3bed"
Jul 27 18:04:43.526527 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:43.526510    2573 scope.go:117] "RemoveContainer" containerID="9c722ea6202c93abd99596bba6b82a28f8bf4163de1860ccd043f2c84597a9b7"
Jul 27 18:04:43.526789 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:04:43.526770    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c722ea6202c93abd99596bba6b82a28f8bf4163de1860ccd043f2c84597a9b7\": container with ID starting with 9c722ea6202c93abd99596bba6b82a28f8bf4163de1860ccd043f2c84597a9b7 not found: ID does not exist" containerID="9c722ea6202c93abd99596bba6b82a28f8bf4163de1860ccd043f2c84597a9b7"
Jul 27 18:04:43.526852 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:43.526797    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c722ea6202c93abd99596bba6b82a28f8bf4163de1860ccd043f2c84597a9b7"} err="failed to get container status \"9c722ea6202c93abd99596bba6b82a28f8bf4163de1860ccd043f2c84597a9b7\": rpc error: code = NotFound desc = could not find container \"9c722ea6202c93abd99596bba6b82a28f8bf4163de1860ccd043f2c84597a9b7\": container with ID starting with 9c722ea6202c93abd99596bba6b82a28f8bf4163de1860ccd043f2c84597a9b7 not found: ID does not exist"
Jul 27 18:04:43.526852 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:43.526813    2573 scope.go:117] "RemoveContainer" containerID="8b920b150df382abe9d651d0b9e92d95dc8158a34d387cc36a504ef9daeff798"
Jul 27 18:04:43.527089 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:04:43.527068    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b920b150df382abe9d651d0b9e92d95dc8158a34d387cc36a504ef9daeff798\": container with ID starting with 8b920b150df382abe9d651d0b9e92d95dc8158a34d387cc36a504ef9daeff798 not found: ID does not exist" containerID="8b920b150df382abe9d651d0b9e92d95dc8158a34d387cc36a504ef9daeff798"
Jul 27 18:04:43.527144 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:43.527094    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b920b150df382abe9d651d0b9e92d95dc8158a34d387cc36a504ef9daeff798"} err="failed to get container status \"8b920b150df382abe9d651d0b9e92d95dc8158a34d387cc36a504ef9daeff798\": rpc error: code = NotFound desc = could not find container \"8b920b150df382abe9d651d0b9e92d95dc8158a34d387cc36a504ef9daeff798\": container with ID starting with 8b920b150df382abe9d651d0b9e92d95dc8158a34d387cc36a504ef9daeff798 not found: ID does not exist"
Jul 27 18:04:43.527144 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:43.527109    2573 scope.go:117] "RemoveContainer" containerID="2360c0ac4a5daaf50f7652fbd7b89fd926486b91e49a88f3f040dc6041cf3bed"
Jul 27 18:04:43.527231 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:43.527140    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-xd7r4"]
Jul 27 18:04:43.527352 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:04:43.527333    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2360c0ac4a5daaf50f7652fbd7b89fd926486b91e49a88f3f040dc6041cf3bed\": container with ID starting with 2360c0ac4a5daaf50f7652fbd7b89fd926486b91e49a88f3f040dc6041cf3bed not found: ID does not exist" containerID="2360c0ac4a5daaf50f7652fbd7b89fd926486b91e49a88f3f040dc6041cf3bed"
Jul 27 18:04:43.527400 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:43.527351    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2360c0ac4a5daaf50f7652fbd7b89fd926486b91e49a88f3f040dc6041cf3bed"} err="failed to get container status \"2360c0ac4a5daaf50f7652fbd7b89fd926486b91e49a88f3f040dc6041cf3bed\": rpc error: code = NotFound desc = could not find container \"2360c0ac4a5daaf50f7652fbd7b89fd926486b91e49a88f3f040dc6041cf3bed\": container with ID starting with 2360c0ac4a5daaf50f7652fbd7b89fd926486b91e49a88f3f040dc6041cf3bed not found: ID does not exist"
Jul 27 18:04:43.532707 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:43.532686    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-xd7r4"]
Jul 27 18:04:43.545024 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:43.545005    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-pq9hv"]
Jul 27 18:04:43.547747 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:43.547728    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-77f778d864-pq9hv"]
Jul 27 18:04:44.468475 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:44.468440    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="410dfec0-91d8-4333-bc35-e7b19a1e314b" path="/var/lib/kubelet/pods/410dfec0-91d8-4333-bc35-e7b19a1e314b/volumes"
Jul 27 18:04:44.468936 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:44.468923    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ea186d7-b2b1-4d9d-bf52-970af0da953f" path="/var/lib/kubelet/pods/6ea186d7-b2b1-4d9d-bf52-970af0da953f/volumes"
Jul 27 18:04:47.962148 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:47.962111    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9hrmm"]
Jul 27 18:04:47.962550 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:47.962433    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="410dfec0-91d8-4333-bc35-e7b19a1e314b" containerName="storage-initializer"
Jul 27 18:04:47.962550 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:47.962445    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="410dfec0-91d8-4333-bc35-e7b19a1e314b" containerName="storage-initializer"
Jul 27 18:04:47.962550 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:47.962462    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ea186d7-b2b1-4d9d-bf52-970af0da953f" containerName="storage-initializer"
Jul 27 18:04:47.962550 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:47.962468    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ea186d7-b2b1-4d9d-bf52-970af0da953f" containerName="storage-initializer"
Jul 27 18:04:47.962550 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:47.962482    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ea186d7-b2b1-4d9d-bf52-970af0da953f" containerName="kserve-container"
Jul 27 18:04:47.962550 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:47.962488    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ea186d7-b2b1-4d9d-bf52-970af0da953f" containerName="kserve-container"
Jul 27 18:04:47.962550 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:47.962495    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="410dfec0-91d8-4333-bc35-e7b19a1e314b" containerName="kserve-container"
Jul 27 18:04:47.962550 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:47.962500    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="410dfec0-91d8-4333-bc35-e7b19a1e314b" containerName="kserve-container"
Jul 27 18:04:47.962550 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:47.962509    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="410dfec0-91d8-4333-bc35-e7b19a1e314b" containerName="kube-rbac-proxy"
Jul 27 18:04:47.962550 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:47.962515    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="410dfec0-91d8-4333-bc35-e7b19a1e314b" containerName="kube-rbac-proxy"
Jul 27 18:04:47.962550 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:47.962521    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ea186d7-b2b1-4d9d-bf52-970af0da953f" containerName="kube-rbac-proxy"
Jul 27 18:04:47.962550 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:47.962526    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ea186d7-b2b1-4d9d-bf52-970af0da953f" containerName="kube-rbac-proxy"
Jul 27 18:04:47.962923 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:47.962573    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ea186d7-b2b1-4d9d-bf52-970af0da953f" containerName="kube-rbac-proxy"
Jul 27 18:04:47.962923 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:47.962581    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="410dfec0-91d8-4333-bc35-e7b19a1e314b" containerName="kube-rbac-proxy"
Jul 27 18:04:47.962923 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:47.962587    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="410dfec0-91d8-4333-bc35-e7b19a1e314b" containerName="kserve-container"
Jul 27 18:04:47.962923 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:47.962595    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ea186d7-b2b1-4d9d-bf52-970af0da953f" containerName="kserve-container"
Jul 27 18:04:47.967035 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:47.967018    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9hrmm"
Jul 27 18:04:47.969994 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:47.969954    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-predictor-serving-cert\""
Jul 27 18:04:47.970126 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:47.970095    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z6ld7\""
Jul 27 18:04:47.970126 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:47.969978    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 18:04:47.970284 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:47.969972    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 18:04:47.971390 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:47.971368    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 18:04:47.971528 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:47.971446    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
Jul 27 18:04:47.971528 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:47.971453    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 18:04:47.974384 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:47.974364    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9hrmm"]
Jul 27 18:04:47.993712 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:47.993688    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:04:48.143797 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:48.143764    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a4373e9-5b32-412b-8111-877ebd3c8550-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-9hrmm\" (UID: \"4a4373e9-5b32-412b-8111-877ebd3c8550\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9hrmm"
Jul 27 18:04:48.143797 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:48.143801    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6txf\" (UniqueName: \"kubernetes.io/projected/4a4373e9-5b32-412b-8111-877ebd3c8550-kube-api-access-b6txf\") pod \"isvc-lightgbm-predictor-54f85ddf88-9hrmm\" (UID: \"4a4373e9-5b32-412b-8111-877ebd3c8550\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9hrmm"
Jul 27 18:04:48.144005 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:48.143819    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a4373e9-5b32-412b-8111-877ebd3c8550-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-9hrmm\" (UID: \"4a4373e9-5b32-412b-8111-877ebd3c8550\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9hrmm"
Jul 27 18:04:48.144005 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:48.143838    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a4373e9-5b32-412b-8111-877ebd3c8550-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-9hrmm\" (UID: \"4a4373e9-5b32-412b-8111-877ebd3c8550\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9hrmm"
Jul 27 18:04:48.144005 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:48.143884    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a4373e9-5b32-412b-8111-877ebd3c8550-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-9hrmm\" (UID: \"4a4373e9-5b32-412b-8111-877ebd3c8550\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9hrmm"
Jul 27 18:04:48.214726 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:48.214639    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:04:48.244693 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:48.244669    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a4373e9-5b32-412b-8111-877ebd3c8550-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-9hrmm\" (UID: \"4a4373e9-5b32-412b-8111-877ebd3c8550\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9hrmm"
Jul 27 18:04:48.244810 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:48.244701    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6txf\" (UniqueName: \"kubernetes.io/projected/4a4373e9-5b32-412b-8111-877ebd3c8550-kube-api-access-b6txf\") pod \"isvc-lightgbm-predictor-54f85ddf88-9hrmm\" (UID: \"4a4373e9-5b32-412b-8111-877ebd3c8550\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9hrmm"
Jul 27 18:04:48.244810 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:48.244721    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a4373e9-5b32-412b-8111-877ebd3c8550-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-9hrmm\" (UID: \"4a4373e9-5b32-412b-8111-877ebd3c8550\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9hrmm"
Jul 27 18:04:48.244889 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:48.244827    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a4373e9-5b32-412b-8111-877ebd3c8550-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-9hrmm\" (UID: \"4a4373e9-5b32-412b-8111-877ebd3c8550\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9hrmm"
Jul 27 18:04:48.244926 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:48.244911    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a4373e9-5b32-412b-8111-877ebd3c8550-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-9hrmm\" (UID: \"4a4373e9-5b32-412b-8111-877ebd3c8550\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9hrmm"
Jul 27 18:04:48.245308 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:48.245280    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a4373e9-5b32-412b-8111-877ebd3c8550-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-9hrmm\" (UID: \"4a4373e9-5b32-412b-8111-877ebd3c8550\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9hrmm"
Jul 27 18:04:48.245485 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:48.245468    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a4373e9-5b32-412b-8111-877ebd3c8550-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-9hrmm\" (UID: \"4a4373e9-5b32-412b-8111-877ebd3c8550\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9hrmm"
Jul 27 18:04:48.245535 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:48.245512    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a4373e9-5b32-412b-8111-877ebd3c8550-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-9hrmm\" (UID: \"4a4373e9-5b32-412b-8111-877ebd3c8550\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9hrmm"
Jul 27 18:04:48.247257 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:48.247224    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a4373e9-5b32-412b-8111-877ebd3c8550-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-9hrmm\" (UID: \"4a4373e9-5b32-412b-8111-877ebd3c8550\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9hrmm"
Jul 27 18:04:48.253759 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:48.253728    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6txf\" (UniqueName: \"kubernetes.io/projected/4a4373e9-5b32-412b-8111-877ebd3c8550-kube-api-access-b6txf\") pod \"isvc-lightgbm-predictor-54f85ddf88-9hrmm\" (UID: \"4a4373e9-5b32-412b-8111-877ebd3c8550\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9hrmm"
Jul 27 18:04:48.278077 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:48.278057    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9hrmm"
Jul 27 18:04:48.406793 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:48.406759    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9hrmm"]
Jul 27 18:04:48.410419 ip-10-0-140-0 kubenswrapper[2573]: W0727 18:04:48.410387    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a4373e9_5b32_412b_8111_877ebd3c8550.slice/crio-8ad0d20eaa5e3f57b26e4ee39906d8b45ad3fd854a4eb5eb8cb62254ea000071 WatchSource:0}: Error finding container 8ad0d20eaa5e3f57b26e4ee39906d8b45ad3fd854a4eb5eb8cb62254ea000071: Status 404 returned error can't find the container with id 8ad0d20eaa5e3f57b26e4ee39906d8b45ad3fd854a4eb5eb8cb62254ea000071
Jul 27 18:04:48.524891 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:48.524857    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9hrmm" event={"ID":"4a4373e9-5b32-412b-8111-877ebd3c8550","Type":"ContainerStarted","Data":"fcd15d68366f250f2e2b6677008fc039be2bf0fe23b2b36b11a5af5a965e60e2"}
Jul 27 18:04:48.525051 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:48.524919    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9hrmm" event={"ID":"4a4373e9-5b32-412b-8111-877ebd3c8550","Type":"ContainerStarted","Data":"8ad0d20eaa5e3f57b26e4ee39906d8b45ad3fd854a4eb5eb8cb62254ea000071"}
Jul 27 18:04:52.544495 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:52.544406    2573 generic.go:356] "Generic (PLEG): container finished" podID="4a4373e9-5b32-412b-8111-877ebd3c8550" containerID="fcd15d68366f250f2e2b6677008fc039be2bf0fe23b2b36b11a5af5a965e60e2" exitCode=0
Jul 27 18:04:52.544495 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:52.544463    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9hrmm" event={"ID":"4a4373e9-5b32-412b-8111-877ebd3c8550","Type":"ContainerDied","Data":"fcd15d68366f250f2e2b6677008fc039be2bf0fe23b2b36b11a5af5a965e60e2"}
Jul 27 18:04:52.545733 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:52.545712    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 18:04:59.583957 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:59.583920    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9hrmm" event={"ID":"4a4373e9-5b32-412b-8111-877ebd3c8550","Type":"ContainerStarted","Data":"916fedb42077a3db901e51f2cc1eb91d049c78bd97881b373f0b6f36a9c2302d"}
Jul 27 18:04:59.584452 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:59.584014    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9hrmm" event={"ID":"4a4373e9-5b32-412b-8111-877ebd3c8550","Type":"ContainerStarted","Data":"6d2f69cabdc8c9f121b043f4b2be6ea1d662f0620c10b68a452fb86784530bb6"}
Jul 27 18:04:59.584452 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:59.584320    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9hrmm"
Jul 27 18:04:59.603323 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:04:59.603269    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9hrmm" podStartSLOduration=2.663038556 podStartE2EDuration="12.603235231s" totalImagesPullingTime="6.261447885s" totalInitContainerRuntime="3.67874879s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:04:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:04:59.601969328 +0000 UTC m=+2241.745371380" watchObservedRunningTime="2026-07-27 18:04:59.603235231 +0000 UTC m=+2241.746637282"
Jul 27 18:05:00.588408 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:05:00.588379    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9hrmm"
Jul 27 18:05:00.589544 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:05:00.589510    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9hrmm" podUID="4a4373e9-5b32-412b-8111-877ebd3c8550" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 27 18:05:01.592739 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:05:01.592698    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9hrmm" podUID="4a4373e9-5b32-412b-8111-877ebd3c8550" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 27 18:05:06.597219 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:05:06.597180    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9hrmm"
Jul 27 18:05:06.597712 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:05:06.597680    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9hrmm" podUID="4a4373e9-5b32-412b-8111-877ebd3c8550" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 27 18:05:16.598530 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:05:16.598485    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9hrmm" podUID="4a4373e9-5b32-412b-8111-877ebd3c8550" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 27 18:05:26.597771 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:05:26.597733    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9hrmm" podUID="4a4373e9-5b32-412b-8111-877ebd3c8550" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 27 18:05:36.598277 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:05:36.598213    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9hrmm" podUID="4a4373e9-5b32-412b-8111-877ebd3c8550" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 27 18:05:46.597757 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:05:46.597709    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9hrmm" podUID="4a4373e9-5b32-412b-8111-877ebd3c8550" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 27 18:05:56.598469 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:05:56.598431    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9hrmm" podUID="4a4373e9-5b32-412b-8111-877ebd3c8550" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 27 18:06:06.598375 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:06.598334    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9hrmm" podUID="4a4373e9-5b32-412b-8111-877ebd3c8550" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 27 18:06:10.464523 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:10.464479    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9hrmm" podUID="4a4373e9-5b32-412b-8111-877ebd3c8550" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 27 18:06:12.751587 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:12.751545    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:06:20.469216 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:20.469185    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9hrmm"
Jul 27 18:06:28.072795 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:28.072762    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9hrmm"]
Jul 27 18:06:28.073278 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:28.073002    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9hrmm" podUID="4a4373e9-5b32-412b-8111-877ebd3c8550" containerName="kserve-container" containerID="cri-o://6d2f69cabdc8c9f121b043f4b2be6ea1d662f0620c10b68a452fb86784530bb6" gracePeriod=30
Jul 27 18:06:28.073278 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:28.073064    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9hrmm" podUID="4a4373e9-5b32-412b-8111-877ebd3c8550" containerName="kube-rbac-proxy" containerID="cri-o://916fedb42077a3db901e51f2cc1eb91d049c78bd97881b373f0b6f36a9c2302d" gracePeriod=30
Jul 27 18:06:28.092912 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:28.092879    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:06:28.412492 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:28.412459    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:06:28.987979 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:28.987941    2573 generic.go:356] "Generic (PLEG): container finished" podID="4a4373e9-5b32-412b-8111-877ebd3c8550" containerID="916fedb42077a3db901e51f2cc1eb91d049c78bd97881b373f0b6f36a9c2302d" exitCode=2
Jul 27 18:06:28.988160 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:28.988015    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9hrmm" event={"ID":"4a4373e9-5b32-412b-8111-877ebd3c8550","Type":"ContainerDied","Data":"916fedb42077a3db901e51f2cc1eb91d049c78bd97881b373f0b6f36a9c2302d"}
Jul 27 18:06:30.465124 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:30.465019    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9hrmm" podUID="4a4373e9-5b32-412b-8111-877ebd3c8550" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 27 18:06:31.593073 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:31.593031    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9hrmm" podUID="4a4373e9-5b32-412b-8111-877ebd3c8550" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.46:8643/healthz\": dial tcp 10.133.0.46:8643: connect: connection refused"
Jul 27 18:06:32.826807 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:32.826779    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9hrmm"
Jul 27 18:06:32.927662 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:32.927625    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a4373e9-5b32-412b-8111-877ebd3c8550-proxy-tls\") pod \"4a4373e9-5b32-412b-8111-877ebd3c8550\" (UID: \"4a4373e9-5b32-412b-8111-877ebd3c8550\") "
Jul 27 18:06:32.927662 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:32.927673    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a4373e9-5b32-412b-8111-877ebd3c8550-kube-rbac-proxy-sar-config\") pod \"4a4373e9-5b32-412b-8111-877ebd3c8550\" (UID: \"4a4373e9-5b32-412b-8111-877ebd3c8550\") "
Jul 27 18:06:32.927897 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:32.927699    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6txf\" (UniqueName: \"kubernetes.io/projected/4a4373e9-5b32-412b-8111-877ebd3c8550-kube-api-access-b6txf\") pod \"4a4373e9-5b32-412b-8111-877ebd3c8550\" (UID: \"4a4373e9-5b32-412b-8111-877ebd3c8550\") "
Jul 27 18:06:32.927897 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:32.927753    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a4373e9-5b32-412b-8111-877ebd3c8550-kserve-provision-location\") pod \"4a4373e9-5b32-412b-8111-877ebd3c8550\" (UID: \"4a4373e9-5b32-412b-8111-877ebd3c8550\") "
Jul 27 18:06:32.927897 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:32.927789    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a4373e9-5b32-412b-8111-877ebd3c8550-cabundle-cert\") pod \"4a4373e9-5b32-412b-8111-877ebd3c8550\" (UID: \"4a4373e9-5b32-412b-8111-877ebd3c8550\") "
Jul 27 18:06:32.928126 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:32.928035    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a4373e9-5b32-412b-8111-877ebd3c8550-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4a4373e9-5b32-412b-8111-877ebd3c8550" (UID: "4a4373e9-5b32-412b-8111-877ebd3c8550"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:06:32.928217 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:32.928096    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a4373e9-5b32-412b-8111-877ebd3c8550-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4a4373e9-5b32-412b-8111-877ebd3c8550" (UID: "4a4373e9-5b32-412b-8111-877ebd3c8550"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:06:32.928304 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:32.928222    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a4373e9-5b32-412b-8111-877ebd3c8550-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4a4373e9-5b32-412b-8111-877ebd3c8550" (UID: "4a4373e9-5b32-412b-8111-877ebd3c8550"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:06:32.929848 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:32.929826    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a4373e9-5b32-412b-8111-877ebd3c8550-kube-api-access-b6txf" (OuterVolumeSpecName: "kube-api-access-b6txf") pod "4a4373e9-5b32-412b-8111-877ebd3c8550" (UID: "4a4373e9-5b32-412b-8111-877ebd3c8550"). InnerVolumeSpecName "kube-api-access-b6txf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:06:32.929848 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:32.929834    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a4373e9-5b32-412b-8111-877ebd3c8550-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4a4373e9-5b32-412b-8111-877ebd3c8550" (UID: "4a4373e9-5b32-412b-8111-877ebd3c8550"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:06:33.008240 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:33.008204    2573 generic.go:356] "Generic (PLEG): container finished" podID="4a4373e9-5b32-412b-8111-877ebd3c8550" containerID="6d2f69cabdc8c9f121b043f4b2be6ea1d662f0620c10b68a452fb86784530bb6" exitCode=0
Jul 27 18:06:33.008425 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:33.008296    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9hrmm"
Jul 27 18:06:33.008425 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:33.008291    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9hrmm" event={"ID":"4a4373e9-5b32-412b-8111-877ebd3c8550","Type":"ContainerDied","Data":"6d2f69cabdc8c9f121b043f4b2be6ea1d662f0620c10b68a452fb86784530bb6"}
Jul 27 18:06:33.008514 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:33.008442    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9hrmm" event={"ID":"4a4373e9-5b32-412b-8111-877ebd3c8550","Type":"ContainerDied","Data":"8ad0d20eaa5e3f57b26e4ee39906d8b45ad3fd854a4eb5eb8cb62254ea000071"}
Jul 27 18:06:33.008514 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:33.008470    2573 scope.go:117] "RemoveContainer" containerID="916fedb42077a3db901e51f2cc1eb91d049c78bd97881b373f0b6f36a9c2302d"
Jul 27 18:06:33.017661 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:33.017643    2573 scope.go:117] "RemoveContainer" containerID="6d2f69cabdc8c9f121b043f4b2be6ea1d662f0620c10b68a452fb86784530bb6"
Jul 27 18:06:33.025191 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:33.025174    2573 scope.go:117] "RemoveContainer" containerID="fcd15d68366f250f2e2b6677008fc039be2bf0fe23b2b36b11a5af5a965e60e2"
Jul 27 18:06:33.028381 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:33.028350    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a4373e9-5b32-412b-8111-877ebd3c8550-cabundle-cert\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:06:33.028487 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:33.028383    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a4373e9-5b32-412b-8111-877ebd3c8550-proxy-tls\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:06:33.028487 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:33.028398    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a4373e9-5b32-412b-8111-877ebd3c8550-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:06:33.028487 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:33.028411    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6txf\" (UniqueName: \"kubernetes.io/projected/4a4373e9-5b32-412b-8111-877ebd3c8550-kube-api-access-b6txf\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:06:33.028487 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:33.028423    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a4373e9-5b32-412b-8111-877ebd3c8550-kserve-provision-location\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:06:33.031568 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:33.031543    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9hrmm"]
Jul 27 18:06:33.033914 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:33.033896    2573 scope.go:117] "RemoveContainer" containerID="916fedb42077a3db901e51f2cc1eb91d049c78bd97881b373f0b6f36a9c2302d"
Jul 27 18:06:33.034194 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:06:33.034176    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"916fedb42077a3db901e51f2cc1eb91d049c78bd97881b373f0b6f36a9c2302d\": container with ID starting with 916fedb42077a3db901e51f2cc1eb91d049c78bd97881b373f0b6f36a9c2302d not found: ID does not exist" containerID="916fedb42077a3db901e51f2cc1eb91d049c78bd97881b373f0b6f36a9c2302d"
Jul 27 18:06:33.034281 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:33.034201    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"916fedb42077a3db901e51f2cc1eb91d049c78bd97881b373f0b6f36a9c2302d"} err="failed to get container status \"916fedb42077a3db901e51f2cc1eb91d049c78bd97881b373f0b6f36a9c2302d\": rpc error: code = NotFound desc = could not find container \"916fedb42077a3db901e51f2cc1eb91d049c78bd97881b373f0b6f36a9c2302d\": container with ID starting with 916fedb42077a3db901e51f2cc1eb91d049c78bd97881b373f0b6f36a9c2302d not found: ID does not exist"
Jul 27 18:06:33.034281 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:33.034220    2573 scope.go:117] "RemoveContainer" containerID="6d2f69cabdc8c9f121b043f4b2be6ea1d662f0620c10b68a452fb86784530bb6"
Jul 27 18:06:33.034476 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:06:33.034462    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d2f69cabdc8c9f121b043f4b2be6ea1d662f0620c10b68a452fb86784530bb6\": container with ID starting with 6d2f69cabdc8c9f121b043f4b2be6ea1d662f0620c10b68a452fb86784530bb6 not found: ID does not exist" containerID="6d2f69cabdc8c9f121b043f4b2be6ea1d662f0620c10b68a452fb86784530bb6"
Jul 27 18:06:33.034528 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:33.034480    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d2f69cabdc8c9f121b043f4b2be6ea1d662f0620c10b68a452fb86784530bb6"} err="failed to get container status \"6d2f69cabdc8c9f121b043f4b2be6ea1d662f0620c10b68a452fb86784530bb6\": rpc error: code = NotFound desc = could not find container \"6d2f69cabdc8c9f121b043f4b2be6ea1d662f0620c10b68a452fb86784530bb6\": container with ID starting with 6d2f69cabdc8c9f121b043f4b2be6ea1d662f0620c10b68a452fb86784530bb6 not found: ID does not exist"
Jul 27 18:06:33.034528 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:33.034501    2573 scope.go:117] "RemoveContainer" containerID="fcd15d68366f250f2e2b6677008fc039be2bf0fe23b2b36b11a5af5a965e60e2"
Jul 27 18:06:33.034736 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:06:33.034720    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcd15d68366f250f2e2b6677008fc039be2bf0fe23b2b36b11a5af5a965e60e2\": container with ID starting with fcd15d68366f250f2e2b6677008fc039be2bf0fe23b2b36b11a5af5a965e60e2 not found: ID does not exist" containerID="fcd15d68366f250f2e2b6677008fc039be2bf0fe23b2b36b11a5af5a965e60e2"
Jul 27 18:06:33.034774 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:33.034738    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcd15d68366f250f2e2b6677008fc039be2bf0fe23b2b36b11a5af5a965e60e2"} err="failed to get container status \"fcd15d68366f250f2e2b6677008fc039be2bf0fe23b2b36b11a5af5a965e60e2\": rpc error: code = NotFound desc = could not find container \"fcd15d68366f250f2e2b6677008fc039be2bf0fe23b2b36b11a5af5a965e60e2\": container with ID starting with fcd15d68366f250f2e2b6677008fc039be2bf0fe23b2b36b11a5af5a965e60e2 not found: ID does not exist"
Jul 27 18:06:33.035129 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:33.035110    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-9hrmm"]
Jul 27 18:06:33.205458 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:33.205352    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:06:33.236716 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:33.236682    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh"]
Jul 27 18:06:33.237007 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:33.236995    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a4373e9-5b32-412b-8111-877ebd3c8550" containerName="storage-initializer"
Jul 27 18:06:33.237050 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:33.237009    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a4373e9-5b32-412b-8111-877ebd3c8550" containerName="storage-initializer"
Jul 27 18:06:33.237050 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:33.237024    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a4373e9-5b32-412b-8111-877ebd3c8550" containerName="kserve-container"
Jul 27 18:06:33.237050 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:33.237029    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a4373e9-5b32-412b-8111-877ebd3c8550" containerName="kserve-container"
Jul 27 18:06:33.237050 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:33.237043    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a4373e9-5b32-412b-8111-877ebd3c8550" containerName="kube-rbac-proxy"
Jul 27 18:06:33.237050 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:33.237049    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a4373e9-5b32-412b-8111-877ebd3c8550" containerName="kube-rbac-proxy"
Jul 27 18:06:33.237205 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:33.237096    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a4373e9-5b32-412b-8111-877ebd3c8550" containerName="kserve-container"
Jul 27 18:06:33.237205 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:33.237106    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a4373e9-5b32-412b-8111-877ebd3c8550" containerName="kube-rbac-proxy"
Jul 27 18:06:33.241555 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:33.241539    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh"
Jul 27 18:06:33.244580 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:33.244554    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 18:06:33.244707 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:33.244609    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
Jul 27 18:06:33.244707 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:33.244673    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z6ld7\""
Jul 27 18:06:33.244831 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:33.244792    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 18:06:33.244831 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:33.244791    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
Jul 27 18:06:33.244831 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:33.244820    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 18:06:33.244948 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:33.244895    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 18:06:33.250088 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:33.250068    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh"]
Jul 27 18:06:33.331599 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:33.331559    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwrfb\" (UniqueName: \"kubernetes.io/projected/40e489de-e909-42a7-bb78-a6b908f6f19b-kube-api-access-dwrfb\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh\" (UID: \"40e489de-e909-42a7-bb78-a6b908f6f19b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh"
Jul 27 18:06:33.331798 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:33.331637    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40e489de-e909-42a7-bb78-a6b908f6f19b-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh\" (UID: \"40e489de-e909-42a7-bb78-a6b908f6f19b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh"
Jul 27 18:06:33.331798 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:33.331667    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40e489de-e909-42a7-bb78-a6b908f6f19b-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh\" (UID: \"40e489de-e909-42a7-bb78-a6b908f6f19b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh"
Jul 27 18:06:33.331798 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:33.331684    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40e489de-e909-42a7-bb78-a6b908f6f19b-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh\" (UID: \"40e489de-e909-42a7-bb78-a6b908f6f19b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh"
Jul 27 18:06:33.331951 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:33.331798    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40e489de-e909-42a7-bb78-a6b908f6f19b-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh\" (UID: \"40e489de-e909-42a7-bb78-a6b908f6f19b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh"
Jul 27 18:06:33.354409 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:33.354375    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:06:33.433099 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:33.433059    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40e489de-e909-42a7-bb78-a6b908f6f19b-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh\" (UID: \"40e489de-e909-42a7-bb78-a6b908f6f19b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh"
Jul 27 18:06:33.433099 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:33.433112    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwrfb\" (UniqueName: \"kubernetes.io/projected/40e489de-e909-42a7-bb78-a6b908f6f19b-kube-api-access-dwrfb\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh\" (UID: \"40e489de-e909-42a7-bb78-a6b908f6f19b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh"
Jul 27 18:06:33.433387 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:33.433165    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40e489de-e909-42a7-bb78-a6b908f6f19b-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh\" (UID: \"40e489de-e909-42a7-bb78-a6b908f6f19b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh"
Jul 27 18:06:33.433387 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:33.433188    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40e489de-e909-42a7-bb78-a6b908f6f19b-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh\" (UID: \"40e489de-e909-42a7-bb78-a6b908f6f19b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh"
Jul 27 18:06:33.433387 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:33.433209    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40e489de-e909-42a7-bb78-a6b908f6f19b-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh\" (UID: \"40e489de-e909-42a7-bb78-a6b908f6f19b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh"
Jul 27 18:06:33.433740 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:33.433716    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40e489de-e909-42a7-bb78-a6b908f6f19b-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh\" (UID: \"40e489de-e909-42a7-bb78-a6b908f6f19b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh"
Jul 27 18:06:33.433922 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:33.433902    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40e489de-e909-42a7-bb78-a6b908f6f19b-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh\" (UID: \"40e489de-e909-42a7-bb78-a6b908f6f19b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh"
Jul 27 18:06:33.433989 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:33.433943    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40e489de-e909-42a7-bb78-a6b908f6f19b-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh\" (UID: \"40e489de-e909-42a7-bb78-a6b908f6f19b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh"
Jul 27 18:06:33.435693 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:33.435671    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40e489de-e909-42a7-bb78-a6b908f6f19b-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh\" (UID: \"40e489de-e909-42a7-bb78-a6b908f6f19b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh"
Jul 27 18:06:33.443415 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:33.443387    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwrfb\" (UniqueName: \"kubernetes.io/projected/40e489de-e909-42a7-bb78-a6b908f6f19b-kube-api-access-dwrfb\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh\" (UID: \"40e489de-e909-42a7-bb78-a6b908f6f19b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh"
Jul 27 18:06:33.552040 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:33.551953    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh"
Jul 27 18:06:33.690716 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:33.690510    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh"]
Jul 27 18:06:33.693295 ip-10-0-140-0 kubenswrapper[2573]: W0727 18:06:33.693262    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e489de_e909_42a7_bb78_a6b908f6f19b.slice/crio-26b7b9da561c375057aeb05a5750caddc5cbbd98454f07540c1cccb10e26ad2e WatchSource:0}: Error finding container 26b7b9da561c375057aeb05a5750caddc5cbbd98454f07540c1cccb10e26ad2e: Status 404 returned error can't find the container with id 26b7b9da561c375057aeb05a5750caddc5cbbd98454f07540c1cccb10e26ad2e
Jul 27 18:06:34.015229 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:34.015195    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh" event={"ID":"40e489de-e909-42a7-bb78-a6b908f6f19b","Type":"ContainerStarted","Data":"9ed1463c48b5336dd181704f04ddfa6e72f33d8d78c05233b0b0b8305236fa90"}
Jul 27 18:06:34.015607 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:34.015268    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh" event={"ID":"40e489de-e909-42a7-bb78-a6b908f6f19b","Type":"ContainerStarted","Data":"26b7b9da561c375057aeb05a5750caddc5cbbd98454f07540c1cccb10e26ad2e"}
Jul 27 18:06:34.469649 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:34.469615    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a4373e9-5b32-412b-8111-877ebd3c8550" path="/var/lib/kubelet/pods/4a4373e9-5b32-412b-8111-877ebd3c8550/volumes"
Jul 27 18:06:38.035078 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:38.035044    2573 generic.go:356] "Generic (PLEG): container finished" podID="40e489de-e909-42a7-bb78-a6b908f6f19b" containerID="9ed1463c48b5336dd181704f04ddfa6e72f33d8d78c05233b0b0b8305236fa90" exitCode=0
Jul 27 18:06:38.035522 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:38.035116    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh" event={"ID":"40e489de-e909-42a7-bb78-a6b908f6f19b","Type":"ContainerDied","Data":"9ed1463c48b5336dd181704f04ddfa6e72f33d8d78c05233b0b0b8305236fa90"}
Jul 27 18:06:38.424515 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:38.424480    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:06:38.570033 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:38.570000    2573 scope.go:117] "RemoveContainer" containerID="6a6b4bb490126bf1947c8626742605d199be62ee0d0392d07cd676ed6ff07962"
Jul 27 18:06:38.578284 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:38.578261    2573 scope.go:117] "RemoveContainer" containerID="4189bee25127481114052277360bb12c93d2e608cfb1992194d8b06f01ba95c0"
Jul 27 18:06:38.585514 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:38.585495    2573 scope.go:117] "RemoveContainer" containerID="866546a58d91a9c8c1acd9b536109a3c848f390ffb3ce2f879d5af97d320fced"
Jul 27 18:06:39.041591 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:39.041555    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh" event={"ID":"40e489de-e909-42a7-bb78-a6b908f6f19b","Type":"ContainerStarted","Data":"06a4ff4bef14e5984de7fa5c1f44db6cb307de8da9cfdefdfcd86c0af884b925"}
Jul 27 18:06:39.042100 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:39.041628    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh" event={"ID":"40e489de-e909-42a7-bb78-a6b908f6f19b","Type":"ContainerStarted","Data":"a237dc50744b7fb93662b893c54192b91ca73ab73309f3bcd21f811443acdc87"}
Jul 27 18:06:39.042100 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:39.041851    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh"
Jul 27 18:06:39.061985 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:39.061936    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh" podStartSLOduration=2.452719705 podStartE2EDuration="6.06192097s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.609201265s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:06:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:06:39.060474625 +0000 UTC m=+2341.203876677" watchObservedRunningTime="2026-07-27 18:06:39.06192097 +0000 UTC m=+2341.205323019"
Jul 27 18:06:39.099368 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:39.099330    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:06:40.046159 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:40.046115    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh"
Jul 27 18:06:40.047480 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:40.047452    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh" podUID="40e489de-e909-42a7-bb78-a6b908f6f19b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 27 18:06:41.050709 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:41.050670    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh" podUID="40e489de-e909-42a7-bb78-a6b908f6f19b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 27 18:06:46.055872 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:46.055841    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh"
Jul 27 18:06:46.056435 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:46.056408    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh" podUID="40e489de-e909-42a7-bb78-a6b908f6f19b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 27 18:06:56.056651 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:06:56.056606    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh" podUID="40e489de-e909-42a7-bb78-a6b908f6f19b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 27 18:07:06.056712 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:07:06.056668    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh" podUID="40e489de-e909-42a7-bb78-a6b908f6f19b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 27 18:07:16.056368 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:07:16.056325    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh" podUID="40e489de-e909-42a7-bb78-a6b908f6f19b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 27 18:07:26.056337 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:07:26.056296    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh" podUID="40e489de-e909-42a7-bb78-a6b908f6f19b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 27 18:07:36.056439 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:07:36.056399    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh" podUID="40e489de-e909-42a7-bb78-a6b908f6f19b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 27 18:07:38.439958 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:07:38.439931    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vc2tz_87dac797-f8d2-4d36-999f-f8ce855dbeff/ovn-acl-logging/0.log"
Jul 27 18:07:38.445970 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:07:38.445951    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vc2tz_87dac797-f8d2-4d36-999f-f8ce855dbeff/ovn-acl-logging/0.log"
Jul 27 18:07:46.056584 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:07:46.056545    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh" podUID="40e489de-e909-42a7-bb78-a6b908f6f19b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 27 18:07:50.464064 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:07:50.464015    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh" podUID="40e489de-e909-42a7-bb78-a6b908f6f19b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 27 18:08:00.469470 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:00.469398    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh"
Jul 27 18:08:03.650326 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:03.650292    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh"]
Jul 27 18:08:03.650737 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:03.650575    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh" podUID="40e489de-e909-42a7-bb78-a6b908f6f19b" containerName="kserve-container" containerID="cri-o://a237dc50744b7fb93662b893c54192b91ca73ab73309f3bcd21f811443acdc87" gracePeriod=30
Jul 27 18:08:03.650737 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:03.650600    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh" podUID="40e489de-e909-42a7-bb78-a6b908f6f19b" containerName="kube-rbac-proxy" containerID="cri-o://06a4ff4bef14e5984de7fa5c1f44db6cb307de8da9cfdefdfcd86c0af884b925" gracePeriod=30
Jul 27 18:08:03.671497 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:03.671467    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:08:04.416104 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:04.416065    2573 generic.go:356] "Generic (PLEG): container finished" podID="40e489de-e909-42a7-bb78-a6b908f6f19b" containerID="06a4ff4bef14e5984de7fa5c1f44db6cb307de8da9cfdefdfcd86c0af884b925" exitCode=2
Jul 27 18:08:04.416302 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:04.416137    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh" event={"ID":"40e489de-e909-42a7-bb78-a6b908f6f19b","Type":"ContainerDied","Data":"06a4ff4bef14e5984de7fa5c1f44db6cb307de8da9cfdefdfcd86c0af884b925"}
Jul 27 18:08:06.051315 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:06.051269    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh" podUID="40e489de-e909-42a7-bb78-a6b908f6f19b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.47:8643/healthz\": dial tcp 10.133.0.47:8643: connect: connection refused"
Jul 27 18:08:08.438553 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:08.438506    2573 generic.go:356] "Generic (PLEG): container finished" podID="40e489de-e909-42a7-bb78-a6b908f6f19b" containerID="a237dc50744b7fb93662b893c54192b91ca73ab73309f3bcd21f811443acdc87" exitCode=0
Jul 27 18:08:08.439000 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:08.438561    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh" event={"ID":"40e489de-e909-42a7-bb78-a6b908f6f19b","Type":"ContainerDied","Data":"a237dc50744b7fb93662b893c54192b91ca73ab73309f3bcd21f811443acdc87"}
Jul 27 18:08:08.501094 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:08.501069    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh"
Jul 27 18:08:08.608279 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:08.608178    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwrfb\" (UniqueName: \"kubernetes.io/projected/40e489de-e909-42a7-bb78-a6b908f6f19b-kube-api-access-dwrfb\") pod \"40e489de-e909-42a7-bb78-a6b908f6f19b\" (UID: \"40e489de-e909-42a7-bb78-a6b908f6f19b\") "
Jul 27 18:08:08.608279 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:08.608233    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40e489de-e909-42a7-bb78-a6b908f6f19b-proxy-tls\") pod \"40e489de-e909-42a7-bb78-a6b908f6f19b\" (UID: \"40e489de-e909-42a7-bb78-a6b908f6f19b\") "
Jul 27 18:08:08.608594 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:08.608302    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40e489de-e909-42a7-bb78-a6b908f6f19b-kube-rbac-proxy-sar-config\") pod \"40e489de-e909-42a7-bb78-a6b908f6f19b\" (UID: \"40e489de-e909-42a7-bb78-a6b908f6f19b\") "
Jul 27 18:08:08.608594 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:08.608337    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40e489de-e909-42a7-bb78-a6b908f6f19b-cabundle-cert\") pod \"40e489de-e909-42a7-bb78-a6b908f6f19b\" (UID: \"40e489de-e909-42a7-bb78-a6b908f6f19b\") "
Jul 27 18:08:08.608594 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:08.608380    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40e489de-e909-42a7-bb78-a6b908f6f19b-kserve-provision-location\") pod \"40e489de-e909-42a7-bb78-a6b908f6f19b\" (UID: \"40e489de-e909-42a7-bb78-a6b908f6f19b\") "
Jul 27 18:08:08.608763 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:08.608732    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40e489de-e909-42a7-bb78-a6b908f6f19b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "40e489de-e909-42a7-bb78-a6b908f6f19b" (UID: "40e489de-e909-42a7-bb78-a6b908f6f19b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:08:08.608830 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:08.608737    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40e489de-e909-42a7-bb78-a6b908f6f19b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "40e489de-e909-42a7-bb78-a6b908f6f19b" (UID: "40e489de-e909-42a7-bb78-a6b908f6f19b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:08:08.608870 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:08.608834    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40e489de-e909-42a7-bb78-a6b908f6f19b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "40e489de-e909-42a7-bb78-a6b908f6f19b" (UID: "40e489de-e909-42a7-bb78-a6b908f6f19b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:08:08.610756 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:08.610734    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40e489de-e909-42a7-bb78-a6b908f6f19b-kube-api-access-dwrfb" (OuterVolumeSpecName: "kube-api-access-dwrfb") pod "40e489de-e909-42a7-bb78-a6b908f6f19b" (UID: "40e489de-e909-42a7-bb78-a6b908f6f19b"). InnerVolumeSpecName "kube-api-access-dwrfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:08:08.610858 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:08.610824    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40e489de-e909-42a7-bb78-a6b908f6f19b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "40e489de-e909-42a7-bb78-a6b908f6f19b" (UID: "40e489de-e909-42a7-bb78-a6b908f6f19b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:08:08.685634 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:08.685593    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:08:08.709916 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:08.709891    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40e489de-e909-42a7-bb78-a6b908f6f19b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:08:08.709916 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:08.709916    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40e489de-e909-42a7-bb78-a6b908f6f19b-cabundle-cert\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:08:08.710121 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:08.709926    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40e489de-e909-42a7-bb78-a6b908f6f19b-kserve-provision-location\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:08:08.710121 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:08.709935    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwrfb\" (UniqueName: \"kubernetes.io/projected/40e489de-e909-42a7-bb78-a6b908f6f19b-kube-api-access-dwrfb\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:08:08.710121 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:08.709943    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40e489de-e909-42a7-bb78-a6b908f6f19b-proxy-tls\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:08:08.744561 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:08.744530    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-sfb8f"]
Jul 27 18:08:08.744869 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:08.744856    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40e489de-e909-42a7-bb78-a6b908f6f19b" containerName="kube-rbac-proxy"
Jul 27 18:08:08.744913 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:08.744870    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="40e489de-e909-42a7-bb78-a6b908f6f19b" containerName="kube-rbac-proxy"
Jul 27 18:08:08.744913 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:08.744882    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40e489de-e909-42a7-bb78-a6b908f6f19b" containerName="storage-initializer"
Jul 27 18:08:08.744913 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:08.744888    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="40e489de-e909-42a7-bb78-a6b908f6f19b" containerName="storage-initializer"
Jul 27 18:08:08.744913 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:08.744902    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40e489de-e909-42a7-bb78-a6b908f6f19b" containerName="kserve-container"
Jul 27 18:08:08.744913 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:08.744908    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="40e489de-e909-42a7-bb78-a6b908f6f19b" containerName="kserve-container"
Jul 27 18:08:08.745070 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:08.744968    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="40e489de-e909-42a7-bb78-a6b908f6f19b" containerName="kube-rbac-proxy"
Jul 27 18:08:08.745070 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:08.744978    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="40e489de-e909-42a7-bb78-a6b908f6f19b" containerName="kserve-container"
Jul 27 18:08:08.748323 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:08.748305    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-sfb8f"
Jul 27 18:08:08.750935 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:08.750910    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 27 18:08:08.750935 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:08.750919    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
Jul 27 18:08:08.757857 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:08.757836    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-sfb8f"]
Jul 27 18:08:08.810638 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:08.810604    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7ab5778-f7dc-4abd-86da-580717b37c7b-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-sfb8f\" (UID: \"f7ab5778-f7dc-4abd-86da-580717b37c7b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-sfb8f"
Jul 27 18:08:08.810814 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:08.810640    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7ab5778-f7dc-4abd-86da-580717b37c7b-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-sfb8f\" (UID: \"f7ab5778-f7dc-4abd-86da-580717b37c7b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-sfb8f"
Jul 27 18:08:08.810814 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:08.810762    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkrfx\" (UniqueName: \"kubernetes.io/projected/f7ab5778-f7dc-4abd-86da-580717b37c7b-kube-api-access-bkrfx\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-sfb8f\" (UID: \"f7ab5778-f7dc-4abd-86da-580717b37c7b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-sfb8f"
Jul 27 18:08:08.810814 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:08.810797    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7ab5778-f7dc-4abd-86da-580717b37c7b-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-sfb8f\" (UID: \"f7ab5778-f7dc-4abd-86da-580717b37c7b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-sfb8f"
Jul 27 18:08:08.810936 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:08.810827    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7ab5778-f7dc-4abd-86da-580717b37c7b-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-sfb8f\" (UID: \"f7ab5778-f7dc-4abd-86da-580717b37c7b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-sfb8f"
Jul 27 18:08:08.911354 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:08.911316    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7ab5778-f7dc-4abd-86da-580717b37c7b-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-sfb8f\" (UID: \"f7ab5778-f7dc-4abd-86da-580717b37c7b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-sfb8f"
Jul 27 18:08:08.911541 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:08.911367    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7ab5778-f7dc-4abd-86da-580717b37c7b-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-sfb8f\" (UID: \"f7ab5778-f7dc-4abd-86da-580717b37c7b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-sfb8f"
Jul 27 18:08:08.911541 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:08.911409    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkrfx\" (UniqueName: \"kubernetes.io/projected/f7ab5778-f7dc-4abd-86da-580717b37c7b-kube-api-access-bkrfx\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-sfb8f\" (UID: \"f7ab5778-f7dc-4abd-86da-580717b37c7b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-sfb8f"
Jul 27 18:08:08.911541 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:08.911430    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7ab5778-f7dc-4abd-86da-580717b37c7b-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-sfb8f\" (UID: \"f7ab5778-f7dc-4abd-86da-580717b37c7b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-sfb8f"
Jul 27 18:08:08.911541 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:08.911448    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7ab5778-f7dc-4abd-86da-580717b37c7b-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-sfb8f\" (UID: \"f7ab5778-f7dc-4abd-86da-580717b37c7b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-sfb8f"
Jul 27 18:08:08.911541 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:08:08.911498    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-serving-cert: secret "isvc-lightgbm-v2-runtime-predictor-serving-cert" not found
Jul 27 18:08:08.911805 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:08:08.911572    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7ab5778-f7dc-4abd-86da-580717b37c7b-proxy-tls podName:f7ab5778-f7dc-4abd-86da-580717b37c7b nodeName:}" failed. No retries permitted until 2026-07-27 18:08:09.411550369 +0000 UTC m=+2431.554952399 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f7ab5778-f7dc-4abd-86da-580717b37c7b-proxy-tls") pod "isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-sfb8f" (UID: "f7ab5778-f7dc-4abd-86da-580717b37c7b") : secret "isvc-lightgbm-v2-runtime-predictor-serving-cert" not found
Jul 27 18:08:08.911871 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:08.911852    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7ab5778-f7dc-4abd-86da-580717b37c7b-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-sfb8f\" (UID: \"f7ab5778-f7dc-4abd-86da-580717b37c7b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-sfb8f"
Jul 27 18:08:08.912178 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:08.912161    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7ab5778-f7dc-4abd-86da-580717b37c7b-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-sfb8f\" (UID: \"f7ab5778-f7dc-4abd-86da-580717b37c7b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-sfb8f"
Jul 27 18:08:08.912219 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:08.912160    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7ab5778-f7dc-4abd-86da-580717b37c7b-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-sfb8f\" (UID: \"f7ab5778-f7dc-4abd-86da-580717b37c7b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-sfb8f"
Jul 27 18:08:08.921018 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:08.920996    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkrfx\" (UniqueName: \"kubernetes.io/projected/f7ab5778-f7dc-4abd-86da-580717b37c7b-kube-api-access-bkrfx\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-sfb8f\" (UID: \"f7ab5778-f7dc-4abd-86da-580717b37c7b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-sfb8f"
Jul 27 18:08:09.036516 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:09.036479    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:08:09.414406 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:09.414365    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7ab5778-f7dc-4abd-86da-580717b37c7b-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-sfb8f\" (UID: \"f7ab5778-f7dc-4abd-86da-580717b37c7b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-sfb8f"
Jul 27 18:08:09.416857 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:09.416837    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7ab5778-f7dc-4abd-86da-580717b37c7b-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-sfb8f\" (UID: \"f7ab5778-f7dc-4abd-86da-580717b37c7b\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-sfb8f"
Jul 27 18:08:09.446149 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:09.446120    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh"
Jul 27 18:08:09.446149 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:09.446135    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh" event={"ID":"40e489de-e909-42a7-bb78-a6b908f6f19b","Type":"ContainerDied","Data":"26b7b9da561c375057aeb05a5750caddc5cbbd98454f07540c1cccb10e26ad2e"}
Jul 27 18:08:09.446589 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:09.446226    2573 scope.go:117] "RemoveContainer" containerID="06a4ff4bef14e5984de7fa5c1f44db6cb307de8da9cfdefdfcd86c0af884b925"
Jul 27 18:08:09.454963 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:09.454944    2573 scope.go:117] "RemoveContainer" containerID="a237dc50744b7fb93662b893c54192b91ca73ab73309f3bcd21f811443acdc87"
Jul 27 18:08:09.462572 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:09.462549    2573 scope.go:117] "RemoveContainer" containerID="9ed1463c48b5336dd181704f04ddfa6e72f33d8d78c05233b0b0b8305236fa90"
Jul 27 18:08:09.470017 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:09.469965    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh"]
Jul 27 18:08:09.473385 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:09.473362    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-hjdhh"]
Jul 27 18:08:09.660157 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:09.660111    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-sfb8f"
Jul 27 18:08:09.794070 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:09.794044    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-sfb8f"]
Jul 27 18:08:09.796610 ip-10-0-140-0 kubenswrapper[2573]: W0727 18:08:09.796585    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7ab5778_f7dc_4abd_86da_580717b37c7b.slice/crio-8db824fbfa90cc3437706922e13aecd38a34c18ebb5641b40aced9ae3cda8858 WatchSource:0}: Error finding container 8db824fbfa90cc3437706922e13aecd38a34c18ebb5641b40aced9ae3cda8858: Status 404 returned error can't find the container with id 8db824fbfa90cc3437706922e13aecd38a34c18ebb5641b40aced9ae3cda8858
Jul 27 18:08:10.453185 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:10.453151    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-sfb8f" event={"ID":"f7ab5778-f7dc-4abd-86da-580717b37c7b","Type":"ContainerStarted","Data":"5c16e12ce98897c7fa8222a52dc07de70b69b694893b1a2de0a11ae5729e7a4b"}
Jul 27 18:08:10.453698 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:10.453219    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-sfb8f" event={"ID":"f7ab5778-f7dc-4abd-86da-580717b37c7b","Type":"ContainerStarted","Data":"8db824fbfa90cc3437706922e13aecd38a34c18ebb5641b40aced9ae3cda8858"}
Jul 27 18:08:10.469198 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:10.469170    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40e489de-e909-42a7-bb78-a6b908f6f19b" path="/var/lib/kubelet/pods/40e489de-e909-42a7-bb78-a6b908f6f19b/volumes"
Jul 27 18:08:14.472120 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:14.472089    2573 generic.go:356] "Generic (PLEG): container finished" podID="f7ab5778-f7dc-4abd-86da-580717b37c7b" containerID="5c16e12ce98897c7fa8222a52dc07de70b69b694893b1a2de0a11ae5729e7a4b" exitCode=0
Jul 27 18:08:14.472535 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:08:14.472153    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-sfb8f" event={"ID":"f7ab5778-f7dc-4abd-86da-580717b37c7b","Type":"ContainerDied","Data":"5c16e12ce98897c7fa8222a52dc07de70b69b694893b1a2de0a11ae5729e7a4b"}
Jul 27 18:09:38.623835 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:09:38.623753    2573 scope.go:117] "RemoveContainer" containerID="319718db98ba35e65b7cf685c8135c958c34525d60c9af7547249a2666fed37b"
Jul 27 18:10:29.893095 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:10:29.892745    2573 scope.go:117] "RemoveContainer" containerID="870f3e9c167f4a0423363ef4ac7d90ddce591e8e6127e09af969f320c9ed23a7"
Jul 27 18:10:29.948024 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:10:29.947997    2573 scope.go:117] "RemoveContainer" containerID="3092a1a28157e5cb16c1c869c8e8a72fb6b1f9ab21c4b7eb06aa3bf5cf9701b9"
Jul 27 18:10:31.180195 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:10:31.180157    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-sfb8f" event={"ID":"f7ab5778-f7dc-4abd-86da-580717b37c7b","Type":"ContainerStarted","Data":"afc31d988734939b473db212b68d1710b4086af91f9cdf88b9aa22cc70c12bd3"}
Jul 27 18:10:31.180684 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:10:31.180208    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-sfb8f" event={"ID":"f7ab5778-f7dc-4abd-86da-580717b37c7b","Type":"ContainerStarted","Data":"4dcbba3faa5da5d82aeaf047cf7afd01fae0f19a528ad242ab7ff8f172580191"}
Jul 27 18:10:31.180684 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:10:31.180302    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-sfb8f"
Jul 27 18:10:31.206606 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:10:31.206557    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-sfb8f" podStartSLOduration=3.4128923110000002 podStartE2EDuration="2m23.206540778s" totalImagesPullingTime="2m15.902927614s" totalInitContainerRuntime="3.890720853s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:08:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:10:31.205744054 +0000 UTC m=+2573.349146109" watchObservedRunningTime="2026-07-27 18:10:31.206540778 +0000 UTC m=+2573.349942832"
Jul 27 18:10:32.184910 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:10:32.184877    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-sfb8f"
Jul 27 18:10:38.198006 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:10:38.197973    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-sfb8f"
Jul 27 18:11:08.202347 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:08.202317    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-sfb8f"
Jul 27 18:11:08.918467 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:08.918435    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-sfb8f"]
Jul 27 18:11:08.918777 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:08.918745    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-sfb8f" podUID="f7ab5778-f7dc-4abd-86da-580717b37c7b" containerName="kube-rbac-proxy" containerID="cri-o://afc31d988734939b473db212b68d1710b4086af91f9cdf88b9aa22cc70c12bd3" gracePeriod=30
Jul 27 18:11:08.918899 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:08.918751    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-sfb8f" podUID="f7ab5778-f7dc-4abd-86da-580717b37c7b" containerName="kserve-container" containerID="cri-o://4dcbba3faa5da5d82aeaf047cf7afd01fae0f19a528ad242ab7ff8f172580191" gracePeriod=30
Jul 27 18:11:08.991081 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:08.991047    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:11:09.365750 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:09.365661    2573 generic.go:356] "Generic (PLEG): container finished" podID="f7ab5778-f7dc-4abd-86da-580717b37c7b" containerID="afc31d988734939b473db212b68d1710b4086af91f9cdf88b9aa22cc70c12bd3" exitCode=2
Jul 27 18:11:09.365750 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:09.365703    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-sfb8f" event={"ID":"f7ab5778-f7dc-4abd-86da-580717b37c7b","Type":"ContainerDied","Data":"afc31d988734939b473db212b68d1710b4086af91f9cdf88b9aa22cc70c12bd3"}
Jul 27 18:11:10.146360 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:10.146335    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-sfb8f"
Jul 27 18:11:10.225423 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:10.225391    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7ab5778-f7dc-4abd-86da-580717b37c7b-kube-rbac-proxy-sar-config\") pod \"f7ab5778-f7dc-4abd-86da-580717b37c7b\" (UID: \"f7ab5778-f7dc-4abd-86da-580717b37c7b\") "
Jul 27 18:11:10.225626 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:10.225449    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkrfx\" (UniqueName: \"kubernetes.io/projected/f7ab5778-f7dc-4abd-86da-580717b37c7b-kube-api-access-bkrfx\") pod \"f7ab5778-f7dc-4abd-86da-580717b37c7b\" (UID: \"f7ab5778-f7dc-4abd-86da-580717b37c7b\") "
Jul 27 18:11:10.225626 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:10.225482    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7ab5778-f7dc-4abd-86da-580717b37c7b-cabundle-cert\") pod \"f7ab5778-f7dc-4abd-86da-580717b37c7b\" (UID: \"f7ab5778-f7dc-4abd-86da-580717b37c7b\") "
Jul 27 18:11:10.225626 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:10.225508    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7ab5778-f7dc-4abd-86da-580717b37c7b-proxy-tls\") pod \"f7ab5778-f7dc-4abd-86da-580717b37c7b\" (UID: \"f7ab5778-f7dc-4abd-86da-580717b37c7b\") "
Jul 27 18:11:10.225812 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:10.225620    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7ab5778-f7dc-4abd-86da-580717b37c7b-kserve-provision-location\") pod \"f7ab5778-f7dc-4abd-86da-580717b37c7b\" (UID: \"f7ab5778-f7dc-4abd-86da-580717b37c7b\") "
Jul 27 18:11:10.225864 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:10.225833    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7ab5778-f7dc-4abd-86da-580717b37c7b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f7ab5778-f7dc-4abd-86da-580717b37c7b" (UID: "f7ab5778-f7dc-4abd-86da-580717b37c7b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:11:10.225902 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:10.225853    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7ab5778-f7dc-4abd-86da-580717b37c7b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f7ab5778-f7dc-4abd-86da-580717b37c7b" (UID: "f7ab5778-f7dc-4abd-86da-580717b37c7b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:11:10.226015 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:10.225990    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7ab5778-f7dc-4abd-86da-580717b37c7b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f7ab5778-f7dc-4abd-86da-580717b37c7b" (UID: "f7ab5778-f7dc-4abd-86da-580717b37c7b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:11:10.227619 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:10.227595    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7ab5778-f7dc-4abd-86da-580717b37c7b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f7ab5778-f7dc-4abd-86da-580717b37c7b" (UID: "f7ab5778-f7dc-4abd-86da-580717b37c7b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:11:10.227723 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:10.227627    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7ab5778-f7dc-4abd-86da-580717b37c7b-kube-api-access-bkrfx" (OuterVolumeSpecName: "kube-api-access-bkrfx") pod "f7ab5778-f7dc-4abd-86da-580717b37c7b" (UID: "f7ab5778-f7dc-4abd-86da-580717b37c7b"). InnerVolumeSpecName "kube-api-access-bkrfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:11:10.326713 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:10.326672    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7ab5778-f7dc-4abd-86da-580717b37c7b-cabundle-cert\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:11:10.326713 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:10.326707    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7ab5778-f7dc-4abd-86da-580717b37c7b-proxy-tls\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:11:10.326713 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:10.326721    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7ab5778-f7dc-4abd-86da-580717b37c7b-kserve-provision-location\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:11:10.326962 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:10.326734    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7ab5778-f7dc-4abd-86da-580717b37c7b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:11:10.326962 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:10.326745    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkrfx\" (UniqueName: \"kubernetes.io/projected/f7ab5778-f7dc-4abd-86da-580717b37c7b-kube-api-access-bkrfx\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:11:10.372234 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:10.372201    2573 generic.go:356] "Generic (PLEG): container finished" podID="f7ab5778-f7dc-4abd-86da-580717b37c7b" containerID="4dcbba3faa5da5d82aeaf047cf7afd01fae0f19a528ad242ab7ff8f172580191" exitCode=0
Jul 27 18:11:10.372652 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:10.372272    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-sfb8f"
Jul 27 18:11:10.372652 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:10.372289    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-sfb8f" event={"ID":"f7ab5778-f7dc-4abd-86da-580717b37c7b","Type":"ContainerDied","Data":"4dcbba3faa5da5d82aeaf047cf7afd01fae0f19a528ad242ab7ff8f172580191"}
Jul 27 18:11:10.372652 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:10.372354    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-sfb8f" event={"ID":"f7ab5778-f7dc-4abd-86da-580717b37c7b","Type":"ContainerDied","Data":"8db824fbfa90cc3437706922e13aecd38a34c18ebb5641b40aced9ae3cda8858"}
Jul 27 18:11:10.372652 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:10.372382    2573 scope.go:117] "RemoveContainer" containerID="afc31d988734939b473db212b68d1710b4086af91f9cdf88b9aa22cc70c12bd3"
Jul 27 18:11:10.381623 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:10.381603    2573 scope.go:117] "RemoveContainer" containerID="4dcbba3faa5da5d82aeaf047cf7afd01fae0f19a528ad242ab7ff8f172580191"
Jul 27 18:11:10.389386 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:10.389362    2573 scope.go:117] "RemoveContainer" containerID="5c16e12ce98897c7fa8222a52dc07de70b69b694893b1a2de0a11ae5729e7a4b"
Jul 27 18:11:10.396782 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:10.396757    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-sfb8f"]
Jul 27 18:11:10.397313 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:10.397291    2573 scope.go:117] "RemoveContainer" containerID="afc31d988734939b473db212b68d1710b4086af91f9cdf88b9aa22cc70c12bd3"
Jul 27 18:11:10.397616 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:11:10.397584    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afc31d988734939b473db212b68d1710b4086af91f9cdf88b9aa22cc70c12bd3\": container with ID starting with afc31d988734939b473db212b68d1710b4086af91f9cdf88b9aa22cc70c12bd3 not found: ID does not exist" containerID="afc31d988734939b473db212b68d1710b4086af91f9cdf88b9aa22cc70c12bd3"
Jul 27 18:11:10.397742 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:10.397616    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afc31d988734939b473db212b68d1710b4086af91f9cdf88b9aa22cc70c12bd3"} err="failed to get container status \"afc31d988734939b473db212b68d1710b4086af91f9cdf88b9aa22cc70c12bd3\": rpc error: code = NotFound desc = could not find container \"afc31d988734939b473db212b68d1710b4086af91f9cdf88b9aa22cc70c12bd3\": container with ID starting with afc31d988734939b473db212b68d1710b4086af91f9cdf88b9aa22cc70c12bd3 not found: ID does not exist"
Jul 27 18:11:10.397742 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:10.397635    2573 scope.go:117] "RemoveContainer" containerID="4dcbba3faa5da5d82aeaf047cf7afd01fae0f19a528ad242ab7ff8f172580191"
Jul 27 18:11:10.397910 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:11:10.397894    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dcbba3faa5da5d82aeaf047cf7afd01fae0f19a528ad242ab7ff8f172580191\": container with ID starting with 4dcbba3faa5da5d82aeaf047cf7afd01fae0f19a528ad242ab7ff8f172580191 not found: ID does not exist" containerID="4dcbba3faa5da5d82aeaf047cf7afd01fae0f19a528ad242ab7ff8f172580191"
Jul 27 18:11:10.397991 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:10.397916    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dcbba3faa5da5d82aeaf047cf7afd01fae0f19a528ad242ab7ff8f172580191"} err="failed to get container status \"4dcbba3faa5da5d82aeaf047cf7afd01fae0f19a528ad242ab7ff8f172580191\": rpc error: code = NotFound desc = could not find container \"4dcbba3faa5da5d82aeaf047cf7afd01fae0f19a528ad242ab7ff8f172580191\": container with ID starting with 4dcbba3faa5da5d82aeaf047cf7afd01fae0f19a528ad242ab7ff8f172580191 not found: ID does not exist"
Jul 27 18:11:10.397991 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:10.397935    2573 scope.go:117] "RemoveContainer" containerID="5c16e12ce98897c7fa8222a52dc07de70b69b694893b1a2de0a11ae5729e7a4b"
Jul 27 18:11:10.398310 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:11:10.398285    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c16e12ce98897c7fa8222a52dc07de70b69b694893b1a2de0a11ae5729e7a4b\": container with ID starting with 5c16e12ce98897c7fa8222a52dc07de70b69b694893b1a2de0a11ae5729e7a4b not found: ID does not exist" containerID="5c16e12ce98897c7fa8222a52dc07de70b69b694893b1a2de0a11ae5729e7a4b"
Jul 27 18:11:10.398402 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:10.398318    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c16e12ce98897c7fa8222a52dc07de70b69b694893b1a2de0a11ae5729e7a4b"} err="failed to get container status \"5c16e12ce98897c7fa8222a52dc07de70b69b694893b1a2de0a11ae5729e7a4b\": rpc error: code = NotFound desc = could not find container \"5c16e12ce98897c7fa8222a52dc07de70b69b694893b1a2de0a11ae5729e7a4b\": container with ID starting with 5c16e12ce98897c7fa8222a52dc07de70b69b694893b1a2de0a11ae5729e7a4b not found: ID does not exist"
Jul 27 18:11:10.400173 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:10.400150    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-sfb8f"]
Jul 27 18:11:10.469625 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:10.469592    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7ab5778-f7dc-4abd-86da-580717b37c7b" path="/var/lib/kubelet/pods/f7ab5778-f7dc-4abd-86da-580717b37c7b/volumes"
Jul 27 18:11:14.013438 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:14.013404    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:11:14.065370 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:14.065337    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cfvrg"]
Jul 27 18:11:14.065665 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:14.065653    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7ab5778-f7dc-4abd-86da-580717b37c7b" containerName="storage-initializer"
Jul 27 18:11:14.065710 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:14.065667    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7ab5778-f7dc-4abd-86da-580717b37c7b" containerName="storage-initializer"
Jul 27 18:11:14.065710 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:14.065676    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7ab5778-f7dc-4abd-86da-580717b37c7b" containerName="kube-rbac-proxy"
Jul 27 18:11:14.065710 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:14.065681    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7ab5778-f7dc-4abd-86da-580717b37c7b" containerName="kube-rbac-proxy"
Jul 27 18:11:14.065710 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:14.065689    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7ab5778-f7dc-4abd-86da-580717b37c7b" containerName="kserve-container"
Jul 27 18:11:14.065710 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:14.065694    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7ab5778-f7dc-4abd-86da-580717b37c7b" containerName="kserve-container"
Jul 27 18:11:14.065865 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:14.065746    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7ab5778-f7dc-4abd-86da-580717b37c7b" containerName="kserve-container"
Jul 27 18:11:14.065865 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:14.065754    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7ab5778-f7dc-4abd-86da-580717b37c7b" containerName="kube-rbac-proxy"
Jul 27 18:11:14.068875 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:14.068856    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cfvrg"
Jul 27 18:11:14.071776 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:14.071752    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
Jul 27 18:11:14.071917 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:14.071880    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 18:11:14.071917 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:14.071911    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 18:11:14.073242 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:14.073218    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 18:11:14.073377 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:14.073231    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z6ld7\""
Jul 27 18:11:14.073377 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:14.073231    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 18:11:14.073377 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:14.073268    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
Jul 27 18:11:14.079143 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:14.079121    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cfvrg"]
Jul 27 18:11:14.106951 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:14.106913    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:11:14.153150 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:14.153110    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f172fe5-22b0-4540-8b4e-c1d57d37ffe5-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cfvrg\" (UID: \"5f172fe5-22b0-4540-8b4e-c1d57d37ffe5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cfvrg"
Jul 27 18:11:14.153150 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:14.153150    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f172fe5-22b0-4540-8b4e-c1d57d37ffe5-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cfvrg\" (UID: \"5f172fe5-22b0-4540-8b4e-c1d57d37ffe5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cfvrg"
Jul 27 18:11:14.153379 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:14.153212    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clzc8\" (UniqueName: \"kubernetes.io/projected/5f172fe5-22b0-4540-8b4e-c1d57d37ffe5-kube-api-access-clzc8\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cfvrg\" (UID: \"5f172fe5-22b0-4540-8b4e-c1d57d37ffe5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cfvrg"
Jul 27 18:11:14.153379 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:14.153296    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f172fe5-22b0-4540-8b4e-c1d57d37ffe5-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cfvrg\" (UID: \"5f172fe5-22b0-4540-8b4e-c1d57d37ffe5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cfvrg"
Jul 27 18:11:14.153379 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:14.153329    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f172fe5-22b0-4540-8b4e-c1d57d37ffe5-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cfvrg\" (UID: \"5f172fe5-22b0-4540-8b4e-c1d57d37ffe5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cfvrg"
Jul 27 18:11:14.254017 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:14.253976    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f172fe5-22b0-4540-8b4e-c1d57d37ffe5-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cfvrg\" (UID: \"5f172fe5-22b0-4540-8b4e-c1d57d37ffe5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cfvrg"
Jul 27 18:11:14.254207 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:14.254137    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f172fe5-22b0-4540-8b4e-c1d57d37ffe5-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cfvrg\" (UID: \"5f172fe5-22b0-4540-8b4e-c1d57d37ffe5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cfvrg"
Jul 27 18:11:14.254207 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:14.254169    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f172fe5-22b0-4540-8b4e-c1d57d37ffe5-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cfvrg\" (UID: \"5f172fe5-22b0-4540-8b4e-c1d57d37ffe5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cfvrg"
Jul 27 18:11:14.254366 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:14.254213    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clzc8\" (UniqueName: \"kubernetes.io/projected/5f172fe5-22b0-4540-8b4e-c1d57d37ffe5-kube-api-access-clzc8\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cfvrg\" (UID: \"5f172fe5-22b0-4540-8b4e-c1d57d37ffe5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cfvrg"
Jul 27 18:11:14.254366 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:14.254243    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f172fe5-22b0-4540-8b4e-c1d57d37ffe5-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cfvrg\" (UID: \"5f172fe5-22b0-4540-8b4e-c1d57d37ffe5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cfvrg"
Jul 27 18:11:14.254692 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:14.254662    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f172fe5-22b0-4540-8b4e-c1d57d37ffe5-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cfvrg\" (UID: \"5f172fe5-22b0-4540-8b4e-c1d57d37ffe5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cfvrg"
Jul 27 18:11:14.254825 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:14.254805    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f172fe5-22b0-4540-8b4e-c1d57d37ffe5-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cfvrg\" (UID: \"5f172fe5-22b0-4540-8b4e-c1d57d37ffe5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cfvrg"
Jul 27 18:11:14.254874 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:14.254850    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f172fe5-22b0-4540-8b4e-c1d57d37ffe5-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cfvrg\" (UID: \"5f172fe5-22b0-4540-8b4e-c1d57d37ffe5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cfvrg"
Jul 27 18:11:14.256641 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:14.256617    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f172fe5-22b0-4540-8b4e-c1d57d37ffe5-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cfvrg\" (UID: \"5f172fe5-22b0-4540-8b4e-c1d57d37ffe5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cfvrg"
Jul 27 18:11:14.262790 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:14.262771    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clzc8\" (UniqueName: \"kubernetes.io/projected/5f172fe5-22b0-4540-8b4e-c1d57d37ffe5-kube-api-access-clzc8\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cfvrg\" (UID: \"5f172fe5-22b0-4540-8b4e-c1d57d37ffe5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cfvrg"
Jul 27 18:11:14.380938 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:14.380832    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cfvrg"
Jul 27 18:11:14.545862 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:14.545836    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cfvrg"]
Jul 27 18:11:14.548291 ip-10-0-140-0 kubenswrapper[2573]: W0727 18:11:14.548261    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f172fe5_22b0_4540_8b4e_c1d57d37ffe5.slice/crio-93cdc1031f1e3f0c489fc11228a8192f797393dd2b32f78308e81610ee9d9fdc WatchSource:0}: Error finding container 93cdc1031f1e3f0c489fc11228a8192f797393dd2b32f78308e81610ee9d9fdc: Status 404 returned error can't find the container with id 93cdc1031f1e3f0c489fc11228a8192f797393dd2b32f78308e81610ee9d9fdc
Jul 27 18:11:15.399186 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:15.399148    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cfvrg" event={"ID":"5f172fe5-22b0-4540-8b4e-c1d57d37ffe5","Type":"ContainerStarted","Data":"e2c3a0cf0b24a15ba6f72df5322dd84c45f7b8c5a7c49fb14df6e81623ae97a6"}
Jul 27 18:11:15.399705 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:15.399233    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cfvrg" event={"ID":"5f172fe5-22b0-4540-8b4e-c1d57d37ffe5","Type":"ContainerStarted","Data":"93cdc1031f1e3f0c489fc11228a8192f797393dd2b32f78308e81610ee9d9fdc"}
Jul 27 18:11:19.427216 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:19.427180    2573 generic.go:356] "Generic (PLEG): container finished" podID="5f172fe5-22b0-4540-8b4e-c1d57d37ffe5" containerID="e2c3a0cf0b24a15ba6f72df5322dd84c45f7b8c5a7c49fb14df6e81623ae97a6" exitCode=0
Jul 27 18:11:19.427623 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:19.427274    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cfvrg" event={"ID":"5f172fe5-22b0-4540-8b4e-c1d57d37ffe5","Type":"ContainerDied","Data":"e2c3a0cf0b24a15ba6f72df5322dd84c45f7b8c5a7c49fb14df6e81623ae97a6"}
Jul 27 18:11:20.433705 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:20.433670    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cfvrg" event={"ID":"5f172fe5-22b0-4540-8b4e-c1d57d37ffe5","Type":"ContainerStarted","Data":"d3971a092449a4e0ca3205a6e0a2830a73b24ef1b0740ff6b2b9950cc4f00e2d"}
Jul 27 18:11:20.434083 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:20.433732    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cfvrg" event={"ID":"5f172fe5-22b0-4540-8b4e-c1d57d37ffe5","Type":"ContainerStarted","Data":"1c6628d1e3f19d039a09eb7f92686eefb77b6b4e003e10f1d3d68b043be28410"}
Jul 27 18:11:20.434083 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:20.434005    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cfvrg"
Jul 27 18:11:20.434242 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:20.434166    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cfvrg"
Jul 27 18:11:20.435622 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:20.435596    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cfvrg" podUID="5f172fe5-22b0-4540-8b4e-c1d57d37ffe5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 27 18:11:20.455398 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:20.455341    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cfvrg" podStartSLOduration=2.390812026 podStartE2EDuration="6.455325914s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.064513888s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:11:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:11:20.454312023 +0000 UTC m=+2622.597714073" watchObservedRunningTime="2026-07-27 18:11:20.455325914 +0000 UTC m=+2622.598727964"
Jul 27 18:11:21.437992 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:21.437950    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cfvrg" podUID="5f172fe5-22b0-4540-8b4e-c1d57d37ffe5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 27 18:11:26.443305 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:26.443277    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cfvrg"
Jul 27 18:11:26.443926 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:26.443894    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cfvrg" podUID="5f172fe5-22b0-4540-8b4e-c1d57d37ffe5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Jul 27 18:11:28.768920 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:28.768880    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:11:36.445336 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:36.445307    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cfvrg"
Jul 27 18:11:44.091202 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:44.091172    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cfvrg"]
Jul 27 18:11:44.091742 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:44.091446    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cfvrg" podUID="5f172fe5-22b0-4540-8b4e-c1d57d37ffe5" containerName="kserve-container" containerID="cri-o://1c6628d1e3f19d039a09eb7f92686eefb77b6b4e003e10f1d3d68b043be28410" gracePeriod=30
Jul 27 18:11:44.091742 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:44.091517    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cfvrg" podUID="5f172fe5-22b0-4540-8b4e-c1d57d37ffe5" containerName="kube-rbac-proxy" containerID="cri-o://d3971a092449a4e0ca3205a6e0a2830a73b24ef1b0740ff6b2b9950cc4f00e2d" gracePeriod=30
Jul 27 18:11:44.109766 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:44.109741    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:11:44.540824 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:44.540790    2573 generic.go:356] "Generic (PLEG): container finished" podID="5f172fe5-22b0-4540-8b4e-c1d57d37ffe5" containerID="d3971a092449a4e0ca3205a6e0a2830a73b24ef1b0740ff6b2b9950cc4f00e2d" exitCode=2
Jul 27 18:11:44.541004 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:44.540864    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cfvrg" event={"ID":"5f172fe5-22b0-4540-8b4e-c1d57d37ffe5","Type":"ContainerDied","Data":"d3971a092449a4e0ca3205a6e0a2830a73b24ef1b0740ff6b2b9950cc4f00e2d"}
Jul 27 18:11:44.830537 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:44.830514    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cfvrg"
Jul 27 18:11:44.907661 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:44.907630    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f172fe5-22b0-4540-8b4e-c1d57d37ffe5-kube-rbac-proxy-sar-config\") pod \"5f172fe5-22b0-4540-8b4e-c1d57d37ffe5\" (UID: \"5f172fe5-22b0-4540-8b4e-c1d57d37ffe5\") "
Jul 27 18:11:44.907840 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:44.907677    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f172fe5-22b0-4540-8b4e-c1d57d37ffe5-kserve-provision-location\") pod \"5f172fe5-22b0-4540-8b4e-c1d57d37ffe5\" (UID: \"5f172fe5-22b0-4540-8b4e-c1d57d37ffe5\") "
Jul 27 18:11:44.907840 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:44.907697    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-clzc8\" (UniqueName: \"kubernetes.io/projected/5f172fe5-22b0-4540-8b4e-c1d57d37ffe5-kube-api-access-clzc8\") pod \"5f172fe5-22b0-4540-8b4e-c1d57d37ffe5\" (UID: \"5f172fe5-22b0-4540-8b4e-c1d57d37ffe5\") "
Jul 27 18:11:44.907840 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:44.907728    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f172fe5-22b0-4540-8b4e-c1d57d37ffe5-cabundle-cert\") pod \"5f172fe5-22b0-4540-8b4e-c1d57d37ffe5\" (UID: \"5f172fe5-22b0-4540-8b4e-c1d57d37ffe5\") "
Jul 27 18:11:44.907840 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:44.907779    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f172fe5-22b0-4540-8b4e-c1d57d37ffe5-proxy-tls\") pod \"5f172fe5-22b0-4540-8b4e-c1d57d37ffe5\" (UID: \"5f172fe5-22b0-4540-8b4e-c1d57d37ffe5\") "
Jul 27 18:11:44.908097 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:44.908069    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f172fe5-22b0-4540-8b4e-c1d57d37ffe5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5f172fe5-22b0-4540-8b4e-c1d57d37ffe5" (UID: "5f172fe5-22b0-4540-8b4e-c1d57d37ffe5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:11:44.908151 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:44.908069    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f172fe5-22b0-4540-8b4e-c1d57d37ffe5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5f172fe5-22b0-4540-8b4e-c1d57d37ffe5" (UID: "5f172fe5-22b0-4540-8b4e-c1d57d37ffe5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:11:44.908151 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:44.908125    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f172fe5-22b0-4540-8b4e-c1d57d37ffe5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5f172fe5-22b0-4540-8b4e-c1d57d37ffe5" (UID: "5f172fe5-22b0-4540-8b4e-c1d57d37ffe5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:11:44.909919 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:44.909896    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f172fe5-22b0-4540-8b4e-c1d57d37ffe5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5f172fe5-22b0-4540-8b4e-c1d57d37ffe5" (UID: "5f172fe5-22b0-4540-8b4e-c1d57d37ffe5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:11:44.910040 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:44.909922    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f172fe5-22b0-4540-8b4e-c1d57d37ffe5-kube-api-access-clzc8" (OuterVolumeSpecName: "kube-api-access-clzc8") pod "5f172fe5-22b0-4540-8b4e-c1d57d37ffe5" (UID: "5f172fe5-22b0-4540-8b4e-c1d57d37ffe5"). InnerVolumeSpecName "kube-api-access-clzc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:11:45.008757 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:45.008721    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f172fe5-22b0-4540-8b4e-c1d57d37ffe5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:11:45.008757 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:45.008753    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f172fe5-22b0-4540-8b4e-c1d57d37ffe5-kserve-provision-location\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:11:45.008757 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:45.008763    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clzc8\" (UniqueName: \"kubernetes.io/projected/5f172fe5-22b0-4540-8b4e-c1d57d37ffe5-kube-api-access-clzc8\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:11:45.009006 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:45.008773    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f172fe5-22b0-4540-8b4e-c1d57d37ffe5-cabundle-cert\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:11:45.009006 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:45.008783    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f172fe5-22b0-4540-8b4e-c1d57d37ffe5-proxy-tls\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:11:45.547341 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:45.547303    2573 generic.go:356] "Generic (PLEG): container finished" podID="5f172fe5-22b0-4540-8b4e-c1d57d37ffe5" containerID="1c6628d1e3f19d039a09eb7f92686eefb77b6b4e003e10f1d3d68b043be28410" exitCode=0
Jul 27 18:11:45.547859 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:45.547365    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cfvrg"
Jul 27 18:11:45.547859 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:45.547387    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cfvrg" event={"ID":"5f172fe5-22b0-4540-8b4e-c1d57d37ffe5","Type":"ContainerDied","Data":"1c6628d1e3f19d039a09eb7f92686eefb77b6b4e003e10f1d3d68b043be28410"}
Jul 27 18:11:45.547859 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:45.547456    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cfvrg" event={"ID":"5f172fe5-22b0-4540-8b4e-c1d57d37ffe5","Type":"ContainerDied","Data":"93cdc1031f1e3f0c489fc11228a8192f797393dd2b32f78308e81610ee9d9fdc"}
Jul 27 18:11:45.547859 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:45.547484    2573 scope.go:117] "RemoveContainer" containerID="d3971a092449a4e0ca3205a6e0a2830a73b24ef1b0740ff6b2b9950cc4f00e2d"
Jul 27 18:11:45.556731 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:45.556716    2573 scope.go:117] "RemoveContainer" containerID="1c6628d1e3f19d039a09eb7f92686eefb77b6b4e003e10f1d3d68b043be28410"
Jul 27 18:11:45.563982 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:45.563962    2573 scope.go:117] "RemoveContainer" containerID="e2c3a0cf0b24a15ba6f72df5322dd84c45f7b8c5a7c49fb14df6e81623ae97a6"
Jul 27 18:11:45.570046 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:45.570023    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cfvrg"]
Jul 27 18:11:45.572126 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:45.572092    2573 scope.go:117] "RemoveContainer" containerID="d3971a092449a4e0ca3205a6e0a2830a73b24ef1b0740ff6b2b9950cc4f00e2d"
Jul 27 18:11:45.572424 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:11:45.572403    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3971a092449a4e0ca3205a6e0a2830a73b24ef1b0740ff6b2b9950cc4f00e2d\": container with ID starting with d3971a092449a4e0ca3205a6e0a2830a73b24ef1b0740ff6b2b9950cc4f00e2d not found: ID does not exist" containerID="d3971a092449a4e0ca3205a6e0a2830a73b24ef1b0740ff6b2b9950cc4f00e2d"
Jul 27 18:11:45.572522 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:45.572437    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3971a092449a4e0ca3205a6e0a2830a73b24ef1b0740ff6b2b9950cc4f00e2d"} err="failed to get container status \"d3971a092449a4e0ca3205a6e0a2830a73b24ef1b0740ff6b2b9950cc4f00e2d\": rpc error: code = NotFound desc = could not find container \"d3971a092449a4e0ca3205a6e0a2830a73b24ef1b0740ff6b2b9950cc4f00e2d\": container with ID starting with d3971a092449a4e0ca3205a6e0a2830a73b24ef1b0740ff6b2b9950cc4f00e2d not found: ID does not exist"
Jul 27 18:11:45.572522 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:45.572464    2573 scope.go:117] "RemoveContainer" containerID="1c6628d1e3f19d039a09eb7f92686eefb77b6b4e003e10f1d3d68b043be28410"
Jul 27 18:11:45.572736 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:11:45.572715    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c6628d1e3f19d039a09eb7f92686eefb77b6b4e003e10f1d3d68b043be28410\": container with ID starting with 1c6628d1e3f19d039a09eb7f92686eefb77b6b4e003e10f1d3d68b043be28410 not found: ID does not exist" containerID="1c6628d1e3f19d039a09eb7f92686eefb77b6b4e003e10f1d3d68b043be28410"
Jul 27 18:11:45.572777 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:45.572747    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c6628d1e3f19d039a09eb7f92686eefb77b6b4e003e10f1d3d68b043be28410"} err="failed to get container status \"1c6628d1e3f19d039a09eb7f92686eefb77b6b4e003e10f1d3d68b043be28410\": rpc error: code = NotFound desc = could not find container \"1c6628d1e3f19d039a09eb7f92686eefb77b6b4e003e10f1d3d68b043be28410\": container with ID starting with 1c6628d1e3f19d039a09eb7f92686eefb77b6b4e003e10f1d3d68b043be28410 not found: ID does not exist"
Jul 27 18:11:45.572777 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:45.572768    2573 scope.go:117] "RemoveContainer" containerID="e2c3a0cf0b24a15ba6f72df5322dd84c45f7b8c5a7c49fb14df6e81623ae97a6"
Jul 27 18:11:45.573030 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:11:45.573010    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2c3a0cf0b24a15ba6f72df5322dd84c45f7b8c5a7c49fb14df6e81623ae97a6\": container with ID starting with e2c3a0cf0b24a15ba6f72df5322dd84c45f7b8c5a7c49fb14df6e81623ae97a6 not found: ID does not exist" containerID="e2c3a0cf0b24a15ba6f72df5322dd84c45f7b8c5a7c49fb14df6e81623ae97a6"
Jul 27 18:11:45.573117 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:45.573033    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2c3a0cf0b24a15ba6f72df5322dd84c45f7b8c5a7c49fb14df6e81623ae97a6"} err="failed to get container status \"e2c3a0cf0b24a15ba6f72df5322dd84c45f7b8c5a7c49fb14df6e81623ae97a6\": rpc error: code = NotFound desc = could not find container \"e2c3a0cf0b24a15ba6f72df5322dd84c45f7b8c5a7c49fb14df6e81623ae97a6\": container with ID starting with e2c3a0cf0b24a15ba6f72df5322dd84c45f7b8c5a7c49fb14df6e81623ae97a6 not found: ID does not exist"
Jul 27 18:11:45.573117 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:45.573054    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-cfvrg"]
Jul 27 18:11:46.469877 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:46.469841    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f172fe5-22b0-4540-8b4e-c1d57d37ffe5" path="/var/lib/kubelet/pods/5f172fe5-22b0-4540-8b4e-c1d57d37ffe5/volumes"
Jul 27 18:11:49.228486 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:49.228452    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:11:49.268329 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:49.268291    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vl4g9"]
Jul 27 18:11:49.268706 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:49.268687    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f172fe5-22b0-4540-8b4e-c1d57d37ffe5" containerName="storage-initializer"
Jul 27 18:11:49.268804 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:49.268709    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f172fe5-22b0-4540-8b4e-c1d57d37ffe5" containerName="storage-initializer"
Jul 27 18:11:49.268804 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:49.268739    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f172fe5-22b0-4540-8b4e-c1d57d37ffe5" containerName="kserve-container"
Jul 27 18:11:49.268804 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:49.268749    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f172fe5-22b0-4540-8b4e-c1d57d37ffe5" containerName="kserve-container"
Jul 27 18:11:49.268804 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:49.268773    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f172fe5-22b0-4540-8b4e-c1d57d37ffe5" containerName="kube-rbac-proxy"
Jul 27 18:11:49.268804 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:49.268781    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f172fe5-22b0-4540-8b4e-c1d57d37ffe5" containerName="kube-rbac-proxy"
Jul 27 18:11:49.269056 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:49.268858    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f172fe5-22b0-4540-8b4e-c1d57d37ffe5" containerName="kserve-container"
Jul 27 18:11:49.269056 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:49.268873    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f172fe5-22b0-4540-8b4e-c1d57d37ffe5" containerName="kube-rbac-proxy"
Jul 27 18:11:49.271321 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:49.271299    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vl4g9"
Jul 27 18:11:49.274499 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:49.274472    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 27 18:11:49.274499 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:49.274490    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 18:11:49.274701 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:49.274473    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 18:11:49.274830 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:49.274813    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 18:11:49.276048 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:49.276028    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
Jul 27 18:11:49.276167 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:49.276150    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z6ld7\""
Jul 27 18:11:49.276167 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:49.276158    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 18:11:49.282441 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:49.282419    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vl4g9"]
Jul 27 18:11:49.345902 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:49.345868    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f2e07fd-c6fc-4bb4-9387-fba98da9986a-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vl4g9\" (UID: \"0f2e07fd-c6fc-4bb4-9387-fba98da9986a\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vl4g9"
Jul 27 18:11:49.346108 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:49.345934    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f2e07fd-c6fc-4bb4-9387-fba98da9986a-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vl4g9\" (UID: \"0f2e07fd-c6fc-4bb4-9387-fba98da9986a\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vl4g9"
Jul 27 18:11:49.346108 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:49.345957    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f2e07fd-c6fc-4bb4-9387-fba98da9986a-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vl4g9\" (UID: \"0f2e07fd-c6fc-4bb4-9387-fba98da9986a\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vl4g9"
Jul 27 18:11:49.346108 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:49.346013    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f2e07fd-c6fc-4bb4-9387-fba98da9986a-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vl4g9\" (UID: \"0f2e07fd-c6fc-4bb4-9387-fba98da9986a\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vl4g9"
Jul 27 18:11:49.346108 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:49.346068    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg9bx\" (UniqueName: \"kubernetes.io/projected/0f2e07fd-c6fc-4bb4-9387-fba98da9986a-kube-api-access-tg9bx\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vl4g9\" (UID: \"0f2e07fd-c6fc-4bb4-9387-fba98da9986a\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vl4g9"
Jul 27 18:11:49.447393 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:49.447341    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f2e07fd-c6fc-4bb4-9387-fba98da9986a-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vl4g9\" (UID: \"0f2e07fd-c6fc-4bb4-9387-fba98da9986a\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vl4g9"
Jul 27 18:11:49.447393 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:49.447396    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f2e07fd-c6fc-4bb4-9387-fba98da9986a-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vl4g9\" (UID: \"0f2e07fd-c6fc-4bb4-9387-fba98da9986a\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vl4g9"
Jul 27 18:11:49.447666 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:49.447429    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f2e07fd-c6fc-4bb4-9387-fba98da9986a-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vl4g9\" (UID: \"0f2e07fd-c6fc-4bb4-9387-fba98da9986a\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vl4g9"
Jul 27 18:11:49.447666 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:49.447454    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg9bx\" (UniqueName: \"kubernetes.io/projected/0f2e07fd-c6fc-4bb4-9387-fba98da9986a-kube-api-access-tg9bx\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vl4g9\" (UID: \"0f2e07fd-c6fc-4bb4-9387-fba98da9986a\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vl4g9"
Jul 27 18:11:49.447666 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:49.447501    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f2e07fd-c6fc-4bb4-9387-fba98da9986a-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vl4g9\" (UID: \"0f2e07fd-c6fc-4bb4-9387-fba98da9986a\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vl4g9"
Jul 27 18:11:49.447666 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:11:49.447651    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-serving-cert: secret "isvc-mlflow-v2-runtime-predictor-serving-cert" not found
Jul 27 18:11:49.447875 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:11:49.447728    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f2e07fd-c6fc-4bb4-9387-fba98da9986a-proxy-tls podName:0f2e07fd-c6fc-4bb4-9387-fba98da9986a nodeName:}" failed. No retries permitted until 2026-07-27 18:11:49.947707286 +0000 UTC m=+2652.091109330 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0f2e07fd-c6fc-4bb4-9387-fba98da9986a-proxy-tls") pod "isvc-mlflow-v2-runtime-predictor-6456f94494-vl4g9" (UID: "0f2e07fd-c6fc-4bb4-9387-fba98da9986a") : secret "isvc-mlflow-v2-runtime-predictor-serving-cert" not found
Jul 27 18:11:49.447875 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:49.447845    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f2e07fd-c6fc-4bb4-9387-fba98da9986a-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vl4g9\" (UID: \"0f2e07fd-c6fc-4bb4-9387-fba98da9986a\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vl4g9"
Jul 27 18:11:49.448149 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:49.448129    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f2e07fd-c6fc-4bb4-9387-fba98da9986a-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vl4g9\" (UID: \"0f2e07fd-c6fc-4bb4-9387-fba98da9986a\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vl4g9"
Jul 27 18:11:49.448191 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:49.448145    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f2e07fd-c6fc-4bb4-9387-fba98da9986a-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vl4g9\" (UID: \"0f2e07fd-c6fc-4bb4-9387-fba98da9986a\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vl4g9"
Jul 27 18:11:49.456403 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:49.456377    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg9bx\" (UniqueName: \"kubernetes.io/projected/0f2e07fd-c6fc-4bb4-9387-fba98da9986a-kube-api-access-tg9bx\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vl4g9\" (UID: \"0f2e07fd-c6fc-4bb4-9387-fba98da9986a\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vl4g9"
Jul 27 18:11:49.713563 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:49.713514    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:11:49.950584 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:49.950525    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f2e07fd-c6fc-4bb4-9387-fba98da9986a-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vl4g9\" (UID: \"0f2e07fd-c6fc-4bb4-9387-fba98da9986a\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vl4g9"
Jul 27 18:11:49.952998 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:49.952970    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f2e07fd-c6fc-4bb4-9387-fba98da9986a-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-vl4g9\" (UID: \"0f2e07fd-c6fc-4bb4-9387-fba98da9986a\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vl4g9"
Jul 27 18:11:50.181828 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:50.181792    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vl4g9"
Jul 27 18:11:50.314776 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:50.314743    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vl4g9"]
Jul 27 18:11:50.318363 ip-10-0-140-0 kubenswrapper[2573]: W0727 18:11:50.318323    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f2e07fd_c6fc_4bb4_9387_fba98da9986a.slice/crio-ac3331bfd6b1d4135f2ad01885a08b0235f5ac34727c1a385e91dc67568f80c8 WatchSource:0}: Error finding container ac3331bfd6b1d4135f2ad01885a08b0235f5ac34727c1a385e91dc67568f80c8: Status 404 returned error can't find the container with id ac3331bfd6b1d4135f2ad01885a08b0235f5ac34727c1a385e91dc67568f80c8
Jul 27 18:11:50.574141 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:50.574050    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vl4g9" event={"ID":"0f2e07fd-c6fc-4bb4-9387-fba98da9986a","Type":"ContainerStarted","Data":"4f469712081a8df917a9ef230aca6a5b74c0ea1c25f757706cbc85d06518b089"}
Jul 27 18:11:50.574141 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:50.574108    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vl4g9" event={"ID":"0f2e07fd-c6fc-4bb4-9387-fba98da9986a","Type":"ContainerStarted","Data":"ac3331bfd6b1d4135f2ad01885a08b0235f5ac34727c1a385e91dc67568f80c8"}
Jul 27 18:11:54.592667 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:54.592631    2573 generic.go:356] "Generic (PLEG): container finished" podID="0f2e07fd-c6fc-4bb4-9387-fba98da9986a" containerID="4f469712081a8df917a9ef230aca6a5b74c0ea1c25f757706cbc85d06518b089" exitCode=0
Jul 27 18:11:54.593032 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:54.592707    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vl4g9" event={"ID":"0f2e07fd-c6fc-4bb4-9387-fba98da9986a","Type":"ContainerDied","Data":"4f469712081a8df917a9ef230aca6a5b74c0ea1c25f757706cbc85d06518b089"}
Jul 27 18:11:55.600114 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:55.600081    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vl4g9" event={"ID":"0f2e07fd-c6fc-4bb4-9387-fba98da9986a","Type":"ContainerStarted","Data":"0528d39558bd7dd0f5e370d1988d2f5c34ab9e55b95769fe1ef1bf15a4e21757"}
Jul 27 18:11:55.600652 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:55.600143    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vl4g9" event={"ID":"0f2e07fd-c6fc-4bb4-9387-fba98da9986a","Type":"ContainerStarted","Data":"4fb063236ff17515f4d0cbff0006dadaf8d14ca5b401e01d4cb9f974f900c4e4"}
Jul 27 18:11:55.600652 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:55.600450    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vl4g9"
Jul 27 18:11:55.625484 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:55.625422    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vl4g9" podStartSLOduration=2.5428419829999998 podStartE2EDuration="6.625404448s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.082562465s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:11:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:11:55.624986674 +0000 UTC m=+2657.768388726" watchObservedRunningTime="2026-07-27 18:11:55.625404448 +0000 UTC m=+2657.768806498"
Jul 27 18:11:56.606013 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:11:56.605979    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vl4g9"
Jul 27 18:12:02.615477 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:02.615449    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vl4g9"
Jul 27 18:12:32.620074 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:32.620036    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vl4g9"
Jul 27 18:12:38.466451 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:38.466419    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vc2tz_87dac797-f8d2-4d36-999f-f8ce855dbeff/ovn-acl-logging/0.log"
Jul 27 18:12:38.473223 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:38.473201    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vc2tz_87dac797-f8d2-4d36-999f-f8ce855dbeff/ovn-acl-logging/0.log"
Jul 27 18:12:39.361151 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:39.361119    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vl4g9"]
Jul 27 18:12:39.361496 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:39.361458    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vl4g9" podUID="0f2e07fd-c6fc-4bb4-9387-fba98da9986a" containerName="kserve-container" containerID="cri-o://4fb063236ff17515f4d0cbff0006dadaf8d14ca5b401e01d4cb9f974f900c4e4" gracePeriod=30
Jul 27 18:12:39.361496 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:39.361477    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vl4g9" podUID="0f2e07fd-c6fc-4bb4-9387-fba98da9986a" containerName="kube-rbac-proxy" containerID="cri-o://0528d39558bd7dd0f5e370d1988d2f5c34ab9e55b95769fe1ef1bf15a4e21757" gracePeriod=30
Jul 27 18:12:39.383201 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:39.383170    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:12:39.801580 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:39.801546    2573 generic.go:356] "Generic (PLEG): container finished" podID="0f2e07fd-c6fc-4bb4-9387-fba98da9986a" containerID="0528d39558bd7dd0f5e370d1988d2f5c34ab9e55b95769fe1ef1bf15a4e21757" exitCode=2
Jul 27 18:12:39.801580 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:39.801585    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vl4g9" event={"ID":"0f2e07fd-c6fc-4bb4-9387-fba98da9986a","Type":"ContainerDied","Data":"0528d39558bd7dd0f5e370d1988d2f5c34ab9e55b95769fe1ef1bf15a4e21757"}
Jul 27 18:12:40.703135 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:40.703108    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vl4g9"
Jul 27 18:12:40.777163 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:40.777066    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tg9bx\" (UniqueName: \"kubernetes.io/projected/0f2e07fd-c6fc-4bb4-9387-fba98da9986a-kube-api-access-tg9bx\") pod \"0f2e07fd-c6fc-4bb4-9387-fba98da9986a\" (UID: \"0f2e07fd-c6fc-4bb4-9387-fba98da9986a\") "
Jul 27 18:12:40.777163 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:40.777125    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f2e07fd-c6fc-4bb4-9387-fba98da9986a-kube-rbac-proxy-sar-config\") pod \"0f2e07fd-c6fc-4bb4-9387-fba98da9986a\" (UID: \"0f2e07fd-c6fc-4bb4-9387-fba98da9986a\") "
Jul 27 18:12:40.777443 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:40.777237    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f2e07fd-c6fc-4bb4-9387-fba98da9986a-cabundle-cert\") pod \"0f2e07fd-c6fc-4bb4-9387-fba98da9986a\" (UID: \"0f2e07fd-c6fc-4bb4-9387-fba98da9986a\") "
Jul 27 18:12:40.777443 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:40.777295    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f2e07fd-c6fc-4bb4-9387-fba98da9986a-kserve-provision-location\") pod \"0f2e07fd-c6fc-4bb4-9387-fba98da9986a\" (UID: \"0f2e07fd-c6fc-4bb4-9387-fba98da9986a\") "
Jul 27 18:12:40.777443 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:40.777320    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f2e07fd-c6fc-4bb4-9387-fba98da9986a-proxy-tls\") pod \"0f2e07fd-c6fc-4bb4-9387-fba98da9986a\" (UID: \"0f2e07fd-c6fc-4bb4-9387-fba98da9986a\") "
Jul 27 18:12:40.777586 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:40.777544    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f2e07fd-c6fc-4bb4-9387-fba98da9986a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0f2e07fd-c6fc-4bb4-9387-fba98da9986a" (UID: "0f2e07fd-c6fc-4bb4-9387-fba98da9986a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:12:40.777627 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:40.777596    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f2e07fd-c6fc-4bb4-9387-fba98da9986a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0f2e07fd-c6fc-4bb4-9387-fba98da9986a" (UID: "0f2e07fd-c6fc-4bb4-9387-fba98da9986a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:12:40.777663 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:40.777634    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f2e07fd-c6fc-4bb4-9387-fba98da9986a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0f2e07fd-c6fc-4bb4-9387-fba98da9986a" (UID: "0f2e07fd-c6fc-4bb4-9387-fba98da9986a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:12:40.779356 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:40.779324    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f2e07fd-c6fc-4bb4-9387-fba98da9986a-kube-api-access-tg9bx" (OuterVolumeSpecName: "kube-api-access-tg9bx") pod "0f2e07fd-c6fc-4bb4-9387-fba98da9986a" (UID: "0f2e07fd-c6fc-4bb4-9387-fba98da9986a"). InnerVolumeSpecName "kube-api-access-tg9bx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:12:40.779469 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:40.779399    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f2e07fd-c6fc-4bb4-9387-fba98da9986a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0f2e07fd-c6fc-4bb4-9387-fba98da9986a" (UID: "0f2e07fd-c6fc-4bb4-9387-fba98da9986a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:12:40.808689 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:40.808654    2573 generic.go:356] "Generic (PLEG): container finished" podID="0f2e07fd-c6fc-4bb4-9387-fba98da9986a" containerID="4fb063236ff17515f4d0cbff0006dadaf8d14ca5b401e01d4cb9f974f900c4e4" exitCode=0
Jul 27 18:12:40.809102 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:40.808709    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vl4g9" event={"ID":"0f2e07fd-c6fc-4bb4-9387-fba98da9986a","Type":"ContainerDied","Data":"4fb063236ff17515f4d0cbff0006dadaf8d14ca5b401e01d4cb9f974f900c4e4"}
Jul 27 18:12:40.809102 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:40.808757    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vl4g9" event={"ID":"0f2e07fd-c6fc-4bb4-9387-fba98da9986a","Type":"ContainerDied","Data":"ac3331bfd6b1d4135f2ad01885a08b0235f5ac34727c1a385e91dc67568f80c8"}
Jul 27 18:12:40.809102 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:40.808781    2573 scope.go:117] "RemoveContainer" containerID="0528d39558bd7dd0f5e370d1988d2f5c34ab9e55b95769fe1ef1bf15a4e21757"
Jul 27 18:12:40.809102 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:40.808710    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vl4g9"
Jul 27 18:12:40.817496 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:40.817477    2573 scope.go:117] "RemoveContainer" containerID="4fb063236ff17515f4d0cbff0006dadaf8d14ca5b401e01d4cb9f974f900c4e4"
Jul 27 18:12:40.825238 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:40.825220    2573 scope.go:117] "RemoveContainer" containerID="4f469712081a8df917a9ef230aca6a5b74c0ea1c25f757706cbc85d06518b089"
Jul 27 18:12:40.830888 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:40.830861    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vl4g9"]
Jul 27 18:12:40.833538 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:40.833516    2573 scope.go:117] "RemoveContainer" containerID="0528d39558bd7dd0f5e370d1988d2f5c34ab9e55b95769fe1ef1bf15a4e21757"
Jul 27 18:12:40.833788 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:12:40.833772    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0528d39558bd7dd0f5e370d1988d2f5c34ab9e55b95769fe1ef1bf15a4e21757\": container with ID starting with 0528d39558bd7dd0f5e370d1988d2f5c34ab9e55b95769fe1ef1bf15a4e21757 not found: ID does not exist" containerID="0528d39558bd7dd0f5e370d1988d2f5c34ab9e55b95769fe1ef1bf15a4e21757"
Jul 27 18:12:40.833848 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:40.833797    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0528d39558bd7dd0f5e370d1988d2f5c34ab9e55b95769fe1ef1bf15a4e21757"} err="failed to get container status \"0528d39558bd7dd0f5e370d1988d2f5c34ab9e55b95769fe1ef1bf15a4e21757\": rpc error: code = NotFound desc = could not find container \"0528d39558bd7dd0f5e370d1988d2f5c34ab9e55b95769fe1ef1bf15a4e21757\": container with ID starting with 0528d39558bd7dd0f5e370d1988d2f5c34ab9e55b95769fe1ef1bf15a4e21757 not found: ID does not exist"
Jul 27 18:12:40.833848 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:40.833814    2573 scope.go:117] "RemoveContainer" containerID="4fb063236ff17515f4d0cbff0006dadaf8d14ca5b401e01d4cb9f974f900c4e4"
Jul 27 18:12:40.834054 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:12:40.834037    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fb063236ff17515f4d0cbff0006dadaf8d14ca5b401e01d4cb9f974f900c4e4\": container with ID starting with 4fb063236ff17515f4d0cbff0006dadaf8d14ca5b401e01d4cb9f974f900c4e4 not found: ID does not exist" containerID="4fb063236ff17515f4d0cbff0006dadaf8d14ca5b401e01d4cb9f974f900c4e4"
Jul 27 18:12:40.834101 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:40.834060    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fb063236ff17515f4d0cbff0006dadaf8d14ca5b401e01d4cb9f974f900c4e4"} err="failed to get container status \"4fb063236ff17515f4d0cbff0006dadaf8d14ca5b401e01d4cb9f974f900c4e4\": rpc error: code = NotFound desc = could not find container \"4fb063236ff17515f4d0cbff0006dadaf8d14ca5b401e01d4cb9f974f900c4e4\": container with ID starting with 4fb063236ff17515f4d0cbff0006dadaf8d14ca5b401e01d4cb9f974f900c4e4 not found: ID does not exist"
Jul 27 18:12:40.834101 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:40.834073    2573 scope.go:117] "RemoveContainer" containerID="4f469712081a8df917a9ef230aca6a5b74c0ea1c25f757706cbc85d06518b089"
Jul 27 18:12:40.834357 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:12:40.834334    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f469712081a8df917a9ef230aca6a5b74c0ea1c25f757706cbc85d06518b089\": container with ID starting with 4f469712081a8df917a9ef230aca6a5b74c0ea1c25f757706cbc85d06518b089 not found: ID does not exist" containerID="4f469712081a8df917a9ef230aca6a5b74c0ea1c25f757706cbc85d06518b089"
Jul 27 18:12:40.834410 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:40.834362    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f469712081a8df917a9ef230aca6a5b74c0ea1c25f757706cbc85d06518b089"} err="failed to get container status \"4f469712081a8df917a9ef230aca6a5b74c0ea1c25f757706cbc85d06518b089\": rpc error: code = NotFound desc = could not find container \"4f469712081a8df917a9ef230aca6a5b74c0ea1c25f757706cbc85d06518b089\": container with ID starting with 4f469712081a8df917a9ef230aca6a5b74c0ea1c25f757706cbc85d06518b089 not found: ID does not exist"
Jul 27 18:12:40.834713 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:40.834695    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-vl4g9"]
Jul 27 18:12:40.877891 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:40.877863    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f2e07fd-c6fc-4bb4-9387-fba98da9986a-cabundle-cert\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:12:40.877891 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:40.877891    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f2e07fd-c6fc-4bb4-9387-fba98da9986a-kserve-provision-location\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:12:40.878063 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:40.877901    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f2e07fd-c6fc-4bb4-9387-fba98da9986a-proxy-tls\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:12:40.878063 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:40.877911    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg9bx\" (UniqueName: \"kubernetes.io/projected/0f2e07fd-c6fc-4bb4-9387-fba98da9986a-kube-api-access-tg9bx\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:12:40.878063 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:40.877921    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f2e07fd-c6fc-4bb4-9387-fba98da9986a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:12:42.468953 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:42.468914    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f2e07fd-c6fc-4bb4-9387-fba98da9986a" path="/var/lib/kubelet/pods/0f2e07fd-c6fc-4bb4-9387-fba98da9986a/volumes"
Jul 27 18:12:44.488470 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:44.488424    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:12:44.520153 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:44.520119    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-2ztx7"]
Jul 27 18:12:44.520465 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:44.520450    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f2e07fd-c6fc-4bb4-9387-fba98da9986a" containerName="storage-initializer"
Jul 27 18:12:44.520519 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:44.520466    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f2e07fd-c6fc-4bb4-9387-fba98da9986a" containerName="storage-initializer"
Jul 27 18:12:44.520519 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:44.520477    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f2e07fd-c6fc-4bb4-9387-fba98da9986a" containerName="kserve-container"
Jul 27 18:12:44.520519 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:44.520483    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f2e07fd-c6fc-4bb4-9387-fba98da9986a" containerName="kserve-container"
Jul 27 18:12:44.520519 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:44.520497    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f2e07fd-c6fc-4bb4-9387-fba98da9986a" containerName="kube-rbac-proxy"
Jul 27 18:12:44.520519 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:44.520503    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f2e07fd-c6fc-4bb4-9387-fba98da9986a" containerName="kube-rbac-proxy"
Jul 27 18:12:44.520673 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:44.520551    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f2e07fd-c6fc-4bb4-9387-fba98da9986a" containerName="kserve-container"
Jul 27 18:12:44.520673 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:44.520563    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f2e07fd-c6fc-4bb4-9387-fba98da9986a" containerName="kube-rbac-proxy"
Jul 27 18:12:44.523889 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:44.523867    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-2ztx7"
Jul 27 18:12:44.526798 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:44.526775    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
Jul 27 18:12:44.526933 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:44.526805    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
Jul 27 18:12:44.527045 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:44.527028    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 18:12:44.527118 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:44.527049    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 18:12:44.527118 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:44.527095    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 18:12:44.527234 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:44.527127    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z6ld7\""
Jul 27 18:12:44.527234 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:44.527139    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 18:12:44.540551 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:44.540522    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-2ztx7"]
Jul 27 18:12:44.608358 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:44.608312    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb0c85c2-3e3e-471d-99d7-021860f4b04b-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-5996957568-2ztx7\" (UID: \"bb0c85c2-3e3e-471d-99d7-021860f4b04b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-2ztx7"
Jul 27 18:12:44.608358 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:44.608354    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bb0c85c2-3e3e-471d-99d7-021860f4b04b-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-5996957568-2ztx7\" (UID: \"bb0c85c2-3e3e-471d-99d7-021860f4b04b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-2ztx7"
Jul 27 18:12:44.608602 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:44.608402    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb0c85c2-3e3e-471d-99d7-021860f4b04b-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-5996957568-2ztx7\" (UID: \"bb0c85c2-3e3e-471d-99d7-021860f4b04b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-2ztx7"
Jul 27 18:12:44.608602 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:44.608428    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkk28\" (UniqueName: \"kubernetes.io/projected/bb0c85c2-3e3e-471d-99d7-021860f4b04b-kube-api-access-gkk28\") pod \"isvc-sklearn-mcp-predictor-5996957568-2ztx7\" (UID: \"bb0c85c2-3e3e-471d-99d7-021860f4b04b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-2ztx7"
Jul 27 18:12:44.608602 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:44.608455    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb0c85c2-3e3e-471d-99d7-021860f4b04b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-5996957568-2ztx7\" (UID: \"bb0c85c2-3e3e-471d-99d7-021860f4b04b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-2ztx7"
Jul 27 18:12:44.709557 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:44.709511    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb0c85c2-3e3e-471d-99d7-021860f4b04b-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-5996957568-2ztx7\" (UID: \"bb0c85c2-3e3e-471d-99d7-021860f4b04b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-2ztx7"
Jul 27 18:12:44.709557 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:44.709556    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bb0c85c2-3e3e-471d-99d7-021860f4b04b-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-5996957568-2ztx7\" (UID: \"bb0c85c2-3e3e-471d-99d7-021860f4b04b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-2ztx7"
Jul 27 18:12:44.709813 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:44.709597    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb0c85c2-3e3e-471d-99d7-021860f4b04b-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-5996957568-2ztx7\" (UID: \"bb0c85c2-3e3e-471d-99d7-021860f4b04b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-2ztx7"
Jul 27 18:12:44.709813 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:44.709616    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkk28\" (UniqueName: \"kubernetes.io/projected/bb0c85c2-3e3e-471d-99d7-021860f4b04b-kube-api-access-gkk28\") pod \"isvc-sklearn-mcp-predictor-5996957568-2ztx7\" (UID: \"bb0c85c2-3e3e-471d-99d7-021860f4b04b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-2ztx7"
Jul 27 18:12:44.709813 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:44.709640    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb0c85c2-3e3e-471d-99d7-021860f4b04b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-5996957568-2ztx7\" (UID: \"bb0c85c2-3e3e-471d-99d7-021860f4b04b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-2ztx7"
Jul 27 18:12:44.709813 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:12:44.709666    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-serving-cert: secret "isvc-sklearn-mcp-predictor-serving-cert" not found
Jul 27 18:12:44.709813 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:12:44.709735    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb0c85c2-3e3e-471d-99d7-021860f4b04b-proxy-tls podName:bb0c85c2-3e3e-471d-99d7-021860f4b04b nodeName:}" failed. No retries permitted until 2026-07-27 18:12:45.209717979 +0000 UTC m=+2707.353120008 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bb0c85c2-3e3e-471d-99d7-021860f4b04b-proxy-tls") pod "isvc-sklearn-mcp-predictor-5996957568-2ztx7" (UID: "bb0c85c2-3e3e-471d-99d7-021860f4b04b") : secret "isvc-sklearn-mcp-predictor-serving-cert" not found
Jul 27 18:12:44.710175 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:44.710152    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb0c85c2-3e3e-471d-99d7-021860f4b04b-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-5996957568-2ztx7\" (UID: \"bb0c85c2-3e3e-471d-99d7-021860f4b04b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-2ztx7"
Jul 27 18:12:44.710360 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:44.710342    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bb0c85c2-3e3e-471d-99d7-021860f4b04b-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-5996957568-2ztx7\" (UID: \"bb0c85c2-3e3e-471d-99d7-021860f4b04b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-2ztx7"
Jul 27 18:12:44.710416 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:44.710403    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb0c85c2-3e3e-471d-99d7-021860f4b04b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-5996957568-2ztx7\" (UID: \"bb0c85c2-3e3e-471d-99d7-021860f4b04b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-2ztx7"
Jul 27 18:12:44.718344 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:44.718323    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkk28\" (UniqueName: \"kubernetes.io/projected/bb0c85c2-3e3e-471d-99d7-021860f4b04b-kube-api-access-gkk28\") pod \"isvc-sklearn-mcp-predictor-5996957568-2ztx7\" (UID: \"bb0c85c2-3e3e-471d-99d7-021860f4b04b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-2ztx7"
Jul 27 18:12:45.142397 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:45.142362    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:12:45.214342 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:45.214310    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb0c85c2-3e3e-471d-99d7-021860f4b04b-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-5996957568-2ztx7\" (UID: \"bb0c85c2-3e3e-471d-99d7-021860f4b04b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-2ztx7"
Jul 27 18:12:45.216872 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:45.216836    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb0c85c2-3e3e-471d-99d7-021860f4b04b-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-5996957568-2ztx7\" (UID: \"bb0c85c2-3e3e-471d-99d7-021860f4b04b\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-2ztx7"
Jul 27 18:12:45.440006 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:45.439909    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-2ztx7"
Jul 27 18:12:45.571724 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:45.571700    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-2ztx7"]
Jul 27 18:12:45.573902 ip-10-0-140-0 kubenswrapper[2573]: W0727 18:12:45.573874    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb0c85c2_3e3e_471d_99d7_021860f4b04b.slice/crio-99c87bf9d0942b36f23fd939562b11df9c9fad6b2654e864019510b9b2d1eb9c WatchSource:0}: Error finding container 99c87bf9d0942b36f23fd939562b11df9c9fad6b2654e864019510b9b2d1eb9c: Status 404 returned error can't find the container with id 99c87bf9d0942b36f23fd939562b11df9c9fad6b2654e864019510b9b2d1eb9c
Jul 27 18:12:45.834180 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:45.834096    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-2ztx7" event={"ID":"bb0c85c2-3e3e-471d-99d7-021860f4b04b","Type":"ContainerStarted","Data":"0fa195fcb9f262169826c156c2ddc4b02dbc353a52817e7151170ef326cb937c"}
Jul 27 18:12:45.834180 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:45.834159    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-2ztx7" event={"ID":"bb0c85c2-3e3e-471d-99d7-021860f4b04b","Type":"ContainerStarted","Data":"99c87bf9d0942b36f23fd939562b11df9c9fad6b2654e864019510b9b2d1eb9c"}
Jul 27 18:12:49.852074 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:49.852042    2573 generic.go:356] "Generic (PLEG): container finished" podID="bb0c85c2-3e3e-471d-99d7-021860f4b04b" containerID="0fa195fcb9f262169826c156c2ddc4b02dbc353a52817e7151170ef326cb937c" exitCode=0
Jul 27 18:12:49.852537 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:49.852087    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-2ztx7" event={"ID":"bb0c85c2-3e3e-471d-99d7-021860f4b04b","Type":"ContainerDied","Data":"0fa195fcb9f262169826c156c2ddc4b02dbc353a52817e7151170ef326cb937c"}
Jul 27 18:12:49.917977 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:49.917955    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 18:12:50.861399 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:50.861364    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-2ztx7" event={"ID":"bb0c85c2-3e3e-471d-99d7-021860f4b04b","Type":"ContainerStarted","Data":"110ac4dc71a6030051dec8a1e4f868c744f9e43530098bb028c1d186e893aaa8"}
Jul 27 18:12:52.874921 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:52.874834    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-2ztx7" event={"ID":"bb0c85c2-3e3e-471d-99d7-021860f4b04b","Type":"ContainerStarted","Data":"9daa98b5e878b2b3d2ee1b0048588542f548f13c5903fe0f91510d04830a0efa"}
Jul 27 18:12:52.874921 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:52.874897    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-2ztx7" event={"ID":"bb0c85c2-3e3e-471d-99d7-021860f4b04b","Type":"ContainerStarted","Data":"15bf4d5a7264f0f8f00646e4bef9aca9e47b0739d429099951ebc9247fe8a42e"}
Jul 27 18:12:52.875355 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:52.875012    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-2ztx7"
Jul 27 18:12:52.875355 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:52.875050    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-2ztx7"
Jul 27 18:12:52.875355 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:52.875083    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-2ztx7"
Jul 27 18:12:52.898939 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:52.898885    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-2ztx7" podStartSLOduration=2.658998219 podStartE2EDuration="8.898869092s" totalImagesPullingTime="2.524212686s" totalInitContainerRuntime="3.715658187s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:12:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:12:52.896682814 +0000 UTC m=+2715.040084885" watchObservedRunningTime="2026-07-27 18:12:52.898869092 +0000 UTC m=+2715.042271142"
Jul 27 18:12:58.885365 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:12:58.885334    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-2ztx7"
Jul 27 18:13:25.440826 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:13:25.440777    2573 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-2ztx7" podUID="bb0c85c2-3e3e-471d-99d7-021860f4b04b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.51:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 27 18:13:28.889916 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:13:28.889875    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-2ztx7"
Jul 27 18:13:58.891524 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:13:58.891493    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-2ztx7"
Jul 27 18:14:04.657221 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:14:04.657184    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-2ztx7"]
Jul 27 18:14:04.657727 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:14:04.657548    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-2ztx7" podUID="bb0c85c2-3e3e-471d-99d7-021860f4b04b" containerName="kserve-container" containerID="cri-o://110ac4dc71a6030051dec8a1e4f868c744f9e43530098bb028c1d186e893aaa8" gracePeriod=30
Jul 27 18:14:04.657727 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:14:04.657589    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-2ztx7" podUID="bb0c85c2-3e3e-471d-99d7-021860f4b04b" containerName="kube-rbac-proxy" containerID="cri-o://9daa98b5e878b2b3d2ee1b0048588542f548f13c5903fe0f91510d04830a0efa" gracePeriod=30
Jul 27 18:14:04.657727 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:14:04.657622    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-2ztx7" podUID="bb0c85c2-3e3e-471d-99d7-021860f4b04b" containerName="kserve-agent" containerID="cri-o://15bf4d5a7264f0f8f00646e4bef9aca9e47b0739d429099951ebc9247fe8a42e" gracePeriod=30
Jul 27 18:14:04.672829 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:14:04.672803    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:14:05.187542 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:14:05.187506    2573 generic.go:356] "Generic (PLEG): container finished" podID="bb0c85c2-3e3e-471d-99d7-021860f4b04b" containerID="9daa98b5e878b2b3d2ee1b0048588542f548f13c5903fe0f91510d04830a0efa" exitCode=2
Jul 27 18:14:05.187711 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:14:05.187566    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-2ztx7" event={"ID":"bb0c85c2-3e3e-471d-99d7-021860f4b04b","Type":"ContainerDied","Data":"9daa98b5e878b2b3d2ee1b0048588542f548f13c5903fe0f91510d04830a0efa"}
Jul 27 18:14:07.204067 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:14:07.204031    2573 generic.go:356] "Generic (PLEG): container finished" podID="bb0c85c2-3e3e-471d-99d7-021860f4b04b" containerID="110ac4dc71a6030051dec8a1e4f868c744f9e43530098bb028c1d186e893aaa8" exitCode=0
Jul 27 18:14:07.204459 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:14:07.204107    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-2ztx7" event={"ID":"bb0c85c2-3e3e-471d-99d7-021860f4b04b","Type":"ContainerDied","Data":"110ac4dc71a6030051dec8a1e4f868c744f9e43530098bb028c1d186e893aaa8"}
Jul 27 18:14:08.880993 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:14:08.880949    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-2ztx7" podUID="bb0c85c2-3e3e-471d-99d7-021860f4b04b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.51:8643/healthz\": dial tcp 10.133.0.51:8643: connect: connection refused"
Jul 27 18:14:08.886401 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:14:08.886375    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-2ztx7" podUID="bb0c85c2-3e3e-471d-99d7-021860f4b04b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.51:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 27 18:14:09.798142 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:14:09.798090    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:14:10.030593 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:14:10.030559    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:14:13.881262 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:14:13.881206    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-2ztx7" podUID="bb0c85c2-3e3e-471d-99d7-021860f4b04b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.51:8643/healthz\": dial tcp 10.133.0.51:8643: connect: connection refused"
Jul 27 18:14:18.881495 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:14:18.881446    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-2ztx7" podUID="bb0c85c2-3e3e-471d-99d7-021860f4b04b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.51:8643/healthz\": dial tcp 10.133.0.51:8643: connect: connection refused"
Jul 27 18:14:18.881936 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:14:18.881611    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-2ztx7"
Jul 27 18:14:18.885885 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:14:18.885855    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-2ztx7" podUID="bb0c85c2-3e3e-471d-99d7-021860f4b04b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.51:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 27 18:14:23.880520 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:14:23.880480    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-2ztx7" podUID="bb0c85c2-3e3e-471d-99d7-021860f4b04b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.51:8643/healthz\": dial tcp 10.133.0.51:8643: connect: connection refused"
Jul 27 18:14:28.881031 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:14:28.880985    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-2ztx7" podUID="bb0c85c2-3e3e-471d-99d7-021860f4b04b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.51:8643/healthz\": dial tcp 10.133.0.51:8643: connect: connection refused"
Jul 27 18:14:28.886449 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:14:28.886420    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-2ztx7" podUID="bb0c85c2-3e3e-471d-99d7-021860f4b04b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.51:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.133.0.51:8080: connect: connection refused"
Jul 27 18:14:28.886571 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:14:28.886557    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-2ztx7"
Jul 27 18:14:33.609477 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:14:33.609441    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:14:33.881476 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:14:33.881377    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-2ztx7" podUID="bb0c85c2-3e3e-471d-99d7-021860f4b04b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.51:8643/healthz\": dial tcp 10.133.0.51:8643: connect: connection refused"
Jul 27 18:14:34.871884 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:14:34.871859    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-2ztx7"
Jul 27 18:14:34.910334 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:14:34.910305    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bb0c85c2-3e3e-471d-99d7-021860f4b04b-cabundle-cert\") pod \"bb0c85c2-3e3e-471d-99d7-021860f4b04b\" (UID: \"bb0c85c2-3e3e-471d-99d7-021860f4b04b\") "
Jul 27 18:14:34.910563 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:14:34.910348    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb0c85c2-3e3e-471d-99d7-021860f4b04b-kube-rbac-proxy-sar-config\") pod \"bb0c85c2-3e3e-471d-99d7-021860f4b04b\" (UID: \"bb0c85c2-3e3e-471d-99d7-021860f4b04b\") "
Jul 27 18:14:34.910563 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:14:34.910410    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb0c85c2-3e3e-471d-99d7-021860f4b04b-kserve-provision-location\") pod \"bb0c85c2-3e3e-471d-99d7-021860f4b04b\" (UID: \"bb0c85c2-3e3e-471d-99d7-021860f4b04b\") "
Jul 27 18:14:34.910563 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:14:34.910439    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb0c85c2-3e3e-471d-99d7-021860f4b04b-proxy-tls\") pod \"bb0c85c2-3e3e-471d-99d7-021860f4b04b\" (UID: \"bb0c85c2-3e3e-471d-99d7-021860f4b04b\") "
Jul 27 18:14:34.910563 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:14:34.910508    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gkk28\" (UniqueName: \"kubernetes.io/projected/bb0c85c2-3e3e-471d-99d7-021860f4b04b-kube-api-access-gkk28\") pod \"bb0c85c2-3e3e-471d-99d7-021860f4b04b\" (UID: \"bb0c85c2-3e3e-471d-99d7-021860f4b04b\") "
Jul 27 18:14:34.910802 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:14:34.910777    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb0c85c2-3e3e-471d-99d7-021860f4b04b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bb0c85c2-3e3e-471d-99d7-021860f4b04b" (UID: "bb0c85c2-3e3e-471d-99d7-021860f4b04b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:14:34.910856 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:14:34.910724    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb0c85c2-3e3e-471d-99d7-021860f4b04b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bb0c85c2-3e3e-471d-99d7-021860f4b04b" (UID: "bb0c85c2-3e3e-471d-99d7-021860f4b04b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:14:34.910949 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:14:34.910914    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb0c85c2-3e3e-471d-99d7-021860f4b04b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bb0c85c2-3e3e-471d-99d7-021860f4b04b" (UID: "bb0c85c2-3e3e-471d-99d7-021860f4b04b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:14:34.912771 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:14:34.912698    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb0c85c2-3e3e-471d-99d7-021860f4b04b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bb0c85c2-3e3e-471d-99d7-021860f4b04b" (UID: "bb0c85c2-3e3e-471d-99d7-021860f4b04b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:14:34.913113 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:14:34.913085    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb0c85c2-3e3e-471d-99d7-021860f4b04b-kube-api-access-gkk28" (OuterVolumeSpecName: "kube-api-access-gkk28") pod "bb0c85c2-3e3e-471d-99d7-021860f4b04b" (UID: "bb0c85c2-3e3e-471d-99d7-021860f4b04b"). InnerVolumeSpecName "kube-api-access-gkk28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:14:35.011816 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:14:35.011776    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bb0c85c2-3e3e-471d-99d7-021860f4b04b-cabundle-cert\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:14:35.011816 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:14:35.011809    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bb0c85c2-3e3e-471d-99d7-021860f4b04b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:14:35.011816 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:14:35.011818    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bb0c85c2-3e3e-471d-99d7-021860f4b04b-kserve-provision-location\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:14:35.011816 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:14:35.011827    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bb0c85c2-3e3e-471d-99d7-021860f4b04b-proxy-tls\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:14:35.012086 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:14:35.011835    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkk28\" (UniqueName: \"kubernetes.io/projected/bb0c85c2-3e3e-471d-99d7-021860f4b04b-kube-api-access-gkk28\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:14:35.329602 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:14:35.329504    2573 generic.go:356] "Generic (PLEG): container finished" podID="bb0c85c2-3e3e-471d-99d7-021860f4b04b" containerID="15bf4d5a7264f0f8f00646e4bef9aca9e47b0739d429099951ebc9247fe8a42e" exitCode=137
Jul 27 18:14:35.329602 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:14:35.329553    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-2ztx7" event={"ID":"bb0c85c2-3e3e-471d-99d7-021860f4b04b","Type":"ContainerDied","Data":"15bf4d5a7264f0f8f00646e4bef9aca9e47b0739d429099951ebc9247fe8a42e"}
Jul 27 18:14:35.329602 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:14:35.329560    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-2ztx7"
Jul 27 18:14:35.329866 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:14:35.329622    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-2ztx7" event={"ID":"bb0c85c2-3e3e-471d-99d7-021860f4b04b","Type":"ContainerDied","Data":"99c87bf9d0942b36f23fd939562b11df9c9fad6b2654e864019510b9b2d1eb9c"}
Jul 27 18:14:35.329866 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:14:35.329665    2573 scope.go:117] "RemoveContainer" containerID="9daa98b5e878b2b3d2ee1b0048588542f548f13c5903fe0f91510d04830a0efa"
Jul 27 18:14:35.338528 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:14:35.338510    2573 scope.go:117] "RemoveContainer" containerID="15bf4d5a7264f0f8f00646e4bef9aca9e47b0739d429099951ebc9247fe8a42e"
Jul 27 18:14:35.346528 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:14:35.346508    2573 scope.go:117] "RemoveContainer" containerID="110ac4dc71a6030051dec8a1e4f868c744f9e43530098bb028c1d186e893aaa8"
Jul 27 18:14:35.354589 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:14:35.354565    2573 scope.go:117] "RemoveContainer" containerID="0fa195fcb9f262169826c156c2ddc4b02dbc353a52817e7151170ef326cb937c"
Jul 27 18:14:35.362834 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:14:35.362814    2573 scope.go:117] "RemoveContainer" containerID="9daa98b5e878b2b3d2ee1b0048588542f548f13c5903fe0f91510d04830a0efa"
Jul 27 18:14:35.363113 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:14:35.363093    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9daa98b5e878b2b3d2ee1b0048588542f548f13c5903fe0f91510d04830a0efa\": container with ID starting with 9daa98b5e878b2b3d2ee1b0048588542f548f13c5903fe0f91510d04830a0efa not found: ID does not exist" containerID="9daa98b5e878b2b3d2ee1b0048588542f548f13c5903fe0f91510d04830a0efa"
Jul 27 18:14:35.363178 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:14:35.363127    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9daa98b5e878b2b3d2ee1b0048588542f548f13c5903fe0f91510d04830a0efa"} err="failed to get container status \"9daa98b5e878b2b3d2ee1b0048588542f548f13c5903fe0f91510d04830a0efa\": rpc error: code = NotFound desc = could not find container \"9daa98b5e878b2b3d2ee1b0048588542f548f13c5903fe0f91510d04830a0efa\": container with ID starting with 9daa98b5e878b2b3d2ee1b0048588542f548f13c5903fe0f91510d04830a0efa not found: ID does not exist"
Jul 27 18:14:35.363178 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:14:35.363146    2573 scope.go:117] "RemoveContainer" containerID="15bf4d5a7264f0f8f00646e4bef9aca9e47b0739d429099951ebc9247fe8a42e"
Jul 27 18:14:35.363408 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:14:35.363389    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15bf4d5a7264f0f8f00646e4bef9aca9e47b0739d429099951ebc9247fe8a42e\": container with ID starting with 15bf4d5a7264f0f8f00646e4bef9aca9e47b0739d429099951ebc9247fe8a42e not found: ID does not exist" containerID="15bf4d5a7264f0f8f00646e4bef9aca9e47b0739d429099951ebc9247fe8a42e"
Jul 27 18:14:35.363468 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:14:35.363412    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15bf4d5a7264f0f8f00646e4bef9aca9e47b0739d429099951ebc9247fe8a42e"} err="failed to get container status \"15bf4d5a7264f0f8f00646e4bef9aca9e47b0739d429099951ebc9247fe8a42e\": rpc error: code = NotFound desc = could not find container \"15bf4d5a7264f0f8f00646e4bef9aca9e47b0739d429099951ebc9247fe8a42e\": container with ID starting with 15bf4d5a7264f0f8f00646e4bef9aca9e47b0739d429099951ebc9247fe8a42e not found: ID does not exist"
Jul 27 18:14:35.363468 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:14:35.363426    2573 scope.go:117] "RemoveContainer" containerID="110ac4dc71a6030051dec8a1e4f868c744f9e43530098bb028c1d186e893aaa8"
Jul 27 18:14:35.363640 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:14:35.363625    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"110ac4dc71a6030051dec8a1e4f868c744f9e43530098bb028c1d186e893aaa8\": container with ID starting with 110ac4dc71a6030051dec8a1e4f868c744f9e43530098bb028c1d186e893aaa8 not found: ID does not exist" containerID="110ac4dc71a6030051dec8a1e4f868c744f9e43530098bb028c1d186e893aaa8"
Jul 27 18:14:35.363697 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:14:35.363644    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"110ac4dc71a6030051dec8a1e4f868c744f9e43530098bb028c1d186e893aaa8"} err="failed to get container status \"110ac4dc71a6030051dec8a1e4f868c744f9e43530098bb028c1d186e893aaa8\": rpc error: code = NotFound desc = could not find container \"110ac4dc71a6030051dec8a1e4f868c744f9e43530098bb028c1d186e893aaa8\": container with ID starting with 110ac4dc71a6030051dec8a1e4f868c744f9e43530098bb028c1d186e893aaa8 not found: ID does not exist"
Jul 27 18:14:35.363697 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:14:35.363657    2573 scope.go:117] "RemoveContainer" containerID="0fa195fcb9f262169826c156c2ddc4b02dbc353a52817e7151170ef326cb937c"
Jul 27 18:14:35.363907 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:14:35.363887    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fa195fcb9f262169826c156c2ddc4b02dbc353a52817e7151170ef326cb937c\": container with ID starting with 0fa195fcb9f262169826c156c2ddc4b02dbc353a52817e7151170ef326cb937c not found: ID does not exist" containerID="0fa195fcb9f262169826c156c2ddc4b02dbc353a52817e7151170ef326cb937c"
Jul 27 18:14:35.363970 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:14:35.363912    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fa195fcb9f262169826c156c2ddc4b02dbc353a52817e7151170ef326cb937c"} err="failed to get container status \"0fa195fcb9f262169826c156c2ddc4b02dbc353a52817e7151170ef326cb937c\": rpc error: code = NotFound desc = could not find container \"0fa195fcb9f262169826c156c2ddc4b02dbc353a52817e7151170ef326cb937c\": container with ID starting with 0fa195fcb9f262169826c156c2ddc4b02dbc353a52817e7151170ef326cb937c not found: ID does not exist"
Jul 27 18:14:35.364028 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:14:35.363982    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-2ztx7"]
Jul 27 18:14:35.371720 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:14:35.371699    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-5996957568-2ztx7"]
Jul 27 18:14:36.469551 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:14:36.469517    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb0c85c2-3e3e-471d-99d7-021860f4b04b" path="/var/lib/kubelet/pods/bb0c85c2-3e3e-471d-99d7-021860f4b04b/volumes"
Jul 27 18:15:21.390091 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:15:21.390055    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:15:26.500577 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:15:26.500538    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:15:26.542210 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:15:26.542178    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-zksr8"]
Jul 27 18:15:26.542541 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:15:26.542527    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb0c85c2-3e3e-471d-99d7-021860f4b04b" containerName="storage-initializer"
Jul 27 18:15:26.542596 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:15:26.542543    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb0c85c2-3e3e-471d-99d7-021860f4b04b" containerName="storage-initializer"
Jul 27 18:15:26.542596 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:15:26.542561    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb0c85c2-3e3e-471d-99d7-021860f4b04b" containerName="kserve-agent"
Jul 27 18:15:26.542596 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:15:26.542567    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb0c85c2-3e3e-471d-99d7-021860f4b04b" containerName="kserve-agent"
Jul 27 18:15:26.542596 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:15:26.542579    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb0c85c2-3e3e-471d-99d7-021860f4b04b" containerName="kserve-container"
Jul 27 18:15:26.542596 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:15:26.542585    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb0c85c2-3e3e-471d-99d7-021860f4b04b" containerName="kserve-container"
Jul 27 18:15:26.542596 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:15:26.542594    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb0c85c2-3e3e-471d-99d7-021860f4b04b" containerName="kube-rbac-proxy"
Jul 27 18:15:26.542792 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:15:26.542600    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb0c85c2-3e3e-471d-99d7-021860f4b04b" containerName="kube-rbac-proxy"
Jul 27 18:15:26.542792 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:15:26.542670    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb0c85c2-3e3e-471d-99d7-021860f4b04b" containerName="kserve-agent"
Jul 27 18:15:26.542792 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:15:26.542684    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb0c85c2-3e3e-471d-99d7-021860f4b04b" containerName="kserve-container"
Jul 27 18:15:26.542792 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:15:26.542690    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb0c85c2-3e3e-471d-99d7-021860f4b04b" containerName="kube-rbac-proxy"
Jul 27 18:15:26.545881 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:15:26.545866    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-zksr8"
Jul 27 18:15:26.549331 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:15:26.549309    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 18:15:26.549568 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:15:26.549551    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z6ld7\""
Jul 27 18:15:26.550055 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:15:26.550036    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 18:15:26.550650 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:15:26.550626    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 18:15:26.550735 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:15:26.550628    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-predictor-serving-cert\""
Jul 27 18:15:26.550735 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:15:26.550681    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
Jul 27 18:15:26.550829 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:15:26.550684    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 18:15:26.568001 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:15:26.567974    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-zksr8"]
Jul 27 18:15:26.656877 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:15:26.656835    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5cr6\" (UniqueName: \"kubernetes.io/projected/a63bc9ce-88bb-40ff-9168-771c90afdc03-kube-api-access-h5cr6\") pod \"isvc-paddle-runtime-predictor-58b99fff76-zksr8\" (UID: \"a63bc9ce-88bb-40ff-9168-771c90afdc03\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-zksr8"
Jul 27 18:15:26.656877 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:15:26.656878    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a63bc9ce-88bb-40ff-9168-771c90afdc03-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-zksr8\" (UID: \"a63bc9ce-88bb-40ff-9168-771c90afdc03\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-zksr8"
Jul 27 18:15:26.657104 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:15:26.656913    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a63bc9ce-88bb-40ff-9168-771c90afdc03-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-zksr8\" (UID: \"a63bc9ce-88bb-40ff-9168-771c90afdc03\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-zksr8"
Jul 27 18:15:26.657104 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:15:26.656977    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a63bc9ce-88bb-40ff-9168-771c90afdc03-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-zksr8\" (UID: \"a63bc9ce-88bb-40ff-9168-771c90afdc03\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-zksr8"
Jul 27 18:15:26.657104 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:15:26.657048    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a63bc9ce-88bb-40ff-9168-771c90afdc03-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-zksr8\" (UID: \"a63bc9ce-88bb-40ff-9168-771c90afdc03\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-zksr8"
Jul 27 18:15:26.757750 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:15:26.757659    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a63bc9ce-88bb-40ff-9168-771c90afdc03-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-zksr8\" (UID: \"a63bc9ce-88bb-40ff-9168-771c90afdc03\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-zksr8"
Jul 27 18:15:26.757750 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:15:26.757718    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5cr6\" (UniqueName: \"kubernetes.io/projected/a63bc9ce-88bb-40ff-9168-771c90afdc03-kube-api-access-h5cr6\") pod \"isvc-paddle-runtime-predictor-58b99fff76-zksr8\" (UID: \"a63bc9ce-88bb-40ff-9168-771c90afdc03\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-zksr8"
Jul 27 18:15:26.757987 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:15:26.757903    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a63bc9ce-88bb-40ff-9168-771c90afdc03-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-zksr8\" (UID: \"a63bc9ce-88bb-40ff-9168-771c90afdc03\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-zksr8"
Jul 27 18:15:26.757987 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:15:26.757968    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a63bc9ce-88bb-40ff-9168-771c90afdc03-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-zksr8\" (UID: \"a63bc9ce-88bb-40ff-9168-771c90afdc03\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-zksr8"
Jul 27 18:15:26.758101 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:15:26.758023    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a63bc9ce-88bb-40ff-9168-771c90afdc03-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-zksr8\" (UID: \"a63bc9ce-88bb-40ff-9168-771c90afdc03\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-zksr8"
Jul 27 18:15:26.758374 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:15:26.758348    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a63bc9ce-88bb-40ff-9168-771c90afdc03-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-zksr8\" (UID: \"a63bc9ce-88bb-40ff-9168-771c90afdc03\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-zksr8"
Jul 27 18:15:26.758625 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:15:26.758606    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a63bc9ce-88bb-40ff-9168-771c90afdc03-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-zksr8\" (UID: \"a63bc9ce-88bb-40ff-9168-771c90afdc03\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-zksr8"
Jul 27 18:15:26.758712 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:15:26.758659    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a63bc9ce-88bb-40ff-9168-771c90afdc03-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-zksr8\" (UID: \"a63bc9ce-88bb-40ff-9168-771c90afdc03\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-zksr8"
Jul 27 18:15:26.760301 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:15:26.760281    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a63bc9ce-88bb-40ff-9168-771c90afdc03-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-zksr8\" (UID: \"a63bc9ce-88bb-40ff-9168-771c90afdc03\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-zksr8"
Jul 27 18:15:26.769411 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:15:26.769383    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5cr6\" (UniqueName: \"kubernetes.io/projected/a63bc9ce-88bb-40ff-9168-771c90afdc03-kube-api-access-h5cr6\") pod \"isvc-paddle-runtime-predictor-58b99fff76-zksr8\" (UID: \"a63bc9ce-88bb-40ff-9168-771c90afdc03\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-zksr8"
Jul 27 18:15:26.803799 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:15:26.803770    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:15:26.856516 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:15:26.856482    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-zksr8"
Jul 27 18:15:26.993865 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:15:26.993821    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-zksr8"]
Jul 27 18:15:27.001791 ip-10-0-140-0 kubenswrapper[2573]: W0727 18:15:27.001760    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda63bc9ce_88bb_40ff_9168_771c90afdc03.slice/crio-6649f6445478fb2090e124f244be5ce652f36f772a92bcb3a691415b69a3699b WatchSource:0}: Error finding container 6649f6445478fb2090e124f244be5ce652f36f772a92bcb3a691415b69a3699b: Status 404 returned error can't find the container with id 6649f6445478fb2090e124f244be5ce652f36f772a92bcb3a691415b69a3699b
Jul 27 18:15:27.549184 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:15:27.549140    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-zksr8" event={"ID":"a63bc9ce-88bb-40ff-9168-771c90afdc03","Type":"ContainerStarted","Data":"c8e0070500c91d2055e18ceea03715acd70c02748be09d510c25d54da84291dd"}
Jul 27 18:15:27.549593 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:15:27.549221    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-zksr8" event={"ID":"a63bc9ce-88bb-40ff-9168-771c90afdc03","Type":"ContainerStarted","Data":"6649f6445478fb2090e124f244be5ce652f36f772a92bcb3a691415b69a3699b"}
Jul 27 18:15:32.572807 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:15:32.572773    2573 generic.go:356] "Generic (PLEG): container finished" podID="a63bc9ce-88bb-40ff-9168-771c90afdc03" containerID="c8e0070500c91d2055e18ceea03715acd70c02748be09d510c25d54da84291dd" exitCode=0
Jul 27 18:15:32.573196 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:15:32.572844    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-zksr8" event={"ID":"a63bc9ce-88bb-40ff-9168-771c90afdc03","Type":"ContainerDied","Data":"c8e0070500c91d2055e18ceea03715acd70c02748be09d510c25d54da84291dd"}
Jul 27 18:15:44.632052 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:15:44.632015    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-zksr8" event={"ID":"a63bc9ce-88bb-40ff-9168-771c90afdc03","Type":"ContainerStarted","Data":"f9e64d2265b342fd784ae152537a09067ee4d322d4535d8ec41f8ce978ccd33c"}
Jul 27 18:15:44.632607 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:15:44.632083    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-zksr8" event={"ID":"a63bc9ce-88bb-40ff-9168-771c90afdc03","Type":"ContainerStarted","Data":"81c00a725e0fa561cc9b0903983159e8f6e6eca61fedfc46670d872b5602e537"}
Jul 27 18:15:44.632607 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:15:44.632421    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-zksr8"
Jul 27 18:15:44.632607 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:15:44.632470    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-zksr8"
Jul 27 18:15:44.633869 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:15:44.633838    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-zksr8" podUID="a63bc9ce-88bb-40ff-9168-771c90afdc03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Jul 27 18:15:44.652703 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:15:44.652655    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-zksr8" podStartSLOduration=2.733507115 podStartE2EDuration="18.652641507s" totalImagesPullingTime="11.306420261s" totalInitContainerRuntime="4.612714131s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:15:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:15:44.651063841 +0000 UTC m=+2886.794465894" watchObservedRunningTime="2026-07-27 18:15:44.652641507 +0000 UTC m=+2886.796043559"
Jul 27 18:15:45.636681 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:15:45.636641    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-zksr8" podUID="a63bc9ce-88bb-40ff-9168-771c90afdc03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Jul 27 18:15:50.641357 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:15:50.641324    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-zksr8"
Jul 27 18:15:50.641860 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:15:50.641829    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-zksr8" podUID="a63bc9ce-88bb-40ff-9168-771c90afdc03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Jul 27 18:16:00.642311 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:00.642276    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-zksr8" podUID="a63bc9ce-88bb-40ff-9168-771c90afdc03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Jul 27 18:16:10.642741 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:10.642694    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-zksr8" podUID="a63bc9ce-88bb-40ff-9168-771c90afdc03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Jul 27 18:16:20.642542 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:20.642499    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-zksr8" podUID="a63bc9ce-88bb-40ff-9168-771c90afdc03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Jul 27 18:16:30.643160 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:30.643131    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-zksr8"
Jul 27 18:16:38.234100 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:38.234067    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-zksr8"]
Jul 27 18:16:38.234553 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:38.234315    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-zksr8" podUID="a63bc9ce-88bb-40ff-9168-771c90afdc03" containerName="kserve-container" containerID="cri-o://81c00a725e0fa561cc9b0903983159e8f6e6eca61fedfc46670d872b5602e537" gracePeriod=30
Jul 27 18:16:38.234553 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:38.234358    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-zksr8" podUID="a63bc9ce-88bb-40ff-9168-771c90afdc03" containerName="kube-rbac-proxy" containerID="cri-o://f9e64d2265b342fd784ae152537a09067ee4d322d4535d8ec41f8ce978ccd33c" gracePeriod=30
Jul 27 18:16:38.254786 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:38.254757    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:16:38.869625 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:38.869587    2573 generic.go:356] "Generic (PLEG): container finished" podID="a63bc9ce-88bb-40ff-9168-771c90afdc03" containerID="f9e64d2265b342fd784ae152537a09067ee4d322d4535d8ec41f8ce978ccd33c" exitCode=2
Jul 27 18:16:38.869809 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:38.869658    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-zksr8" event={"ID":"a63bc9ce-88bb-40ff-9168-771c90afdc03","Type":"ContainerDied","Data":"f9e64d2265b342fd784ae152537a09067ee4d322d4535d8ec41f8ce978ccd33c"}
Jul 27 18:16:40.637871 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:40.637826    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-zksr8" podUID="a63bc9ce-88bb-40ff-9168-771c90afdc03" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.52:8643/healthz\": dial tcp 10.133.0.52:8643: connect: connection refused"
Jul 27 18:16:40.642790 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:40.642759    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-zksr8" podUID="a63bc9ce-88bb-40ff-9168-771c90afdc03" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Jul 27 18:16:40.927702 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:40.927674    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:16:41.129169 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:41.129139    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-zksr8"
Jul 27 18:16:41.200118 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:41.200030    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a63bc9ce-88bb-40ff-9168-771c90afdc03-kube-rbac-proxy-sar-config\") pod \"a63bc9ce-88bb-40ff-9168-771c90afdc03\" (UID: \"a63bc9ce-88bb-40ff-9168-771c90afdc03\") "
Jul 27 18:16:41.200118 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:41.200080    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a63bc9ce-88bb-40ff-9168-771c90afdc03-cabundle-cert\") pod \"a63bc9ce-88bb-40ff-9168-771c90afdc03\" (UID: \"a63bc9ce-88bb-40ff-9168-771c90afdc03\") "
Jul 27 18:16:41.200118 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:41.200102    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a63bc9ce-88bb-40ff-9168-771c90afdc03-proxy-tls\") pod \"a63bc9ce-88bb-40ff-9168-771c90afdc03\" (UID: \"a63bc9ce-88bb-40ff-9168-771c90afdc03\") "
Jul 27 18:16:41.200399 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:41.200172    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5cr6\" (UniqueName: \"kubernetes.io/projected/a63bc9ce-88bb-40ff-9168-771c90afdc03-kube-api-access-h5cr6\") pod \"a63bc9ce-88bb-40ff-9168-771c90afdc03\" (UID: \"a63bc9ce-88bb-40ff-9168-771c90afdc03\") "
Jul 27 18:16:41.200399 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:41.200222    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a63bc9ce-88bb-40ff-9168-771c90afdc03-kserve-provision-location\") pod \"a63bc9ce-88bb-40ff-9168-771c90afdc03\" (UID: \"a63bc9ce-88bb-40ff-9168-771c90afdc03\") "
Jul 27 18:16:41.200517 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:41.200487    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a63bc9ce-88bb-40ff-9168-771c90afdc03-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a63bc9ce-88bb-40ff-9168-771c90afdc03" (UID: "a63bc9ce-88bb-40ff-9168-771c90afdc03"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:16:41.200723 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:41.200687    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a63bc9ce-88bb-40ff-9168-771c90afdc03-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a63bc9ce-88bb-40ff-9168-771c90afdc03" (UID: "a63bc9ce-88bb-40ff-9168-771c90afdc03"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:16:41.202315 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:41.202292    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a63bc9ce-88bb-40ff-9168-771c90afdc03-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a63bc9ce-88bb-40ff-9168-771c90afdc03" (UID: "a63bc9ce-88bb-40ff-9168-771c90afdc03"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:16:41.202391 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:41.202360    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a63bc9ce-88bb-40ff-9168-771c90afdc03-kube-api-access-h5cr6" (OuterVolumeSpecName: "kube-api-access-h5cr6") pod "a63bc9ce-88bb-40ff-9168-771c90afdc03" (UID: "a63bc9ce-88bb-40ff-9168-771c90afdc03"). InnerVolumeSpecName "kube-api-access-h5cr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:16:41.213179 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:41.213148    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a63bc9ce-88bb-40ff-9168-771c90afdc03-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a63bc9ce-88bb-40ff-9168-771c90afdc03" (UID: "a63bc9ce-88bb-40ff-9168-771c90afdc03"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:16:41.301288 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:41.301224    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a63bc9ce-88bb-40ff-9168-771c90afdc03-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:16:41.301288 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:41.301282    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a63bc9ce-88bb-40ff-9168-771c90afdc03-cabundle-cert\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:16:41.301288 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:41.301294    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a63bc9ce-88bb-40ff-9168-771c90afdc03-proxy-tls\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:16:41.301495 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:41.301302    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5cr6\" (UniqueName: \"kubernetes.io/projected/a63bc9ce-88bb-40ff-9168-771c90afdc03-kube-api-access-h5cr6\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:16:41.301495 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:41.301311    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a63bc9ce-88bb-40ff-9168-771c90afdc03-kserve-provision-location\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:16:41.885040 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:41.885002    2573 generic.go:356] "Generic (PLEG): container finished" podID="a63bc9ce-88bb-40ff-9168-771c90afdc03" containerID="81c00a725e0fa561cc9b0903983159e8f6e6eca61fedfc46670d872b5602e537" exitCode=0
Jul 27 18:16:41.885480 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:41.885059    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-zksr8"
Jul 27 18:16:41.885480 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:41.885086    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-zksr8" event={"ID":"a63bc9ce-88bb-40ff-9168-771c90afdc03","Type":"ContainerDied","Data":"81c00a725e0fa561cc9b0903983159e8f6e6eca61fedfc46670d872b5602e537"}
Jul 27 18:16:41.885480 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:41.885146    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-zksr8" event={"ID":"a63bc9ce-88bb-40ff-9168-771c90afdc03","Type":"ContainerDied","Data":"6649f6445478fb2090e124f244be5ce652f36f772a92bcb3a691415b69a3699b"}
Jul 27 18:16:41.885480 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:41.885176    2573 scope.go:117] "RemoveContainer" containerID="f9e64d2265b342fd784ae152537a09067ee4d322d4535d8ec41f8ce978ccd33c"
Jul 27 18:16:41.893656 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:41.893639    2573 scope.go:117] "RemoveContainer" containerID="81c00a725e0fa561cc9b0903983159e8f6e6eca61fedfc46670d872b5602e537"
Jul 27 18:16:41.901061 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:41.901043    2573 scope.go:117] "RemoveContainer" containerID="c8e0070500c91d2055e18ceea03715acd70c02748be09d510c25d54da84291dd"
Jul 27 18:16:41.909206 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:41.909186    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-zksr8"]
Jul 27 18:16:41.909343 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:41.909263    2573 scope.go:117] "RemoveContainer" containerID="f9e64d2265b342fd784ae152537a09067ee4d322d4535d8ec41f8ce978ccd33c"
Jul 27 18:16:41.909524 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:16:41.909506    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9e64d2265b342fd784ae152537a09067ee4d322d4535d8ec41f8ce978ccd33c\": container with ID starting with f9e64d2265b342fd784ae152537a09067ee4d322d4535d8ec41f8ce978ccd33c not found: ID does not exist" containerID="f9e64d2265b342fd784ae152537a09067ee4d322d4535d8ec41f8ce978ccd33c"
Jul 27 18:16:41.909566 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:41.909532    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9e64d2265b342fd784ae152537a09067ee4d322d4535d8ec41f8ce978ccd33c"} err="failed to get container status \"f9e64d2265b342fd784ae152537a09067ee4d322d4535d8ec41f8ce978ccd33c\": rpc error: code = NotFound desc = could not find container \"f9e64d2265b342fd784ae152537a09067ee4d322d4535d8ec41f8ce978ccd33c\": container with ID starting with f9e64d2265b342fd784ae152537a09067ee4d322d4535d8ec41f8ce978ccd33c not found: ID does not exist"
Jul 27 18:16:41.909566 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:41.909549    2573 scope.go:117] "RemoveContainer" containerID="81c00a725e0fa561cc9b0903983159e8f6e6eca61fedfc46670d872b5602e537"
Jul 27 18:16:41.909774 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:16:41.909756    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81c00a725e0fa561cc9b0903983159e8f6e6eca61fedfc46670d872b5602e537\": container with ID starting with 81c00a725e0fa561cc9b0903983159e8f6e6eca61fedfc46670d872b5602e537 not found: ID does not exist" containerID="81c00a725e0fa561cc9b0903983159e8f6e6eca61fedfc46670d872b5602e537"
Jul 27 18:16:41.909811 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:41.909780    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81c00a725e0fa561cc9b0903983159e8f6e6eca61fedfc46670d872b5602e537"} err="failed to get container status \"81c00a725e0fa561cc9b0903983159e8f6e6eca61fedfc46670d872b5602e537\": rpc error: code = NotFound desc = could not find container \"81c00a725e0fa561cc9b0903983159e8f6e6eca61fedfc46670d872b5602e537\": container with ID starting with 81c00a725e0fa561cc9b0903983159e8f6e6eca61fedfc46670d872b5602e537 not found: ID does not exist"
Jul 27 18:16:41.909811 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:41.909795    2573 scope.go:117] "RemoveContainer" containerID="c8e0070500c91d2055e18ceea03715acd70c02748be09d510c25d54da84291dd"
Jul 27 18:16:41.909993 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:16:41.909979    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8e0070500c91d2055e18ceea03715acd70c02748be09d510c25d54da84291dd\": container with ID starting with c8e0070500c91d2055e18ceea03715acd70c02748be09d510c25d54da84291dd not found: ID does not exist" containerID="c8e0070500c91d2055e18ceea03715acd70c02748be09d510c25d54da84291dd"
Jul 27 18:16:41.910029 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:41.909997    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8e0070500c91d2055e18ceea03715acd70c02748be09d510c25d54da84291dd"} err="failed to get container status \"c8e0070500c91d2055e18ceea03715acd70c02748be09d510c25d54da84291dd\": rpc error: code = NotFound desc = could not find container \"c8e0070500c91d2055e18ceea03715acd70c02748be09d510c25d54da84291dd\": container with ID starting with c8e0070500c91d2055e18ceea03715acd70c02748be09d510c25d54da84291dd not found: ID does not exist"
Jul 27 18:16:41.913761 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:41.913741    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-zksr8"]
Jul 27 18:16:42.469881 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:42.469849    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a63bc9ce-88bb-40ff-9168-771c90afdc03" path="/var/lib/kubelet/pods/a63bc9ce-88bb-40ff-9168-771c90afdc03/volumes"
Jul 27 18:16:43.361295 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:43.361265    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:16:43.405975 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:43.405938    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8"]
Jul 27 18:16:43.406306 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:43.406290    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a63bc9ce-88bb-40ff-9168-771c90afdc03" containerName="kserve-container"
Jul 27 18:16:43.406370 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:43.406308    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a63bc9ce-88bb-40ff-9168-771c90afdc03" containerName="kserve-container"
Jul 27 18:16:43.406370 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:43.406329    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a63bc9ce-88bb-40ff-9168-771c90afdc03" containerName="storage-initializer"
Jul 27 18:16:43.406370 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:43.406335    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a63bc9ce-88bb-40ff-9168-771c90afdc03" containerName="storage-initializer"
Jul 27 18:16:43.406370 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:43.406344    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a63bc9ce-88bb-40ff-9168-771c90afdc03" containerName="kube-rbac-proxy"
Jul 27 18:16:43.406370 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:43.406350    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a63bc9ce-88bb-40ff-9168-771c90afdc03" containerName="kube-rbac-proxy"
Jul 27 18:16:43.406528 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:43.406404    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a63bc9ce-88bb-40ff-9168-771c90afdc03" containerName="kube-rbac-proxy"
Jul 27 18:16:43.406528 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:43.406414    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a63bc9ce-88bb-40ff-9168-771c90afdc03" containerName="kserve-container"
Jul 27 18:16:43.409553 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:43.409534    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8"
Jul 27 18:16:43.412638 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:43.412608    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 18:16:43.412638 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:43.412621    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 18:16:43.413970 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:43.413928    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z6ld7\""
Jul 27 18:16:43.413970 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:43.413948    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
Jul 27 18:16:43.414155 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:43.414050    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 18:16:43.414155 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:43.413966    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
Jul 27 18:16:43.414296 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:43.414166    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 18:16:43.418762 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:43.418739    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8"]
Jul 27 18:16:43.476627 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:43.476589    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:16:43.515778 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:43.515751    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/65b5085c-6947-4858-9f3f-b27f5b08baf8-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8\" (UID: \"65b5085c-6947-4858-9f3f-b27f5b08baf8\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8"
Jul 27 18:16:43.515936 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:43.515798    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/65b5085c-6947-4858-9f3f-b27f5b08baf8-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8\" (UID: \"65b5085c-6947-4858-9f3f-b27f5b08baf8\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8"
Jul 27 18:16:43.515936 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:43.515818    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65b5085c-6947-4858-9f3f-b27f5b08baf8-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8\" (UID: \"65b5085c-6947-4858-9f3f-b27f5b08baf8\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8"
Jul 27 18:16:43.515936 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:43.515870    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gnmz\" (UniqueName: \"kubernetes.io/projected/65b5085c-6947-4858-9f3f-b27f5b08baf8-kube-api-access-2gnmz\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8\" (UID: \"65b5085c-6947-4858-9f3f-b27f5b08baf8\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8"
Jul 27 18:16:43.515936 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:43.515896    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65b5085c-6947-4858-9f3f-b27f5b08baf8-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8\" (UID: \"65b5085c-6947-4858-9f3f-b27f5b08baf8\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8"
Jul 27 18:16:43.617152 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:43.617060    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/65b5085c-6947-4858-9f3f-b27f5b08baf8-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8\" (UID: \"65b5085c-6947-4858-9f3f-b27f5b08baf8\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8"
Jul 27 18:16:43.617152 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:43.617122    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/65b5085c-6947-4858-9f3f-b27f5b08baf8-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8\" (UID: \"65b5085c-6947-4858-9f3f-b27f5b08baf8\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8"
Jul 27 18:16:43.617152 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:43.617141    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65b5085c-6947-4858-9f3f-b27f5b08baf8-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8\" (UID: \"65b5085c-6947-4858-9f3f-b27f5b08baf8\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8"
Jul 27 18:16:43.617458 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:43.617167    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gnmz\" (UniqueName: \"kubernetes.io/projected/65b5085c-6947-4858-9f3f-b27f5b08baf8-kube-api-access-2gnmz\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8\" (UID: \"65b5085c-6947-4858-9f3f-b27f5b08baf8\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8"
Jul 27 18:16:43.617458 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:43.617195    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65b5085c-6947-4858-9f3f-b27f5b08baf8-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8\" (UID: \"65b5085c-6947-4858-9f3f-b27f5b08baf8\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8"
Jul 27 18:16:43.617574 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:43.617553    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/65b5085c-6947-4858-9f3f-b27f5b08baf8-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8\" (UID: \"65b5085c-6947-4858-9f3f-b27f5b08baf8\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8"
Jul 27 18:16:43.617825 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:43.617797    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/65b5085c-6947-4858-9f3f-b27f5b08baf8-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8\" (UID: \"65b5085c-6947-4858-9f3f-b27f5b08baf8\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8"
Jul 27 18:16:43.617944 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:43.617890    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65b5085c-6947-4858-9f3f-b27f5b08baf8-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8\" (UID: \"65b5085c-6947-4858-9f3f-b27f5b08baf8\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8"
Jul 27 18:16:43.619687 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:43.619669    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65b5085c-6947-4858-9f3f-b27f5b08baf8-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8\" (UID: \"65b5085c-6947-4858-9f3f-b27f5b08baf8\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8"
Jul 27 18:16:43.626507 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:43.626487    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gnmz\" (UniqueName: \"kubernetes.io/projected/65b5085c-6947-4858-9f3f-b27f5b08baf8-kube-api-access-2gnmz\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8\" (UID: \"65b5085c-6947-4858-9f3f-b27f5b08baf8\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8"
Jul 27 18:16:43.720419 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:43.720377    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8"
Jul 27 18:16:43.853181 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:43.853156    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8"]
Jul 27 18:16:43.855767 ip-10-0-140-0 kubenswrapper[2573]: W0727 18:16:43.855743    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65b5085c_6947_4858_9f3f_b27f5b08baf8.slice/crio-326b0f791b39e4ad727e32553a67c7dbe8c47560ed75b0ecc501d24fadbdde49 WatchSource:0}: Error finding container 326b0f791b39e4ad727e32553a67c7dbe8c47560ed75b0ecc501d24fadbdde49: Status 404 returned error can't find the container with id 326b0f791b39e4ad727e32553a67c7dbe8c47560ed75b0ecc501d24fadbdde49
Jul 27 18:16:43.896718 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:43.896690    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8" event={"ID":"65b5085c-6947-4858-9f3f-b27f5b08baf8","Type":"ContainerStarted","Data":"326b0f791b39e4ad727e32553a67c7dbe8c47560ed75b0ecc501d24fadbdde49"}
Jul 27 18:16:44.901889 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:44.901859    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8" event={"ID":"65b5085c-6947-4858-9f3f-b27f5b08baf8","Type":"ContainerStarted","Data":"811a87b58de01210cbb0adbdc12198c52c25f964e8e3554a000b1d1741d86112"}
Jul 27 18:16:48.919895 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:48.919858    2573 generic.go:356] "Generic (PLEG): container finished" podID="65b5085c-6947-4858-9f3f-b27f5b08baf8" containerID="811a87b58de01210cbb0adbdc12198c52c25f964e8e3554a000b1d1741d86112" exitCode=0
Jul 27 18:16:48.920385 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:48.919915    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8" event={"ID":"65b5085c-6947-4858-9f3f-b27f5b08baf8","Type":"ContainerDied","Data":"811a87b58de01210cbb0adbdc12198c52c25f964e8e3554a000b1d1741d86112"}
Jul 27 18:16:49.926694 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:49.926657    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8" event={"ID":"65b5085c-6947-4858-9f3f-b27f5b08baf8","Type":"ContainerStarted","Data":"b62e4aeb6fb6c9b878212f194a365afa7f6b4f695ca1c09af0521041861b875b"}
Jul 27 18:16:49.927225 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:49.926722    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8" event={"ID":"65b5085c-6947-4858-9f3f-b27f5b08baf8","Type":"ContainerStarted","Data":"cb883456ccc8a480e80db54631b68a614e7ac7403240fca97dc6cb4993c05108"}
Jul 27 18:16:49.927225 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:49.926996    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8"
Jul 27 18:16:49.927225 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:49.927162    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8"
Jul 27 18:16:49.928481 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:49.928457    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8" podUID="65b5085c-6947-4858-9f3f-b27f5b08baf8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Jul 27 18:16:50.931594 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:50.931558    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8" podUID="65b5085c-6947-4858-9f3f-b27f5b08baf8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Jul 27 18:16:55.937215 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:55.937187    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8"
Jul 27 18:16:55.937722 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:55.937694    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8" podUID="65b5085c-6947-4858-9f3f-b27f5b08baf8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Jul 27 18:16:55.957683 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:16:55.957635    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8" podStartSLOduration=8.708833802000001 podStartE2EDuration="12.957622968s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.248789166s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:16:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:16:49.945825343 +0000 UTC m=+2952.089227394" watchObservedRunningTime="2026-07-27 18:16:55.957622968 +0000 UTC m=+2958.101025019"
Jul 27 18:17:05.937652 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:05.937611    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8" podUID="65b5085c-6947-4858-9f3f-b27f5b08baf8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Jul 27 18:17:15.937658 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:15.937618    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8" podUID="65b5085c-6947-4858-9f3f-b27f5b08baf8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Jul 27 18:17:25.938268 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:25.938208    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8" podUID="65b5085c-6947-4858-9f3f-b27f5b08baf8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Jul 27 18:17:35.938489 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:35.938450    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8"
Jul 27 18:17:38.493029 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:38.492995    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vc2tz_87dac797-f8d2-4d36-999f-f8ce855dbeff/ovn-acl-logging/0.log"
Jul 27 18:17:38.499123 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:38.499103    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vc2tz_87dac797-f8d2-4d36-999f-f8ce855dbeff/ovn-acl-logging/0.log"
Jul 27 18:17:45.247795 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:45.247761    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8"]
Jul 27 18:17:45.248285 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:45.248016    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8" podUID="65b5085c-6947-4858-9f3f-b27f5b08baf8" containerName="kserve-container" containerID="cri-o://cb883456ccc8a480e80db54631b68a614e7ac7403240fca97dc6cb4993c05108" gracePeriod=30
Jul 27 18:17:45.248285 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:45.248042    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8" podUID="65b5085c-6947-4858-9f3f-b27f5b08baf8" containerName="kube-rbac-proxy" containerID="cri-o://b62e4aeb6fb6c9b878212f194a365afa7f6b4f695ca1c09af0521041861b875b" gracePeriod=30
Jul 27 18:17:45.306037 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:45.306002    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:17:45.932682 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:45.932642    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8" podUID="65b5085c-6947-4858-9f3f-b27f5b08baf8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.53:8643/healthz\": dial tcp 10.133.0.53:8643: connect: connection refused"
Jul 27 18:17:45.938041 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:45.938015    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8" podUID="65b5085c-6947-4858-9f3f-b27f5b08baf8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Jul 27 18:17:46.175410 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:46.175373    2573 generic.go:356] "Generic (PLEG): container finished" podID="65b5085c-6947-4858-9f3f-b27f5b08baf8" containerID="b62e4aeb6fb6c9b878212f194a365afa7f6b4f695ca1c09af0521041861b875b" exitCode=2
Jul 27 18:17:46.175587 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:46.175443    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8" event={"ID":"65b5085c-6947-4858-9f3f-b27f5b08baf8","Type":"ContainerDied","Data":"b62e4aeb6fb6c9b878212f194a365afa7f6b4f695ca1c09af0521041861b875b"}
Jul 27 18:17:48.096181 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:48.096151    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8"
Jul 27 18:17:48.185981 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:48.185894    2573 generic.go:356] "Generic (PLEG): container finished" podID="65b5085c-6947-4858-9f3f-b27f5b08baf8" containerID="cb883456ccc8a480e80db54631b68a614e7ac7403240fca97dc6cb4993c05108" exitCode=0
Jul 27 18:17:48.185981 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:48.185948    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8"
Jul 27 18:17:48.185981 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:48.185952    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8" event={"ID":"65b5085c-6947-4858-9f3f-b27f5b08baf8","Type":"ContainerDied","Data":"cb883456ccc8a480e80db54631b68a614e7ac7403240fca97dc6cb4993c05108"}
Jul 27 18:17:48.186200 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:48.186018    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8" event={"ID":"65b5085c-6947-4858-9f3f-b27f5b08baf8","Type":"ContainerDied","Data":"326b0f791b39e4ad727e32553a67c7dbe8c47560ed75b0ecc501d24fadbdde49"}
Jul 27 18:17:48.186200 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:48.186046    2573 scope.go:117] "RemoveContainer" containerID="b62e4aeb6fb6c9b878212f194a365afa7f6b4f695ca1c09af0521041861b875b"
Jul 27 18:17:48.194004 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:48.193990    2573 scope.go:117] "RemoveContainer" containerID="cb883456ccc8a480e80db54631b68a614e7ac7403240fca97dc6cb4993c05108"
Jul 27 18:17:48.201318 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:48.201296    2573 scope.go:117] "RemoveContainer" containerID="811a87b58de01210cbb0adbdc12198c52c25f964e8e3554a000b1d1741d86112"
Jul 27 18:17:48.208726 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:48.208707    2573 scope.go:117] "RemoveContainer" containerID="b62e4aeb6fb6c9b878212f194a365afa7f6b4f695ca1c09af0521041861b875b"
Jul 27 18:17:48.209007 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:17:48.208984    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b62e4aeb6fb6c9b878212f194a365afa7f6b4f695ca1c09af0521041861b875b\": container with ID starting with b62e4aeb6fb6c9b878212f194a365afa7f6b4f695ca1c09af0521041861b875b not found: ID does not exist" containerID="b62e4aeb6fb6c9b878212f194a365afa7f6b4f695ca1c09af0521041861b875b"
Jul 27 18:17:48.209115 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:48.209013    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b62e4aeb6fb6c9b878212f194a365afa7f6b4f695ca1c09af0521041861b875b"} err="failed to get container status \"b62e4aeb6fb6c9b878212f194a365afa7f6b4f695ca1c09af0521041861b875b\": rpc error: code = NotFound desc = could not find container \"b62e4aeb6fb6c9b878212f194a365afa7f6b4f695ca1c09af0521041861b875b\": container with ID starting with b62e4aeb6fb6c9b878212f194a365afa7f6b4f695ca1c09af0521041861b875b not found: ID does not exist"
Jul 27 18:17:48.209115 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:48.209030    2573 scope.go:117] "RemoveContainer" containerID="cb883456ccc8a480e80db54631b68a614e7ac7403240fca97dc6cb4993c05108"
Jul 27 18:17:48.209259 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:17:48.209231    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb883456ccc8a480e80db54631b68a614e7ac7403240fca97dc6cb4993c05108\": container with ID starting with cb883456ccc8a480e80db54631b68a614e7ac7403240fca97dc6cb4993c05108 not found: ID does not exist" containerID="cb883456ccc8a480e80db54631b68a614e7ac7403240fca97dc6cb4993c05108"
Jul 27 18:17:48.209316 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:48.209264    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb883456ccc8a480e80db54631b68a614e7ac7403240fca97dc6cb4993c05108"} err="failed to get container status \"cb883456ccc8a480e80db54631b68a614e7ac7403240fca97dc6cb4993c05108\": rpc error: code = NotFound desc = could not find container \"cb883456ccc8a480e80db54631b68a614e7ac7403240fca97dc6cb4993c05108\": container with ID starting with cb883456ccc8a480e80db54631b68a614e7ac7403240fca97dc6cb4993c05108 not found: ID does not exist"
Jul 27 18:17:48.209316 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:48.209279    2573 scope.go:117] "RemoveContainer" containerID="811a87b58de01210cbb0adbdc12198c52c25f964e8e3554a000b1d1741d86112"
Jul 27 18:17:48.209508 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:17:48.209493    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"811a87b58de01210cbb0adbdc12198c52c25f964e8e3554a000b1d1741d86112\": container with ID starting with 811a87b58de01210cbb0adbdc12198c52c25f964e8e3554a000b1d1741d86112 not found: ID does not exist" containerID="811a87b58de01210cbb0adbdc12198c52c25f964e8e3554a000b1d1741d86112"
Jul 27 18:17:48.209549 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:48.209514    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"811a87b58de01210cbb0adbdc12198c52c25f964e8e3554a000b1d1741d86112"} err="failed to get container status \"811a87b58de01210cbb0adbdc12198c52c25f964e8e3554a000b1d1741d86112\": rpc error: code = NotFound desc = could not find container \"811a87b58de01210cbb0adbdc12198c52c25f964e8e3554a000b1d1741d86112\": container with ID starting with 811a87b58de01210cbb0adbdc12198c52c25f964e8e3554a000b1d1741d86112 not found: ID does not exist"
Jul 27 18:17:48.274861 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:48.274823    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/65b5085c-6947-4858-9f3f-b27f5b08baf8-kserve-provision-location\") pod \"65b5085c-6947-4858-9f3f-b27f5b08baf8\" (UID: \"65b5085c-6947-4858-9f3f-b27f5b08baf8\") "
Jul 27 18:17:48.275036 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:48.274878    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/65b5085c-6947-4858-9f3f-b27f5b08baf8-cabundle-cert\") pod \"65b5085c-6947-4858-9f3f-b27f5b08baf8\" (UID: \"65b5085c-6947-4858-9f3f-b27f5b08baf8\") "
Jul 27 18:17:48.275036 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:48.274906    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65b5085c-6947-4858-9f3f-b27f5b08baf8-proxy-tls\") pod \"65b5085c-6947-4858-9f3f-b27f5b08baf8\" (UID: \"65b5085c-6947-4858-9f3f-b27f5b08baf8\") "
Jul 27 18:17:48.275036 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:48.274940    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2gnmz\" (UniqueName: \"kubernetes.io/projected/65b5085c-6947-4858-9f3f-b27f5b08baf8-kube-api-access-2gnmz\") pod \"65b5085c-6947-4858-9f3f-b27f5b08baf8\" (UID: \"65b5085c-6947-4858-9f3f-b27f5b08baf8\") "
Jul 27 18:17:48.275036 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:48.274987    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65b5085c-6947-4858-9f3f-b27f5b08baf8-kube-rbac-proxy-sar-config\") pod \"65b5085c-6947-4858-9f3f-b27f5b08baf8\" (UID: \"65b5085c-6947-4858-9f3f-b27f5b08baf8\") "
Jul 27 18:17:48.275306 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:48.275282    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65b5085c-6947-4858-9f3f-b27f5b08baf8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "65b5085c-6947-4858-9f3f-b27f5b08baf8" (UID: "65b5085c-6947-4858-9f3f-b27f5b08baf8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:17:48.275512 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:48.275485    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65b5085c-6947-4858-9f3f-b27f5b08baf8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "65b5085c-6947-4858-9f3f-b27f5b08baf8" (UID: "65b5085c-6947-4858-9f3f-b27f5b08baf8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:17:48.277135 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:48.277112    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65b5085c-6947-4858-9f3f-b27f5b08baf8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "65b5085c-6947-4858-9f3f-b27f5b08baf8" (UID: "65b5085c-6947-4858-9f3f-b27f5b08baf8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:17:48.277271 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:48.277153    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65b5085c-6947-4858-9f3f-b27f5b08baf8-kube-api-access-2gnmz" (OuterVolumeSpecName: "kube-api-access-2gnmz") pod "65b5085c-6947-4858-9f3f-b27f5b08baf8" (UID: "65b5085c-6947-4858-9f3f-b27f5b08baf8"). InnerVolumeSpecName "kube-api-access-2gnmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:17:48.285342 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:48.285316    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65b5085c-6947-4858-9f3f-b27f5b08baf8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "65b5085c-6947-4858-9f3f-b27f5b08baf8" (UID: "65b5085c-6947-4858-9f3f-b27f5b08baf8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:17:48.376463 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:48.376422    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65b5085c-6947-4858-9f3f-b27f5b08baf8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:17:48.376463 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:48.376458    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/65b5085c-6947-4858-9f3f-b27f5b08baf8-kserve-provision-location\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:17:48.376463 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:48.376471    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/65b5085c-6947-4858-9f3f-b27f5b08baf8-cabundle-cert\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:17:48.376721 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:48.376484    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65b5085c-6947-4858-9f3f-b27f5b08baf8-proxy-tls\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:17:48.376721 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:48.376494    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gnmz\" (UniqueName: \"kubernetes.io/projected/65b5085c-6947-4858-9f3f-b27f5b08baf8-kube-api-access-2gnmz\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:17:48.504970 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:48.504935    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8"]
Jul 27 18:17:48.509889 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:48.509862    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-5frv8"]
Jul 27 18:17:50.443780 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:50.443746    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-r8hc6"]
Jul 27 18:17:50.444162 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:50.444062    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65b5085c-6947-4858-9f3f-b27f5b08baf8" containerName="storage-initializer"
Jul 27 18:17:50.444162 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:50.444073    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="65b5085c-6947-4858-9f3f-b27f5b08baf8" containerName="storage-initializer"
Jul 27 18:17:50.444162 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:50.444083    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65b5085c-6947-4858-9f3f-b27f5b08baf8" containerName="kserve-container"
Jul 27 18:17:50.444162 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:50.444088    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="65b5085c-6947-4858-9f3f-b27f5b08baf8" containerName="kserve-container"
Jul 27 18:17:50.444162 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:50.444096    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65b5085c-6947-4858-9f3f-b27f5b08baf8" containerName="kube-rbac-proxy"
Jul 27 18:17:50.444162 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:50.444101    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="65b5085c-6947-4858-9f3f-b27f5b08baf8" containerName="kube-rbac-proxy"
Jul 27 18:17:50.444162 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:50.444153    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="65b5085c-6947-4858-9f3f-b27f5b08baf8" containerName="kube-rbac-proxy"
Jul 27 18:17:50.444162 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:50.444160    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="65b5085c-6947-4858-9f3f-b27f5b08baf8" containerName="kserve-container"
Jul 27 18:17:50.447077 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:50.447060    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-r8hc6"
Jul 27 18:17:50.450049 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:50.450028    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 18:17:50.450200 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:50.450027    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 18:17:50.450200 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:50.450081    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z6ld7\""
Jul 27 18:17:50.450200 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:50.450027    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
Jul 27 18:17:50.450200 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:50.450183    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 18:17:50.450200 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:50.450186    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 18:17:50.451183 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:50.451167    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-predictor-serving-cert\""
Jul 27 18:17:50.458366 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:50.458347    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-r8hc6"]
Jul 27 18:17:50.470651 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:50.470615    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65b5085c-6947-4858-9f3f-b27f5b08baf8" path="/var/lib/kubelet/pods/65b5085c-6947-4858-9f3f-b27f5b08baf8/volumes"
Jul 27 18:17:50.589507 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:50.589473    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc2d5075-5d42-4924-a873-c64dd46eafb4-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-r8hc6\" (UID: \"fc2d5075-5d42-4924-a873-c64dd46eafb4\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-r8hc6"
Jul 27 18:17:50.589685 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:50.589517    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc2d5075-5d42-4924-a873-c64dd46eafb4-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-r8hc6\" (UID: \"fc2d5075-5d42-4924-a873-c64dd46eafb4\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-r8hc6"
Jul 27 18:17:50.589685 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:50.589576    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc2d5075-5d42-4924-a873-c64dd46eafb4-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-r8hc6\" (UID: \"fc2d5075-5d42-4924-a873-c64dd46eafb4\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-r8hc6"
Jul 27 18:17:50.589685 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:50.589595    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc2d5075-5d42-4924-a873-c64dd46eafb4-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-r8hc6\" (UID: \"fc2d5075-5d42-4924-a873-c64dd46eafb4\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-r8hc6"
Jul 27 18:17:50.589685 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:50.589611    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mvvs\" (UniqueName: \"kubernetes.io/projected/fc2d5075-5d42-4924-a873-c64dd46eafb4-kube-api-access-7mvvs\") pod \"isvc-pmml-predictor-8674795f59-r8hc6\" (UID: \"fc2d5075-5d42-4924-a873-c64dd46eafb4\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-r8hc6"
Jul 27 18:17:50.690305 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:50.690267    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc2d5075-5d42-4924-a873-c64dd46eafb4-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-r8hc6\" (UID: \"fc2d5075-5d42-4924-a873-c64dd46eafb4\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-r8hc6"
Jul 27 18:17:50.690489 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:50.690360    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc2d5075-5d42-4924-a873-c64dd46eafb4-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-r8hc6\" (UID: \"fc2d5075-5d42-4924-a873-c64dd46eafb4\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-r8hc6"
Jul 27 18:17:50.690489 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:50.690401    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc2d5075-5d42-4924-a873-c64dd46eafb4-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-r8hc6\" (UID: \"fc2d5075-5d42-4924-a873-c64dd46eafb4\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-r8hc6"
Jul 27 18:17:50.690489 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:50.690422    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc2d5075-5d42-4924-a873-c64dd46eafb4-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-r8hc6\" (UID: \"fc2d5075-5d42-4924-a873-c64dd46eafb4\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-r8hc6"
Jul 27 18:17:50.690489 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:50.690439    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mvvs\" (UniqueName: \"kubernetes.io/projected/fc2d5075-5d42-4924-a873-c64dd46eafb4-kube-api-access-7mvvs\") pod \"isvc-pmml-predictor-8674795f59-r8hc6\" (UID: \"fc2d5075-5d42-4924-a873-c64dd46eafb4\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-r8hc6"
Jul 27 18:17:50.690750 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:50.690726    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc2d5075-5d42-4924-a873-c64dd46eafb4-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-r8hc6\" (UID: \"fc2d5075-5d42-4924-a873-c64dd46eafb4\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-r8hc6"
Jul 27 18:17:50.691018 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:50.691000    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc2d5075-5d42-4924-a873-c64dd46eafb4-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-r8hc6\" (UID: \"fc2d5075-5d42-4924-a873-c64dd46eafb4\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-r8hc6"
Jul 27 18:17:50.691094 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:50.691045    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc2d5075-5d42-4924-a873-c64dd46eafb4-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-r8hc6\" (UID: \"fc2d5075-5d42-4924-a873-c64dd46eafb4\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-r8hc6"
Jul 27 18:17:50.692902 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:50.692878    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc2d5075-5d42-4924-a873-c64dd46eafb4-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-r8hc6\" (UID: \"fc2d5075-5d42-4924-a873-c64dd46eafb4\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-r8hc6"
Jul 27 18:17:50.700466 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:50.700402    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mvvs\" (UniqueName: \"kubernetes.io/projected/fc2d5075-5d42-4924-a873-c64dd46eafb4-kube-api-access-7mvvs\") pod \"isvc-pmml-predictor-8674795f59-r8hc6\" (UID: \"fc2d5075-5d42-4924-a873-c64dd46eafb4\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-r8hc6"
Jul 27 18:17:50.757294 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:50.757230    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-r8hc6"
Jul 27 18:17:50.899040 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:50.899016    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-r8hc6"]
Jul 27 18:17:50.901546 ip-10-0-140-0 kubenswrapper[2573]: W0727 18:17:50.901517    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc2d5075_5d42_4924_a873_c64dd46eafb4.slice/crio-5299a70386984bb29e341ab4c1f7c6fd7a1d4f54da7a823eca908d91bbe8291b WatchSource:0}: Error finding container 5299a70386984bb29e341ab4c1f7c6fd7a1d4f54da7a823eca908d91bbe8291b: Status 404 returned error can't find the container with id 5299a70386984bb29e341ab4c1f7c6fd7a1d4f54da7a823eca908d91bbe8291b
Jul 27 18:17:51.202996 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:51.202961    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-r8hc6" event={"ID":"fc2d5075-5d42-4924-a873-c64dd46eafb4","Type":"ContainerStarted","Data":"4b2ffb378c08d1d4256cf69869882cc7916126800b46f0b9c973cbd8431418d4"}
Jul 27 18:17:51.203160 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:51.203046    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-r8hc6" event={"ID":"fc2d5075-5d42-4924-a873-c64dd46eafb4","Type":"ContainerStarted","Data":"5299a70386984bb29e341ab4c1f7c6fd7a1d4f54da7a823eca908d91bbe8291b"}
Jul 27 18:17:51.341426 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:51.341351    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:17:51.398582 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:51.398546    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:17:55.221718 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:55.221685    2573 generic.go:356] "Generic (PLEG): container finished" podID="fc2d5075-5d42-4924-a873-c64dd46eafb4" containerID="4b2ffb378c08d1d4256cf69869882cc7916126800b46f0b9c973cbd8431418d4" exitCode=0
Jul 27 18:17:55.222201 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:55.221733    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-r8hc6" event={"ID":"fc2d5075-5d42-4924-a873-c64dd46eafb4","Type":"ContainerDied","Data":"4b2ffb378c08d1d4256cf69869882cc7916126800b46f0b9c973cbd8431418d4"}
Jul 27 18:17:55.223160 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:17:55.223141    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 18:18:03.275338 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:18:03.275302    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-r8hc6" event={"ID":"fc2d5075-5d42-4924-a873-c64dd46eafb4","Type":"ContainerStarted","Data":"23e5c2867efa399316c3c1490c22ab25843c6caf52019aef8560ced0a593fc8b"}
Jul 27 18:18:03.275743 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:18:03.275368    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-r8hc6" event={"ID":"fc2d5075-5d42-4924-a873-c64dd46eafb4","Type":"ContainerStarted","Data":"156a421e2f915918923bb106bf8910f429ce7c379284590225aa757de438c4fc"}
Jul 27 18:18:03.275743 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:18:03.275563    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-r8hc6"
Jul 27 18:18:03.294846 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:18:03.294794    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-r8hc6" podStartSLOduration=2.6011199769999997 podStartE2EDuration="13.294777826s" totalImagesPullingTime="7.046692888s" totalInitContainerRuntime="3.646964961s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:17:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:18:03.294237323 +0000 UTC m=+3025.437639374" watchObservedRunningTime="2026-07-27 18:18:03.294777826 +0000 UTC m=+3025.438179878"
Jul 27 18:18:04.280422 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:18:04.280386    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-r8hc6"
Jul 27 18:18:04.281861 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:18:04.281833    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-r8hc6" podUID="fc2d5075-5d42-4924-a873-c64dd46eafb4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 27 18:18:05.284500 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:18:05.284463    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-r8hc6" podUID="fc2d5075-5d42-4924-a873-c64dd46eafb4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 27 18:18:10.289161 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:18:10.289126    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-r8hc6"
Jul 27 18:18:10.289781 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:18:10.289752    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-r8hc6" podUID="fc2d5075-5d42-4924-a873-c64dd46eafb4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 27 18:18:20.290193 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:18:20.290144    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-r8hc6" podUID="fc2d5075-5d42-4924-a873-c64dd46eafb4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 27 18:18:30.290542 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:18:30.290437    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-r8hc6" podUID="fc2d5075-5d42-4924-a873-c64dd46eafb4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 27 18:18:40.289758 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:18:40.289711    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-r8hc6" podUID="fc2d5075-5d42-4924-a873-c64dd46eafb4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 27 18:18:44.932378 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:18:44.932342    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:18:50.290418 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:18:50.290375    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-r8hc6" podUID="fc2d5075-5d42-4924-a873-c64dd46eafb4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 27 18:19:00.289963 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:00.289922    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-r8hc6" podUID="fc2d5075-5d42-4924-a873-c64dd46eafb4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 27 18:19:10.290604 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:10.290556    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-r8hc6" podUID="fc2d5075-5d42-4924-a873-c64dd46eafb4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 27 18:19:20.289902 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:20.289858    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-r8hc6" podUID="fc2d5075-5d42-4924-a873-c64dd46eafb4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.54:8080: connect: connection refused"
Jul 27 18:19:30.290429 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:30.290397    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-r8hc6"
Jul 27 18:19:41.678292 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:41.678257    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-r8hc6"]
Jul 27 18:19:41.678823 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:41.678536    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-r8hc6" podUID="fc2d5075-5d42-4924-a873-c64dd46eafb4" containerName="kserve-container" containerID="cri-o://156a421e2f915918923bb106bf8910f429ce7c379284590225aa757de438c4fc" gracePeriod=30
Jul 27 18:19:41.678823 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:41.678554    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-r8hc6" podUID="fc2d5075-5d42-4924-a873-c64dd46eafb4" containerName="kube-rbac-proxy" containerID="cri-o://23e5c2867efa399316c3c1490c22ab25843c6caf52019aef8560ced0a593fc8b" gracePeriod=30
Jul 27 18:19:41.703149 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:41.703122    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:19:42.709583 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:42.709551    2573 generic.go:356] "Generic (PLEG): container finished" podID="fc2d5075-5d42-4924-a873-c64dd46eafb4" containerID="23e5c2867efa399316c3c1490c22ab25843c6caf52019aef8560ced0a593fc8b" exitCode=2
Jul 27 18:19:42.709972 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:42.709627    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-r8hc6" event={"ID":"fc2d5075-5d42-4924-a873-c64dd46eafb4","Type":"ContainerDied","Data":"23e5c2867efa399316c3c1490c22ab25843c6caf52019aef8560ced0a593fc8b"}
Jul 27 18:19:45.285409 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:45.285371    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-r8hc6" podUID="fc2d5075-5d42-4924-a873-c64dd46eafb4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.54:8643/healthz\": dial tcp 10.133.0.54:8643: connect: connection refused"
Jul 27 18:19:45.464753 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:45.464726    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-r8hc6"
Jul 27 18:19:45.565597 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:45.565559    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc2d5075-5d42-4924-a873-c64dd46eafb4-kube-rbac-proxy-sar-config\") pod \"fc2d5075-5d42-4924-a873-c64dd46eafb4\" (UID: \"fc2d5075-5d42-4924-a873-c64dd46eafb4\") "
Jul 27 18:19:45.565772 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:45.565610    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc2d5075-5d42-4924-a873-c64dd46eafb4-proxy-tls\") pod \"fc2d5075-5d42-4924-a873-c64dd46eafb4\" (UID: \"fc2d5075-5d42-4924-a873-c64dd46eafb4\") "
Jul 27 18:19:45.565772 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:45.565644    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mvvs\" (UniqueName: \"kubernetes.io/projected/fc2d5075-5d42-4924-a873-c64dd46eafb4-kube-api-access-7mvvs\") pod \"fc2d5075-5d42-4924-a873-c64dd46eafb4\" (UID: \"fc2d5075-5d42-4924-a873-c64dd46eafb4\") "
Jul 27 18:19:45.565854 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:45.565824    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc2d5075-5d42-4924-a873-c64dd46eafb4-kserve-provision-location\") pod \"fc2d5075-5d42-4924-a873-c64dd46eafb4\" (UID: \"fc2d5075-5d42-4924-a873-c64dd46eafb4\") "
Jul 27 18:19:45.565913 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:45.565897    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc2d5075-5d42-4924-a873-c64dd46eafb4-cabundle-cert\") pod \"fc2d5075-5d42-4924-a873-c64dd46eafb4\" (UID: \"fc2d5075-5d42-4924-a873-c64dd46eafb4\") "
Jul 27 18:19:45.566098 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:45.566069    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc2d5075-5d42-4924-a873-c64dd46eafb4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fc2d5075-5d42-4924-a873-c64dd46eafb4" (UID: "fc2d5075-5d42-4924-a873-c64dd46eafb4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:19:45.566098 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:45.566082    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc2d5075-5d42-4924-a873-c64dd46eafb4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fc2d5075-5d42-4924-a873-c64dd46eafb4" (UID: "fc2d5075-5d42-4924-a873-c64dd46eafb4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:19:45.566317 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:45.566155    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc2d5075-5d42-4924-a873-c64dd46eafb4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:19:45.566317 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:45.566175    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc2d5075-5d42-4924-a873-c64dd46eafb4-kserve-provision-location\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:19:45.566398 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:45.566346    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc2d5075-5d42-4924-a873-c64dd46eafb4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fc2d5075-5d42-4924-a873-c64dd46eafb4" (UID: "fc2d5075-5d42-4924-a873-c64dd46eafb4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:19:45.567754 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:45.567733    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc2d5075-5d42-4924-a873-c64dd46eafb4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fc2d5075-5d42-4924-a873-c64dd46eafb4" (UID: "fc2d5075-5d42-4924-a873-c64dd46eafb4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:19:45.567861 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:45.567844    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc2d5075-5d42-4924-a873-c64dd46eafb4-kube-api-access-7mvvs" (OuterVolumeSpecName: "kube-api-access-7mvvs") pod "fc2d5075-5d42-4924-a873-c64dd46eafb4" (UID: "fc2d5075-5d42-4924-a873-c64dd46eafb4"). InnerVolumeSpecName "kube-api-access-7mvvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:19:45.667140 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:45.667102    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mvvs\" (UniqueName: \"kubernetes.io/projected/fc2d5075-5d42-4924-a873-c64dd46eafb4-kube-api-access-7mvvs\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:19:45.667140 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:45.667131    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc2d5075-5d42-4924-a873-c64dd46eafb4-cabundle-cert\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:19:45.667140 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:45.667140    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc2d5075-5d42-4924-a873-c64dd46eafb4-proxy-tls\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:19:45.723148 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:45.723064    2573 generic.go:356] "Generic (PLEG): container finished" podID="fc2d5075-5d42-4924-a873-c64dd46eafb4" containerID="156a421e2f915918923bb106bf8910f429ce7c379284590225aa757de438c4fc" exitCode=0
Jul 27 18:19:45.723148 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:45.723113    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-r8hc6" event={"ID":"fc2d5075-5d42-4924-a873-c64dd46eafb4","Type":"ContainerDied","Data":"156a421e2f915918923bb106bf8910f429ce7c379284590225aa757de438c4fc"}
Jul 27 18:19:45.723148 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:45.723133    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-r8hc6"
Jul 27 18:19:45.723393 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:45.723177    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-r8hc6" event={"ID":"fc2d5075-5d42-4924-a873-c64dd46eafb4","Type":"ContainerDied","Data":"5299a70386984bb29e341ab4c1f7c6fd7a1d4f54da7a823eca908d91bbe8291b"}
Jul 27 18:19:45.723393 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:45.723214    2573 scope.go:117] "RemoveContainer" containerID="23e5c2867efa399316c3c1490c22ab25843c6caf52019aef8560ced0a593fc8b"
Jul 27 18:19:45.732096 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:45.732072    2573 scope.go:117] "RemoveContainer" containerID="156a421e2f915918923bb106bf8910f429ce7c379284590225aa757de438c4fc"
Jul 27 18:19:45.740105 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:45.740083    2573 scope.go:117] "RemoveContainer" containerID="4b2ffb378c08d1d4256cf69869882cc7916126800b46f0b9c973cbd8431418d4"
Jul 27 18:19:45.745638 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:45.745613    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-r8hc6"]
Jul 27 18:19:45.747599 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:45.747582    2573 scope.go:117] "RemoveContainer" containerID="23e5c2867efa399316c3c1490c22ab25843c6caf52019aef8560ced0a593fc8b"
Jul 27 18:19:45.747872 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:19:45.747850    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23e5c2867efa399316c3c1490c22ab25843c6caf52019aef8560ced0a593fc8b\": container with ID starting with 23e5c2867efa399316c3c1490c22ab25843c6caf52019aef8560ced0a593fc8b not found: ID does not exist" containerID="23e5c2867efa399316c3c1490c22ab25843c6caf52019aef8560ced0a593fc8b"
Jul 27 18:19:45.747942 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:45.747885    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23e5c2867efa399316c3c1490c22ab25843c6caf52019aef8560ced0a593fc8b"} err="failed to get container status \"23e5c2867efa399316c3c1490c22ab25843c6caf52019aef8560ced0a593fc8b\": rpc error: code = NotFound desc = could not find container \"23e5c2867efa399316c3c1490c22ab25843c6caf52019aef8560ced0a593fc8b\": container with ID starting with 23e5c2867efa399316c3c1490c22ab25843c6caf52019aef8560ced0a593fc8b not found: ID does not exist"
Jul 27 18:19:45.747942 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:45.747910    2573 scope.go:117] "RemoveContainer" containerID="156a421e2f915918923bb106bf8910f429ce7c379284590225aa757de438c4fc"
Jul 27 18:19:45.748156 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:19:45.748136    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"156a421e2f915918923bb106bf8910f429ce7c379284590225aa757de438c4fc\": container with ID starting with 156a421e2f915918923bb106bf8910f429ce7c379284590225aa757de438c4fc not found: ID does not exist" containerID="156a421e2f915918923bb106bf8910f429ce7c379284590225aa757de438c4fc"
Jul 27 18:19:45.748199 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:45.748163    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"156a421e2f915918923bb106bf8910f429ce7c379284590225aa757de438c4fc"} err="failed to get container status \"156a421e2f915918923bb106bf8910f429ce7c379284590225aa757de438c4fc\": rpc error: code = NotFound desc = could not find container \"156a421e2f915918923bb106bf8910f429ce7c379284590225aa757de438c4fc\": container with ID starting with 156a421e2f915918923bb106bf8910f429ce7c379284590225aa757de438c4fc not found: ID does not exist"
Jul 27 18:19:45.748199 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:45.748179    2573 scope.go:117] "RemoveContainer" containerID="4b2ffb378c08d1d4256cf69869882cc7916126800b46f0b9c973cbd8431418d4"
Jul 27 18:19:45.748444 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:19:45.748418    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b2ffb378c08d1d4256cf69869882cc7916126800b46f0b9c973cbd8431418d4\": container with ID starting with 4b2ffb378c08d1d4256cf69869882cc7916126800b46f0b9c973cbd8431418d4 not found: ID does not exist" containerID="4b2ffb378c08d1d4256cf69869882cc7916126800b46f0b9c973cbd8431418d4"
Jul 27 18:19:45.748508 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:45.748444    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b2ffb378c08d1d4256cf69869882cc7916126800b46f0b9c973cbd8431418d4"} err="failed to get container status \"4b2ffb378c08d1d4256cf69869882cc7916126800b46f0b9c973cbd8431418d4\": rpc error: code = NotFound desc = could not find container \"4b2ffb378c08d1d4256cf69869882cc7916126800b46f0b9c973cbd8431418d4\": container with ID starting with 4b2ffb378c08d1d4256cf69869882cc7916126800b46f0b9c973cbd8431418d4 not found: ID does not exist"
Jul 27 18:19:45.751458 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:45.751436    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-r8hc6"]
Jul 27 18:19:46.468853 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:46.468822    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc2d5075-5d42-4924-a873-c64dd46eafb4" path="/var/lib/kubelet/pods/fc2d5075-5d42-4924-a873-c64dd46eafb4/volumes"
Jul 27 18:19:46.772704 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:46.772613    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:19:46.835628 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:46.835592    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-n7l2t"]
Jul 27 18:19:46.835938 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:46.835925    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc2d5075-5d42-4924-a873-c64dd46eafb4" containerName="kube-rbac-proxy"
Jul 27 18:19:46.835981 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:46.835940    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc2d5075-5d42-4924-a873-c64dd46eafb4" containerName="kube-rbac-proxy"
Jul 27 18:19:46.835981 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:46.835958    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc2d5075-5d42-4924-a873-c64dd46eafb4" containerName="kserve-container"
Jul 27 18:19:46.835981 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:46.835964    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc2d5075-5d42-4924-a873-c64dd46eafb4" containerName="kserve-container"
Jul 27 18:19:46.835981 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:46.835977    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc2d5075-5d42-4924-a873-c64dd46eafb4" containerName="storage-initializer"
Jul 27 18:19:46.836112 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:46.835982    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc2d5075-5d42-4924-a873-c64dd46eafb4" containerName="storage-initializer"
Jul 27 18:19:46.836112 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:46.836033    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc2d5075-5d42-4924-a873-c64dd46eafb4" containerName="kserve-container"
Jul 27 18:19:46.836112 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:46.836040    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc2d5075-5d42-4924-a873-c64dd46eafb4" containerName="kube-rbac-proxy"
Jul 27 18:19:46.840266 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:46.840235    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-n7l2t"
Jul 27 18:19:46.843185 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:46.843152    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 18:19:46.844571 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:46.844538    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
Jul 27 18:19:46.844714 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:46.844603    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z6ld7\""
Jul 27 18:19:46.844714 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:46.844636    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 18:19:46.844714 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:46.844549    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 18:19:46.844714 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:46.844703    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 18:19:46.844977 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:46.844720    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-predictor-serving-cert\""
Jul 27 18:19:46.854333 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:46.854306    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-n7l2t"]
Jul 27 18:19:46.876008 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:46.875962    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzzll\" (UniqueName: \"kubernetes.io/projected/03a86fa0-04d7-47d6-90f3-f055a39bdd87-kube-api-access-hzzll\") pod \"isvc-pmml-runtime-predictor-885fc8777-n7l2t\" (UID: \"03a86fa0-04d7-47d6-90f3-f055a39bdd87\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-n7l2t"
Jul 27 18:19:46.876008 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:46.876006    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03a86fa0-04d7-47d6-90f3-f055a39bdd87-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-n7l2t\" (UID: \"03a86fa0-04d7-47d6-90f3-f055a39bdd87\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-n7l2t"
Jul 27 18:19:46.876235 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:46.876026    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/03a86fa0-04d7-47d6-90f3-f055a39bdd87-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-n7l2t\" (UID: \"03a86fa0-04d7-47d6-90f3-f055a39bdd87\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-n7l2t"
Jul 27 18:19:46.876235 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:46.876104    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03a86fa0-04d7-47d6-90f3-f055a39bdd87-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-n7l2t\" (UID: \"03a86fa0-04d7-47d6-90f3-f055a39bdd87\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-n7l2t"
Jul 27 18:19:46.876235 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:46.876133    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03a86fa0-04d7-47d6-90f3-f055a39bdd87-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-n7l2t\" (UID: \"03a86fa0-04d7-47d6-90f3-f055a39bdd87\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-n7l2t"
Jul 27 18:19:46.918535 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:46.918501    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:19:46.977473 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:46.977431    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzzll\" (UniqueName: \"kubernetes.io/projected/03a86fa0-04d7-47d6-90f3-f055a39bdd87-kube-api-access-hzzll\") pod \"isvc-pmml-runtime-predictor-885fc8777-n7l2t\" (UID: \"03a86fa0-04d7-47d6-90f3-f055a39bdd87\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-n7l2t"
Jul 27 18:19:46.977473 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:46.977471    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03a86fa0-04d7-47d6-90f3-f055a39bdd87-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-n7l2t\" (UID: \"03a86fa0-04d7-47d6-90f3-f055a39bdd87\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-n7l2t"
Jul 27 18:19:46.977824 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:46.977497    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/03a86fa0-04d7-47d6-90f3-f055a39bdd87-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-n7l2t\" (UID: \"03a86fa0-04d7-47d6-90f3-f055a39bdd87\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-n7l2t"
Jul 27 18:19:46.977824 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:46.977540    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03a86fa0-04d7-47d6-90f3-f055a39bdd87-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-n7l2t\" (UID: \"03a86fa0-04d7-47d6-90f3-f055a39bdd87\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-n7l2t"
Jul 27 18:19:46.977824 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:46.977620    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03a86fa0-04d7-47d6-90f3-f055a39bdd87-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-n7l2t\" (UID: \"03a86fa0-04d7-47d6-90f3-f055a39bdd87\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-n7l2t"
Jul 27 18:19:46.978032 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:46.978012    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03a86fa0-04d7-47d6-90f3-f055a39bdd87-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-n7l2t\" (UID: \"03a86fa0-04d7-47d6-90f3-f055a39bdd87\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-n7l2t"
Jul 27 18:19:46.978354 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:46.978330    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03a86fa0-04d7-47d6-90f3-f055a39bdd87-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-n7l2t\" (UID: \"03a86fa0-04d7-47d6-90f3-f055a39bdd87\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-n7l2t"
Jul 27 18:19:46.978354 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:46.978343    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/03a86fa0-04d7-47d6-90f3-f055a39bdd87-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-n7l2t\" (UID: \"03a86fa0-04d7-47d6-90f3-f055a39bdd87\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-n7l2t"
Jul 27 18:19:46.980182 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:46.980164    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03a86fa0-04d7-47d6-90f3-f055a39bdd87-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-n7l2t\" (UID: \"03a86fa0-04d7-47d6-90f3-f055a39bdd87\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-n7l2t"
Jul 27 18:19:46.986874 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:46.986847    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzzll\" (UniqueName: \"kubernetes.io/projected/03a86fa0-04d7-47d6-90f3-f055a39bdd87-kube-api-access-hzzll\") pod \"isvc-pmml-runtime-predictor-885fc8777-n7l2t\" (UID: \"03a86fa0-04d7-47d6-90f3-f055a39bdd87\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-n7l2t"
Jul 27 18:19:47.151162 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:47.151119    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-n7l2t"
Jul 27 18:19:47.292449 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:47.292424    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-n7l2t"]
Jul 27 18:19:47.295058 ip-10-0-140-0 kubenswrapper[2573]: W0727 18:19:47.295023    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03a86fa0_04d7_47d6_90f3_f055a39bdd87.slice/crio-ef0f00b5d16e52bff6106867b6ae8eacc3850d4cce7b1e97067fc0fe43f595e6 WatchSource:0}: Error finding container ef0f00b5d16e52bff6106867b6ae8eacc3850d4cce7b1e97067fc0fe43f595e6: Status 404 returned error can't find the container with id ef0f00b5d16e52bff6106867b6ae8eacc3850d4cce7b1e97067fc0fe43f595e6
Jul 27 18:19:47.740706 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:47.740667    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-n7l2t" event={"ID":"03a86fa0-04d7-47d6-90f3-f055a39bdd87","Type":"ContainerStarted","Data":"be9b3a34ac30c1e12aafa3002cdb2643102f323faff32a30c6e949cc1cb312cd"}
Jul 27 18:19:47.741114 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:47.740724    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-n7l2t" event={"ID":"03a86fa0-04d7-47d6-90f3-f055a39bdd87","Type":"ContainerStarted","Data":"ef0f00b5d16e52bff6106867b6ae8eacc3850d4cce7b1e97067fc0fe43f595e6"}
Jul 27 18:19:51.758509 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:51.758476    2573 generic.go:356] "Generic (PLEG): container finished" podID="03a86fa0-04d7-47d6-90f3-f055a39bdd87" containerID="be9b3a34ac30c1e12aafa3002cdb2643102f323faff32a30c6e949cc1cb312cd" exitCode=0
Jul 27 18:19:51.758913 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:51.758557    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-n7l2t" event={"ID":"03a86fa0-04d7-47d6-90f3-f055a39bdd87","Type":"ContainerDied","Data":"be9b3a34ac30c1e12aafa3002cdb2643102f323faff32a30c6e949cc1cb312cd"}
Jul 27 18:19:52.765231 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:52.765198    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-n7l2t" event={"ID":"03a86fa0-04d7-47d6-90f3-f055a39bdd87","Type":"ContainerStarted","Data":"a2b0c97c5f3ca8d202bd74886fdb7b004580ca1905043ed4ed9f340e8fc9223a"}
Jul 27 18:19:52.765625 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:52.765279    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-n7l2t" event={"ID":"03a86fa0-04d7-47d6-90f3-f055a39bdd87","Type":"ContainerStarted","Data":"afbf247b9663efe32acf3a74200706b5f76a43b04a84a292ffbfcb86614240cb"}
Jul 27 18:19:52.765625 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:52.765472    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-n7l2t"
Jul 27 18:19:52.787645 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:52.787590    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-n7l2t" podStartSLOduration=2.992837839 podStartE2EDuration="6.787575648s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.794737809s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:19:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:19:52.786589008 +0000 UTC m=+3134.929991082" watchObservedRunningTime="2026-07-27 18:19:52.787575648 +0000 UTC m=+3134.930977699"
Jul 27 18:19:53.769951 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:53.769918    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-n7l2t"
Jul 27 18:19:53.771165 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:53.771136    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-n7l2t" podUID="03a86fa0-04d7-47d6-90f3-f055a39bdd87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Jul 27 18:19:54.773512 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:54.773478    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-n7l2t" podUID="03a86fa0-04d7-47d6-90f3-f055a39bdd87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Jul 27 18:19:59.778919 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:59.778843    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-n7l2t"
Jul 27 18:19:59.779564 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:19:59.779536    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-n7l2t" podUID="03a86fa0-04d7-47d6-90f3-f055a39bdd87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Jul 27 18:20:09.780289 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:20:09.780233    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-n7l2t" podUID="03a86fa0-04d7-47d6-90f3-f055a39bdd87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Jul 27 18:20:19.780336 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:20:19.780295    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-n7l2t" podUID="03a86fa0-04d7-47d6-90f3-f055a39bdd87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Jul 27 18:20:29.779779 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:20:29.779734    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-n7l2t" podUID="03a86fa0-04d7-47d6-90f3-f055a39bdd87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Jul 27 18:20:39.780301 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:20:39.780235    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-n7l2t" podUID="03a86fa0-04d7-47d6-90f3-f055a39bdd87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Jul 27 18:20:49.779577 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:20:49.779537    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-n7l2t" podUID="03a86fa0-04d7-47d6-90f3-f055a39bdd87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Jul 27 18:20:59.779477 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:20:59.779425    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-n7l2t" podUID="03a86fa0-04d7-47d6-90f3-f055a39bdd87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Jul 27 18:21:09.780027 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:09.779986    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-n7l2t" podUID="03a86fa0-04d7-47d6-90f3-f055a39bdd87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Jul 27 18:21:19.780305 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:19.780273    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-n7l2t"
Jul 27 18:21:27.852845 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:27.852812    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-n7l2t"]
Jul 27 18:21:27.853314 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:27.853055    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-n7l2t" podUID="03a86fa0-04d7-47d6-90f3-f055a39bdd87" containerName="kserve-container" containerID="cri-o://afbf247b9663efe32acf3a74200706b5f76a43b04a84a292ffbfcb86614240cb" gracePeriod=30
Jul 27 18:21:27.853314 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:27.853094    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-n7l2t" podUID="03a86fa0-04d7-47d6-90f3-f055a39bdd87" containerName="kube-rbac-proxy" containerID="cri-o://a2b0c97c5f3ca8d202bd74886fdb7b004580ca1905043ed4ed9f340e8fc9223a" gracePeriod=30
Jul 27 18:21:27.889504 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:27.889474    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:21:28.181016 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:28.180987    2573 generic.go:356] "Generic (PLEG): container finished" podID="03a86fa0-04d7-47d6-90f3-f055a39bdd87" containerID="a2b0c97c5f3ca8d202bd74886fdb7b004580ca1905043ed4ed9f340e8fc9223a" exitCode=2
Jul 27 18:21:28.181207 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:28.181054    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-n7l2t" event={"ID":"03a86fa0-04d7-47d6-90f3-f055a39bdd87","Type":"ContainerDied","Data":"a2b0c97c5f3ca8d202bd74886fdb7b004580ca1905043ed4ed9f340e8fc9223a"}
Jul 27 18:21:29.774242 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:29.774147    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-n7l2t" podUID="03a86fa0-04d7-47d6-90f3-f055a39bdd87" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.55:8643/healthz\": dial tcp 10.133.0.55:8643: connect: connection refused"
Jul 27 18:21:29.779552 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:29.779523    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-n7l2t" podUID="03a86fa0-04d7-47d6-90f3-f055a39bdd87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Jul 27 18:21:31.701568 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:31.701541    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-n7l2t"
Jul 27 18:21:31.859750 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:31.859661    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/03a86fa0-04d7-47d6-90f3-f055a39bdd87-cabundle-cert\") pod \"03a86fa0-04d7-47d6-90f3-f055a39bdd87\" (UID: \"03a86fa0-04d7-47d6-90f3-f055a39bdd87\") "
Jul 27 18:21:31.859750 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:31.859747    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03a86fa0-04d7-47d6-90f3-f055a39bdd87-proxy-tls\") pod \"03a86fa0-04d7-47d6-90f3-f055a39bdd87\" (UID: \"03a86fa0-04d7-47d6-90f3-f055a39bdd87\") "
Jul 27 18:21:31.859984 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:31.859773    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03a86fa0-04d7-47d6-90f3-f055a39bdd87-kube-rbac-proxy-sar-config\") pod \"03a86fa0-04d7-47d6-90f3-f055a39bdd87\" (UID: \"03a86fa0-04d7-47d6-90f3-f055a39bdd87\") "
Jul 27 18:21:31.859984 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:31.859796    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03a86fa0-04d7-47d6-90f3-f055a39bdd87-kserve-provision-location\") pod \"03a86fa0-04d7-47d6-90f3-f055a39bdd87\" (UID: \"03a86fa0-04d7-47d6-90f3-f055a39bdd87\") "
Jul 27 18:21:31.859984 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:31.859823    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzzll\" (UniqueName: \"kubernetes.io/projected/03a86fa0-04d7-47d6-90f3-f055a39bdd87-kube-api-access-hzzll\") pod \"03a86fa0-04d7-47d6-90f3-f055a39bdd87\" (UID: \"03a86fa0-04d7-47d6-90f3-f055a39bdd87\") "
Jul 27 18:21:31.860151 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:31.860083    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03a86fa0-04d7-47d6-90f3-f055a39bdd87-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "03a86fa0-04d7-47d6-90f3-f055a39bdd87" (UID: "03a86fa0-04d7-47d6-90f3-f055a39bdd87"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:21:31.860280 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:31.860229    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03a86fa0-04d7-47d6-90f3-f055a39bdd87-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "03a86fa0-04d7-47d6-90f3-f055a39bdd87" (UID: "03a86fa0-04d7-47d6-90f3-f055a39bdd87"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:21:31.860357 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:31.860232    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03a86fa0-04d7-47d6-90f3-f055a39bdd87-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "03a86fa0-04d7-47d6-90f3-f055a39bdd87" (UID: "03a86fa0-04d7-47d6-90f3-f055a39bdd87"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:21:31.861962 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:31.861929    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03a86fa0-04d7-47d6-90f3-f055a39bdd87-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "03a86fa0-04d7-47d6-90f3-f055a39bdd87" (UID: "03a86fa0-04d7-47d6-90f3-f055a39bdd87"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:21:31.862072 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:31.861971    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03a86fa0-04d7-47d6-90f3-f055a39bdd87-kube-api-access-hzzll" (OuterVolumeSpecName: "kube-api-access-hzzll") pod "03a86fa0-04d7-47d6-90f3-f055a39bdd87" (UID: "03a86fa0-04d7-47d6-90f3-f055a39bdd87"). InnerVolumeSpecName "kube-api-access-hzzll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:21:31.960586 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:31.960542    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzzll\" (UniqueName: \"kubernetes.io/projected/03a86fa0-04d7-47d6-90f3-f055a39bdd87-kube-api-access-hzzll\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:21:31.960586 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:31.960581    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/03a86fa0-04d7-47d6-90f3-f055a39bdd87-cabundle-cert\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:21:31.960586 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:31.960594    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03a86fa0-04d7-47d6-90f3-f055a39bdd87-proxy-tls\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:21:31.960879 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:31.960605    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03a86fa0-04d7-47d6-90f3-f055a39bdd87-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:21:31.960879 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:31.960617    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/03a86fa0-04d7-47d6-90f3-f055a39bdd87-kserve-provision-location\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:21:32.198453 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:32.198414    2573 generic.go:356] "Generic (PLEG): container finished" podID="03a86fa0-04d7-47d6-90f3-f055a39bdd87" containerID="afbf247b9663efe32acf3a74200706b5f76a43b04a84a292ffbfcb86614240cb" exitCode=0
Jul 27 18:21:32.198630 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:32.198471    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-n7l2t"
Jul 27 18:21:32.198630 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:32.198497    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-n7l2t" event={"ID":"03a86fa0-04d7-47d6-90f3-f055a39bdd87","Type":"ContainerDied","Data":"afbf247b9663efe32acf3a74200706b5f76a43b04a84a292ffbfcb86614240cb"}
Jul 27 18:21:32.198630 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:32.198588    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-n7l2t" event={"ID":"03a86fa0-04d7-47d6-90f3-f055a39bdd87","Type":"ContainerDied","Data":"ef0f00b5d16e52bff6106867b6ae8eacc3850d4cce7b1e97067fc0fe43f595e6"}
Jul 27 18:21:32.198744 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:32.198634    2573 scope.go:117] "RemoveContainer" containerID="a2b0c97c5f3ca8d202bd74886fdb7b004580ca1905043ed4ed9f340e8fc9223a"
Jul 27 18:21:32.207689 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:32.207668    2573 scope.go:117] "RemoveContainer" containerID="afbf247b9663efe32acf3a74200706b5f76a43b04a84a292ffbfcb86614240cb"
Jul 27 18:21:32.215496 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:32.215477    2573 scope.go:117] "RemoveContainer" containerID="be9b3a34ac30c1e12aafa3002cdb2643102f323faff32a30c6e949cc1cb312cd"
Jul 27 18:21:32.220464 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:32.220437    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-n7l2t"]
Jul 27 18:21:32.223833 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:32.223813    2573 scope.go:117] "RemoveContainer" containerID="a2b0c97c5f3ca8d202bd74886fdb7b004580ca1905043ed4ed9f340e8fc9223a"
Jul 27 18:21:32.224113 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:21:32.224092    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2b0c97c5f3ca8d202bd74886fdb7b004580ca1905043ed4ed9f340e8fc9223a\": container with ID starting with a2b0c97c5f3ca8d202bd74886fdb7b004580ca1905043ed4ed9f340e8fc9223a not found: ID does not exist" containerID="a2b0c97c5f3ca8d202bd74886fdb7b004580ca1905043ed4ed9f340e8fc9223a"
Jul 27 18:21:32.224197 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:32.224126    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2b0c97c5f3ca8d202bd74886fdb7b004580ca1905043ed4ed9f340e8fc9223a"} err="failed to get container status \"a2b0c97c5f3ca8d202bd74886fdb7b004580ca1905043ed4ed9f340e8fc9223a\": rpc error: code = NotFound desc = could not find container \"a2b0c97c5f3ca8d202bd74886fdb7b004580ca1905043ed4ed9f340e8fc9223a\": container with ID starting with a2b0c97c5f3ca8d202bd74886fdb7b004580ca1905043ed4ed9f340e8fc9223a not found: ID does not exist"
Jul 27 18:21:32.224197 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:32.224152    2573 scope.go:117] "RemoveContainer" containerID="afbf247b9663efe32acf3a74200706b5f76a43b04a84a292ffbfcb86614240cb"
Jul 27 18:21:32.224348 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:32.224323    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-n7l2t"]
Jul 27 18:21:32.224483 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:21:32.224466    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afbf247b9663efe32acf3a74200706b5f76a43b04a84a292ffbfcb86614240cb\": container with ID starting with afbf247b9663efe32acf3a74200706b5f76a43b04a84a292ffbfcb86614240cb not found: ID does not exist" containerID="afbf247b9663efe32acf3a74200706b5f76a43b04a84a292ffbfcb86614240cb"
Jul 27 18:21:32.224535 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:32.224497    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afbf247b9663efe32acf3a74200706b5f76a43b04a84a292ffbfcb86614240cb"} err="failed to get container status \"afbf247b9663efe32acf3a74200706b5f76a43b04a84a292ffbfcb86614240cb\": rpc error: code = NotFound desc = could not find container \"afbf247b9663efe32acf3a74200706b5f76a43b04a84a292ffbfcb86614240cb\": container with ID starting with afbf247b9663efe32acf3a74200706b5f76a43b04a84a292ffbfcb86614240cb not found: ID does not exist"
Jul 27 18:21:32.224535 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:32.224515    2573 scope.go:117] "RemoveContainer" containerID="be9b3a34ac30c1e12aafa3002cdb2643102f323faff32a30c6e949cc1cb312cd"
Jul 27 18:21:32.224775 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:21:32.224757    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be9b3a34ac30c1e12aafa3002cdb2643102f323faff32a30c6e949cc1cb312cd\": container with ID starting with be9b3a34ac30c1e12aafa3002cdb2643102f323faff32a30c6e949cc1cb312cd not found: ID does not exist" containerID="be9b3a34ac30c1e12aafa3002cdb2643102f323faff32a30c6e949cc1cb312cd"
Jul 27 18:21:32.224821 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:32.224782    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be9b3a34ac30c1e12aafa3002cdb2643102f323faff32a30c6e949cc1cb312cd"} err="failed to get container status \"be9b3a34ac30c1e12aafa3002cdb2643102f323faff32a30c6e949cc1cb312cd\": rpc error: code = NotFound desc = could not find container \"be9b3a34ac30c1e12aafa3002cdb2643102f323faff32a30c6e949cc1cb312cd\": container with ID starting with be9b3a34ac30c1e12aafa3002cdb2643102f323faff32a30c6e949cc1cb312cd not found: ID does not exist"
Jul 27 18:21:32.469043 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:32.468962    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03a86fa0-04d7-47d6-90f3-f055a39bdd87" path="/var/lib/kubelet/pods/03a86fa0-04d7-47d6-90f3-f055a39bdd87/volumes"
Jul 27 18:21:32.872597 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:32.872509    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:21:32.925073 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:32.925036    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6"]
Jul 27 18:21:32.925429 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:32.925414    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03a86fa0-04d7-47d6-90f3-f055a39bdd87" containerName="kube-rbac-proxy"
Jul 27 18:21:32.925488 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:32.925430    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="03a86fa0-04d7-47d6-90f3-f055a39bdd87" containerName="kube-rbac-proxy"
Jul 27 18:21:32.925488 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:32.925449    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03a86fa0-04d7-47d6-90f3-f055a39bdd87" containerName="kserve-container"
Jul 27 18:21:32.925488 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:32.925456    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="03a86fa0-04d7-47d6-90f3-f055a39bdd87" containerName="kserve-container"
Jul 27 18:21:32.925488 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:32.925465    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03a86fa0-04d7-47d6-90f3-f055a39bdd87" containerName="storage-initializer"
Jul 27 18:21:32.925488 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:32.925472    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="03a86fa0-04d7-47d6-90f3-f055a39bdd87" containerName="storage-initializer"
Jul 27 18:21:32.925641 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:32.925530    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="03a86fa0-04d7-47d6-90f3-f055a39bdd87" containerName="kserve-container"
Jul 27 18:21:32.925641 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:32.925539    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="03a86fa0-04d7-47d6-90f3-f055a39bdd87" containerName="kube-rbac-proxy"
Jul 27 18:21:32.930243 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:32.930222    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6"
Jul 27 18:21:32.932984 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:32.932951    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\""
Jul 27 18:21:32.933098 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:32.933037    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 18:21:32.933098 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:32.933058    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z6ld7\""
Jul 27 18:21:32.933215 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:32.933193    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 18:21:32.933215 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:32.933205    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 18:21:32.933326 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:32.933294    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\""
Jul 27 18:21:32.934529 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:32.934475    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 18:21:32.937120 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:32.937097    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6"]
Jul 27 18:21:33.070597 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:33.070564    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3e2bbdf-3803-4c9c-8281-9b93ead601fb-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6\" (UID: \"c3e2bbdf-3803-4c9c-8281-9b93ead601fb\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6"
Jul 27 18:21:33.070825 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:33.070604    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3e2bbdf-3803-4c9c-8281-9b93ead601fb-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6\" (UID: \"c3e2bbdf-3803-4c9c-8281-9b93ead601fb\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6"
Jul 27 18:21:33.070825 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:33.070631    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3e2bbdf-3803-4c9c-8281-9b93ead601fb-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6\" (UID: \"c3e2bbdf-3803-4c9c-8281-9b93ead601fb\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6"
Jul 27 18:21:33.070825 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:33.070698    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c52jt\" (UniqueName: \"kubernetes.io/projected/c3e2bbdf-3803-4c9c-8281-9b93ead601fb-kube-api-access-c52jt\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6\" (UID: \"c3e2bbdf-3803-4c9c-8281-9b93ead601fb\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6"
Jul 27 18:21:33.070825 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:33.070800    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3e2bbdf-3803-4c9c-8281-9b93ead601fb-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6\" (UID: \"c3e2bbdf-3803-4c9c-8281-9b93ead601fb\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6"
Jul 27 18:21:33.110490 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:33.110461    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:21:33.171867 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:33.171819    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3e2bbdf-3803-4c9c-8281-9b93ead601fb-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6\" (UID: \"c3e2bbdf-3803-4c9c-8281-9b93ead601fb\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6"
Jul 27 18:21:33.172072 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:33.171883    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3e2bbdf-3803-4c9c-8281-9b93ead601fb-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6\" (UID: \"c3e2bbdf-3803-4c9c-8281-9b93ead601fb\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6"
Jul 27 18:21:33.172072 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:33.171915    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3e2bbdf-3803-4c9c-8281-9b93ead601fb-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6\" (UID: \"c3e2bbdf-3803-4c9c-8281-9b93ead601fb\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6"
Jul 27 18:21:33.172072 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:33.171940    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c52jt\" (UniqueName: \"kubernetes.io/projected/c3e2bbdf-3803-4c9c-8281-9b93ead601fb-kube-api-access-c52jt\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6\" (UID: \"c3e2bbdf-3803-4c9c-8281-9b93ead601fb\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6"
Jul 27 18:21:33.172072 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:33.172002    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3e2bbdf-3803-4c9c-8281-9b93ead601fb-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6\" (UID: \"c3e2bbdf-3803-4c9c-8281-9b93ead601fb\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6"
Jul 27 18:21:33.172427 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:33.172402    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3e2bbdf-3803-4c9c-8281-9b93ead601fb-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6\" (UID: \"c3e2bbdf-3803-4c9c-8281-9b93ead601fb\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6"
Jul 27 18:21:33.172703 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:33.172684    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3e2bbdf-3803-4c9c-8281-9b93ead601fb-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6\" (UID: \"c3e2bbdf-3803-4c9c-8281-9b93ead601fb\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6"
Jul 27 18:21:33.172780 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:33.172728    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3e2bbdf-3803-4c9c-8281-9b93ead601fb-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6\" (UID: \"c3e2bbdf-3803-4c9c-8281-9b93ead601fb\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6"
Jul 27 18:21:33.174450 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:33.174428    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3e2bbdf-3803-4c9c-8281-9b93ead601fb-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6\" (UID: \"c3e2bbdf-3803-4c9c-8281-9b93ead601fb\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6"
Jul 27 18:21:33.180792 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:33.180769    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c52jt\" (UniqueName: \"kubernetes.io/projected/c3e2bbdf-3803-4c9c-8281-9b93ead601fb-kube-api-access-c52jt\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6\" (UID: \"c3e2bbdf-3803-4c9c-8281-9b93ead601fb\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6"
Jul 27 18:21:33.241635 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:33.241602    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6"
Jul 27 18:21:33.376331 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:33.376307    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6"]
Jul 27 18:21:33.379000 ip-10-0-140-0 kubenswrapper[2573]: W0727 18:21:33.378957    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3e2bbdf_3803_4c9c_8281_9b93ead601fb.slice/crio-ddd5367c486e516648f6c82cef10ba235c46ca12a7b1a1258136f10c0cb63676 WatchSource:0}: Error finding container ddd5367c486e516648f6c82cef10ba235c46ca12a7b1a1258136f10c0cb63676: Status 404 returned error can't find the container with id ddd5367c486e516648f6c82cef10ba235c46ca12a7b1a1258136f10c0cb63676
Jul 27 18:21:34.210454 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:34.210421    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6" event={"ID":"c3e2bbdf-3803-4c9c-8281-9b93ead601fb","Type":"ContainerStarted","Data":"00ce20a31a3eacd8166bd99aea01b56a28d0908bc5706eb652fc64b9bedf1a48"}
Jul 27 18:21:34.210851 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:34.210477    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6" event={"ID":"c3e2bbdf-3803-4c9c-8281-9b93ead601fb","Type":"ContainerStarted","Data":"ddd5367c486e516648f6c82cef10ba235c46ca12a7b1a1258136f10c0cb63676"}
Jul 27 18:21:38.228915 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:38.228879    2573 generic.go:356] "Generic (PLEG): container finished" podID="c3e2bbdf-3803-4c9c-8281-9b93ead601fb" containerID="00ce20a31a3eacd8166bd99aea01b56a28d0908bc5706eb652fc64b9bedf1a48" exitCode=0
Jul 27 18:21:38.229335 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:38.228955    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6" event={"ID":"c3e2bbdf-3803-4c9c-8281-9b93ead601fb","Type":"ContainerDied","Data":"00ce20a31a3eacd8166bd99aea01b56a28d0908bc5706eb652fc64b9bedf1a48"}
Jul 27 18:21:39.235187 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:39.235150    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6" event={"ID":"c3e2bbdf-3803-4c9c-8281-9b93ead601fb","Type":"ContainerStarted","Data":"07bfc1379e1d7fc11eade1ff2f3964ac80778961ec5186aac778a38688e6a50d"}
Jul 27 18:21:39.235658 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:39.235213    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6" event={"ID":"c3e2bbdf-3803-4c9c-8281-9b93ead601fb","Type":"ContainerStarted","Data":"ad2174a71a3c001cbeae7738e086706ea7d19e2546fd2dbbb749f36258f0fae5"}
Jul 27 18:21:39.235658 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:39.235424    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6"
Jul 27 18:21:39.256651 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:39.256599    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6" podStartSLOduration=3.47347398 podStartE2EDuration="7.256583175s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.783109195s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:21:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:21:39.254548646 +0000 UTC m=+3241.397950697" watchObservedRunningTime="2026-07-27 18:21:39.256583175 +0000 UTC m=+3241.399985226"
Jul 27 18:21:40.239610 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:40.239577    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6"
Jul 27 18:21:40.241020 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:40.240988    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6" podUID="c3e2bbdf-3803-4c9c-8281-9b93ead601fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Jul 27 18:21:41.249089 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:41.249050    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6" podUID="c3e2bbdf-3803-4c9c-8281-9b93ead601fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Jul 27 18:21:46.253723 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:46.253696    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6"
Jul 27 18:21:46.254220 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:46.254189    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6" podUID="c3e2bbdf-3803-4c9c-8281-9b93ead601fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Jul 27 18:21:56.255102 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:21:56.255061    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6" podUID="c3e2bbdf-3803-4c9c-8281-9b93ead601fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Jul 27 18:22:06.254799 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:22:06.254757    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6" podUID="c3e2bbdf-3803-4c9c-8281-9b93ead601fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Jul 27 18:22:16.254602 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:22:16.254562    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6" podUID="c3e2bbdf-3803-4c9c-8281-9b93ead601fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Jul 27 18:22:26.254223 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:22:26.254182    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6" podUID="c3e2bbdf-3803-4c9c-8281-9b93ead601fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Jul 27 18:22:36.254893 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:22:36.254850    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6" podUID="c3e2bbdf-3803-4c9c-8281-9b93ead601fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Jul 27 18:22:38.518095 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:22:38.518066    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vc2tz_87dac797-f8d2-4d36-999f-f8ce855dbeff/ovn-acl-logging/0.log"
Jul 27 18:22:38.525736 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:22:38.525717    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vc2tz_87dac797-f8d2-4d36-999f-f8ce855dbeff/ovn-acl-logging/0.log"
Jul 27 18:22:46.254463 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:22:46.254424    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6" podUID="c3e2bbdf-3803-4c9c-8281-9b93ead601fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Jul 27 18:22:56.254816 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:22:56.254775    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6" podUID="c3e2bbdf-3803-4c9c-8281-9b93ead601fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.56:8080: connect: connection refused"
Jul 27 18:22:59.465360 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:22:59.465332    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6"
Jul 27 18:23:04.073880 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:04.073844    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6"]
Jul 27 18:23:04.074394 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:04.074075    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6" podUID="c3e2bbdf-3803-4c9c-8281-9b93ead601fb" containerName="kserve-container" containerID="cri-o://ad2174a71a3c001cbeae7738e086706ea7d19e2546fd2dbbb749f36258f0fae5" gracePeriod=30
Jul 27 18:23:04.074394 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:04.074102    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6" podUID="c3e2bbdf-3803-4c9c-8281-9b93ead601fb" containerName="kube-rbac-proxy" containerID="cri-o://07bfc1379e1d7fc11eade1ff2f3964ac80778961ec5186aac778a38688e6a50d" gracePeriod=30
Jul 27 18:23:04.131865 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:04.131824    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:23:04.604126 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:04.604088    2573 generic.go:356] "Generic (PLEG): container finished" podID="c3e2bbdf-3803-4c9c-8281-9b93ead601fb" containerID="07bfc1379e1d7fc11eade1ff2f3964ac80778961ec5186aac778a38688e6a50d" exitCode=2
Jul 27 18:23:04.604347 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:04.604144    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6" event={"ID":"c3e2bbdf-3803-4c9c-8281-9b93ead601fb","Type":"ContainerDied","Data":"07bfc1379e1d7fc11eade1ff2f3964ac80778961ec5186aac778a38688e6a50d"}
Jul 27 18:23:06.249910 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:06.249867    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6" podUID="c3e2bbdf-3803-4c9c-8281-9b93ead601fb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.56:8643/healthz\": dial tcp 10.133.0.56:8643: connect: connection refused"
Jul 27 18:23:07.928680 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:07.928650    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6"
Jul 27 18:23:08.045218 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:08.045125    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3e2bbdf-3803-4c9c-8281-9b93ead601fb-kserve-provision-location\") pod \"c3e2bbdf-3803-4c9c-8281-9b93ead601fb\" (UID: \"c3e2bbdf-3803-4c9c-8281-9b93ead601fb\") "
Jul 27 18:23:08.045218 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:08.045189    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3e2bbdf-3803-4c9c-8281-9b93ead601fb-kube-rbac-proxy-sar-config\") pod \"c3e2bbdf-3803-4c9c-8281-9b93ead601fb\" (UID: \"c3e2bbdf-3803-4c9c-8281-9b93ead601fb\") "
Jul 27 18:23:08.045466 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:08.045239    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3e2bbdf-3803-4c9c-8281-9b93ead601fb-cabundle-cert\") pod \"c3e2bbdf-3803-4c9c-8281-9b93ead601fb\" (UID: \"c3e2bbdf-3803-4c9c-8281-9b93ead601fb\") "
Jul 27 18:23:08.045466 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:08.045301    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3e2bbdf-3803-4c9c-8281-9b93ead601fb-proxy-tls\") pod \"c3e2bbdf-3803-4c9c-8281-9b93ead601fb\" (UID: \"c3e2bbdf-3803-4c9c-8281-9b93ead601fb\") "
Jul 27 18:23:08.045466 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:08.045336    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c52jt\" (UniqueName: \"kubernetes.io/projected/c3e2bbdf-3803-4c9c-8281-9b93ead601fb-kube-api-access-c52jt\") pod \"c3e2bbdf-3803-4c9c-8281-9b93ead601fb\" (UID: \"c3e2bbdf-3803-4c9c-8281-9b93ead601fb\") "
Jul 27 18:23:08.045637 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:08.045535    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3e2bbdf-3803-4c9c-8281-9b93ead601fb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c3e2bbdf-3803-4c9c-8281-9b93ead601fb" (UID: "c3e2bbdf-3803-4c9c-8281-9b93ead601fb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:23:08.045702 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:08.045627    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3e2bbdf-3803-4c9c-8281-9b93ead601fb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c3e2bbdf-3803-4c9c-8281-9b93ead601fb" (UID: "c3e2bbdf-3803-4c9c-8281-9b93ead601fb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:23:08.045702 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:08.045643    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3e2bbdf-3803-4c9c-8281-9b93ead601fb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c3e2bbdf-3803-4c9c-8281-9b93ead601fb" (UID: "c3e2bbdf-3803-4c9c-8281-9b93ead601fb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:23:08.047456 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:08.047428    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3e2bbdf-3803-4c9c-8281-9b93ead601fb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c3e2bbdf-3803-4c9c-8281-9b93ead601fb" (UID: "c3e2bbdf-3803-4c9c-8281-9b93ead601fb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:23:08.047537 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:08.047478    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3e2bbdf-3803-4c9c-8281-9b93ead601fb-kube-api-access-c52jt" (OuterVolumeSpecName: "kube-api-access-c52jt") pod "c3e2bbdf-3803-4c9c-8281-9b93ead601fb" (UID: "c3e2bbdf-3803-4c9c-8281-9b93ead601fb"). InnerVolumeSpecName "kube-api-access-c52jt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:23:08.146081 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:08.146027    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3e2bbdf-3803-4c9c-8281-9b93ead601fb-kserve-provision-location\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:23:08.146081 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:08.146073    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c3e2bbdf-3803-4c9c-8281-9b93ead601fb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:23:08.146081 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:08.146084    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3e2bbdf-3803-4c9c-8281-9b93ead601fb-cabundle-cert\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:23:08.146081 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:08.146093    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c3e2bbdf-3803-4c9c-8281-9b93ead601fb-proxy-tls\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:23:08.146081 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:08.146101    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c52jt\" (UniqueName: \"kubernetes.io/projected/c3e2bbdf-3803-4c9c-8281-9b93ead601fb-kube-api-access-c52jt\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:23:08.624587 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:08.624554    2573 generic.go:356] "Generic (PLEG): container finished" podID="c3e2bbdf-3803-4c9c-8281-9b93ead601fb" containerID="ad2174a71a3c001cbeae7738e086706ea7d19e2546fd2dbbb749f36258f0fae5" exitCode=0
Jul 27 18:23:08.624587 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:08.624597    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6" event={"ID":"c3e2bbdf-3803-4c9c-8281-9b93ead601fb","Type":"ContainerDied","Data":"ad2174a71a3c001cbeae7738e086706ea7d19e2546fd2dbbb749f36258f0fae5"}
Jul 27 18:23:08.624846 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:08.624619    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6"
Jul 27 18:23:08.624846 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:08.624661    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6" event={"ID":"c3e2bbdf-3803-4c9c-8281-9b93ead601fb","Type":"ContainerDied","Data":"ddd5367c486e516648f6c82cef10ba235c46ca12a7b1a1258136f10c0cb63676"}
Jul 27 18:23:08.624846 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:08.624699    2573 scope.go:117] "RemoveContainer" containerID="07bfc1379e1d7fc11eade1ff2f3964ac80778961ec5186aac778a38688e6a50d"
Jul 27 18:23:08.632770 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:08.632752    2573 scope.go:117] "RemoveContainer" containerID="ad2174a71a3c001cbeae7738e086706ea7d19e2546fd2dbbb749f36258f0fae5"
Jul 27 18:23:08.640345 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:08.640321    2573 scope.go:117] "RemoveContainer" containerID="00ce20a31a3eacd8166bd99aea01b56a28d0908bc5706eb652fc64b9bedf1a48"
Jul 27 18:23:08.646901 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:08.646876    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6"]
Jul 27 18:23:08.648143 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:08.648125    2573 scope.go:117] "RemoveContainer" containerID="07bfc1379e1d7fc11eade1ff2f3964ac80778961ec5186aac778a38688e6a50d"
Jul 27 18:23:08.648490 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:23:08.648464    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07bfc1379e1d7fc11eade1ff2f3964ac80778961ec5186aac778a38688e6a50d\": container with ID starting with 07bfc1379e1d7fc11eade1ff2f3964ac80778961ec5186aac778a38688e6a50d not found: ID does not exist" containerID="07bfc1379e1d7fc11eade1ff2f3964ac80778961ec5186aac778a38688e6a50d"
Jul 27 18:23:08.648632 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:08.648495    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07bfc1379e1d7fc11eade1ff2f3964ac80778961ec5186aac778a38688e6a50d"} err="failed to get container status \"07bfc1379e1d7fc11eade1ff2f3964ac80778961ec5186aac778a38688e6a50d\": rpc error: code = NotFound desc = could not find container \"07bfc1379e1d7fc11eade1ff2f3964ac80778961ec5186aac778a38688e6a50d\": container with ID starting with 07bfc1379e1d7fc11eade1ff2f3964ac80778961ec5186aac778a38688e6a50d not found: ID does not exist"
Jul 27 18:23:08.648632 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:08.648514    2573 scope.go:117] "RemoveContainer" containerID="ad2174a71a3c001cbeae7738e086706ea7d19e2546fd2dbbb749f36258f0fae5"
Jul 27 18:23:08.648798 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:23:08.648748    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad2174a71a3c001cbeae7738e086706ea7d19e2546fd2dbbb749f36258f0fae5\": container with ID starting with ad2174a71a3c001cbeae7738e086706ea7d19e2546fd2dbbb749f36258f0fae5 not found: ID does not exist" containerID="ad2174a71a3c001cbeae7738e086706ea7d19e2546fd2dbbb749f36258f0fae5"
Jul 27 18:23:08.648798 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:08.648767    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad2174a71a3c001cbeae7738e086706ea7d19e2546fd2dbbb749f36258f0fae5"} err="failed to get container status \"ad2174a71a3c001cbeae7738e086706ea7d19e2546fd2dbbb749f36258f0fae5\": rpc error: code = NotFound desc = could not find container \"ad2174a71a3c001cbeae7738e086706ea7d19e2546fd2dbbb749f36258f0fae5\": container with ID starting with ad2174a71a3c001cbeae7738e086706ea7d19e2546fd2dbbb749f36258f0fae5 not found: ID does not exist"
Jul 27 18:23:08.648798 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:08.648786    2573 scope.go:117] "RemoveContainer" containerID="00ce20a31a3eacd8166bd99aea01b56a28d0908bc5706eb652fc64b9bedf1a48"
Jul 27 18:23:08.649110 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:23:08.649045    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00ce20a31a3eacd8166bd99aea01b56a28d0908bc5706eb652fc64b9bedf1a48\": container with ID starting with 00ce20a31a3eacd8166bd99aea01b56a28d0908bc5706eb652fc64b9bedf1a48 not found: ID does not exist" containerID="00ce20a31a3eacd8166bd99aea01b56a28d0908bc5706eb652fc64b9bedf1a48"
Jul 27 18:23:08.649110 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:08.649075    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00ce20a31a3eacd8166bd99aea01b56a28d0908bc5706eb652fc64b9bedf1a48"} err="failed to get container status \"00ce20a31a3eacd8166bd99aea01b56a28d0908bc5706eb652fc64b9bedf1a48\": rpc error: code = NotFound desc = could not find container \"00ce20a31a3eacd8166bd99aea01b56a28d0908bc5706eb652fc64b9bedf1a48\": container with ID starting with 00ce20a31a3eacd8166bd99aea01b56a28d0908bc5706eb652fc64b9bedf1a48 not found: ID does not exist"
Jul 27 18:23:08.650903 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:08.650879    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-9rxq6"]
Jul 27 18:23:09.182632 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:09.182596    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:23:09.240518 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:09.240482    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-c58edb-predictor-6844999f8f-g9f9h"]
Jul 27 18:23:09.240844 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:09.240832    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3e2bbdf-3803-4c9c-8281-9b93ead601fb" containerName="kube-rbac-proxy"
Jul 27 18:23:09.240893 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:09.240846    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3e2bbdf-3803-4c9c-8281-9b93ead601fb" containerName="kube-rbac-proxy"
Jul 27 18:23:09.240893 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:09.240859    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3e2bbdf-3803-4c9c-8281-9b93ead601fb" containerName="storage-initializer"
Jul 27 18:23:09.240893 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:09.240865    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3e2bbdf-3803-4c9c-8281-9b93ead601fb" containerName="storage-initializer"
Jul 27 18:23:09.240893 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:09.240872    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3e2bbdf-3803-4c9c-8281-9b93ead601fb" containerName="kserve-container"
Jul 27 18:23:09.240893 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:09.240878    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3e2bbdf-3803-4c9c-8281-9b93ead601fb" containerName="kserve-container"
Jul 27 18:23:09.241045 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:09.240927    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3e2bbdf-3803-4c9c-8281-9b93ead601fb" containerName="kserve-container"
Jul 27 18:23:09.241045 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:09.240934    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3e2bbdf-3803-4c9c-8281-9b93ead601fb" containerName="kube-rbac-proxy"
Jul 27 18:23:09.245728 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:09.245708    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-c58edb-predictor-6844999f8f-g9f9h"
Jul 27 18:23:09.249078 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:09.249052    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 18:23:09.249217 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:09.249064    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 18:23:09.249359 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:09.249335    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 18:23:09.249432 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:09.249418    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z6ld7\""
Jul 27 18:23:09.249493 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:09.249454    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-c58edb-predictor-serving-cert\""
Jul 27 18:23:09.249611 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:09.249586    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 18:23:09.249718 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:09.249622    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-c58edb-kube-rbac-proxy-sar-config\""
Jul 27 18:23:09.255922 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:09.255899    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-c58edb-predictor-6844999f8f-g9f9h"]
Jul 27 18:23:09.356977 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:09.356937    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83658999-b2b6-4ac2-883b-8d8bae85d445-cabundle-cert\") pod \"isvc-primary-c58edb-predictor-6844999f8f-g9f9h\" (UID: \"83658999-b2b6-4ac2-883b-8d8bae85d445\") " pod="kserve-ci-e2e-test/isvc-primary-c58edb-predictor-6844999f8f-g9f9h"
Jul 27 18:23:09.357180 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:09.356985    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83658999-b2b6-4ac2-883b-8d8bae85d445-kserve-provision-location\") pod \"isvc-primary-c58edb-predictor-6844999f8f-g9f9h\" (UID: \"83658999-b2b6-4ac2-883b-8d8bae85d445\") " pod="kserve-ci-e2e-test/isvc-primary-c58edb-predictor-6844999f8f-g9f9h"
Jul 27 18:23:09.357180 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:09.357057    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83658999-b2b6-4ac2-883b-8d8bae85d445-proxy-tls\") pod \"isvc-primary-c58edb-predictor-6844999f8f-g9f9h\" (UID: \"83658999-b2b6-4ac2-883b-8d8bae85d445\") " pod="kserve-ci-e2e-test/isvc-primary-c58edb-predictor-6844999f8f-g9f9h"
Jul 27 18:23:09.357180 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:09.357096    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p5t5\" (UniqueName: \"kubernetes.io/projected/83658999-b2b6-4ac2-883b-8d8bae85d445-kube-api-access-2p5t5\") pod \"isvc-primary-c58edb-predictor-6844999f8f-g9f9h\" (UID: \"83658999-b2b6-4ac2-883b-8d8bae85d445\") " pod="kserve-ci-e2e-test/isvc-primary-c58edb-predictor-6844999f8f-g9f9h"
Jul 27 18:23:09.357180 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:09.357140    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83658999-b2b6-4ac2-883b-8d8bae85d445-kube-rbac-proxy-sar-config\") pod \"isvc-primary-c58edb-predictor-6844999f8f-g9f9h\" (UID: \"83658999-b2b6-4ac2-883b-8d8bae85d445\") " pod="kserve-ci-e2e-test/isvc-primary-c58edb-predictor-6844999f8f-g9f9h"
Jul 27 18:23:09.361751 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:09.361727    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:23:09.458370 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:09.458276    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83658999-b2b6-4ac2-883b-8d8bae85d445-cabundle-cert\") pod \"isvc-primary-c58edb-predictor-6844999f8f-g9f9h\" (UID: \"83658999-b2b6-4ac2-883b-8d8bae85d445\") " pod="kserve-ci-e2e-test/isvc-primary-c58edb-predictor-6844999f8f-g9f9h"
Jul 27 18:23:09.458370 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:09.458333    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83658999-b2b6-4ac2-883b-8d8bae85d445-kserve-provision-location\") pod \"isvc-primary-c58edb-predictor-6844999f8f-g9f9h\" (UID: \"83658999-b2b6-4ac2-883b-8d8bae85d445\") " pod="kserve-ci-e2e-test/isvc-primary-c58edb-predictor-6844999f8f-g9f9h"
Jul 27 18:23:09.458581 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:09.458374    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83658999-b2b6-4ac2-883b-8d8bae85d445-proxy-tls\") pod \"isvc-primary-c58edb-predictor-6844999f8f-g9f9h\" (UID: \"83658999-b2b6-4ac2-883b-8d8bae85d445\") " pod="kserve-ci-e2e-test/isvc-primary-c58edb-predictor-6844999f8f-g9f9h"
Jul 27 18:23:09.458581 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:09.458403    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p5t5\" (UniqueName: \"kubernetes.io/projected/83658999-b2b6-4ac2-883b-8d8bae85d445-kube-api-access-2p5t5\") pod \"isvc-primary-c58edb-predictor-6844999f8f-g9f9h\" (UID: \"83658999-b2b6-4ac2-883b-8d8bae85d445\") " pod="kserve-ci-e2e-test/isvc-primary-c58edb-predictor-6844999f8f-g9f9h"
Jul 27 18:23:09.458581 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:09.458443    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83658999-b2b6-4ac2-883b-8d8bae85d445-kube-rbac-proxy-sar-config\") pod \"isvc-primary-c58edb-predictor-6844999f8f-g9f9h\" (UID: \"83658999-b2b6-4ac2-883b-8d8bae85d445\") " pod="kserve-ci-e2e-test/isvc-primary-c58edb-predictor-6844999f8f-g9f9h"
Jul 27 18:23:09.458806 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:09.458773    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83658999-b2b6-4ac2-883b-8d8bae85d445-kserve-provision-location\") pod \"isvc-primary-c58edb-predictor-6844999f8f-g9f9h\" (UID: \"83658999-b2b6-4ac2-883b-8d8bae85d445\") " pod="kserve-ci-e2e-test/isvc-primary-c58edb-predictor-6844999f8f-g9f9h"
Jul 27 18:23:09.459049 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:09.459021    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83658999-b2b6-4ac2-883b-8d8bae85d445-cabundle-cert\") pod \"isvc-primary-c58edb-predictor-6844999f8f-g9f9h\" (UID: \"83658999-b2b6-4ac2-883b-8d8bae85d445\") " pod="kserve-ci-e2e-test/isvc-primary-c58edb-predictor-6844999f8f-g9f9h"
Jul 27 18:23:09.459125 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:09.459047    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83658999-b2b6-4ac2-883b-8d8bae85d445-kube-rbac-proxy-sar-config\") pod \"isvc-primary-c58edb-predictor-6844999f8f-g9f9h\" (UID: \"83658999-b2b6-4ac2-883b-8d8bae85d445\") " pod="kserve-ci-e2e-test/isvc-primary-c58edb-predictor-6844999f8f-g9f9h"
Jul 27 18:23:09.460787 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:09.460768    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83658999-b2b6-4ac2-883b-8d8bae85d445-proxy-tls\") pod \"isvc-primary-c58edb-predictor-6844999f8f-g9f9h\" (UID: \"83658999-b2b6-4ac2-883b-8d8bae85d445\") " pod="kserve-ci-e2e-test/isvc-primary-c58edb-predictor-6844999f8f-g9f9h"
Jul 27 18:23:09.468175 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:09.468153    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p5t5\" (UniqueName: \"kubernetes.io/projected/83658999-b2b6-4ac2-883b-8d8bae85d445-kube-api-access-2p5t5\") pod \"isvc-primary-c58edb-predictor-6844999f8f-g9f9h\" (UID: \"83658999-b2b6-4ac2-883b-8d8bae85d445\") " pod="kserve-ci-e2e-test/isvc-primary-c58edb-predictor-6844999f8f-g9f9h"
Jul 27 18:23:09.556913 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:09.556869    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-c58edb-predictor-6844999f8f-g9f9h"
Jul 27 18:23:09.696986 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:09.696962    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-c58edb-predictor-6844999f8f-g9f9h"]
Jul 27 18:23:09.699444 ip-10-0-140-0 kubenswrapper[2573]: W0727 18:23:09.699411    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83658999_b2b6_4ac2_883b_8d8bae85d445.slice/crio-589c084b05f4f3e6a4f29b8a95bf020ba28c1a4ab45ce9ca4b034c6267e2359c WatchSource:0}: Error finding container 589c084b05f4f3e6a4f29b8a95bf020ba28c1a4ab45ce9ca4b034c6267e2359c: Status 404 returned error can't find the container with id 589c084b05f4f3e6a4f29b8a95bf020ba28c1a4ab45ce9ca4b034c6267e2359c
Jul 27 18:23:10.470078 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:10.470040    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3e2bbdf-3803-4c9c-8281-9b93ead601fb" path="/var/lib/kubelet/pods/c3e2bbdf-3803-4c9c-8281-9b93ead601fb/volumes"
Jul 27 18:23:10.636903 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:10.636862    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-c58edb-predictor-6844999f8f-g9f9h" event={"ID":"83658999-b2b6-4ac2-883b-8d8bae85d445","Type":"ContainerStarted","Data":"122dc99245374497556267238b683c29f02cd4c937679d7b8fb2c3a3eb655596"}
Jul 27 18:23:10.637088 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:10.636938    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-c58edb-predictor-6844999f8f-g9f9h" event={"ID":"83658999-b2b6-4ac2-883b-8d8bae85d445","Type":"ContainerStarted","Data":"589c084b05f4f3e6a4f29b8a95bf020ba28c1a4ab45ce9ca4b034c6267e2359c"}
Jul 27 18:23:14.655714 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:14.655679    2573 generic.go:356] "Generic (PLEG): container finished" podID="83658999-b2b6-4ac2-883b-8d8bae85d445" containerID="122dc99245374497556267238b683c29f02cd4c937679d7b8fb2c3a3eb655596" exitCode=0
Jul 27 18:23:14.656148 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:14.655740    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-c58edb-predictor-6844999f8f-g9f9h" event={"ID":"83658999-b2b6-4ac2-883b-8d8bae85d445","Type":"ContainerDied","Data":"122dc99245374497556267238b683c29f02cd4c937679d7b8fb2c3a3eb655596"}
Jul 27 18:23:15.662048 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:15.662013    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-c58edb-predictor-6844999f8f-g9f9h" event={"ID":"83658999-b2b6-4ac2-883b-8d8bae85d445","Type":"ContainerStarted","Data":"b50347c3130689498d238ebefde644c7a9b5791ef4d981571e24b736732bd0d9"}
Jul 27 18:23:15.662462 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:15.662075    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-c58edb-predictor-6844999f8f-g9f9h" event={"ID":"83658999-b2b6-4ac2-883b-8d8bae85d445","Type":"ContainerStarted","Data":"0e621d421c426591cec041e972a17050fcfa105b212ce551679ac9ac0270ad31"}
Jul 27 18:23:15.662462 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:15.662278    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-c58edb-predictor-6844999f8f-g9f9h"
Jul 27 18:23:15.689918 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:15.686939    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-c58edb-predictor-6844999f8f-g9f9h" podStartSLOduration=2.711628635 podStartE2EDuration="6.686916467s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.975287832s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:23:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:23:15.684153937 +0000 UTC m=+3337.827555989" watchObservedRunningTime="2026-07-27 18:23:15.686916467 +0000 UTC m=+3337.830318519"
Jul 27 18:23:16.666002 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:16.665960    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-c58edb-predictor-6844999f8f-g9f9h"
Jul 27 18:23:16.667270 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:16.667223    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-c58edb-predictor-6844999f8f-g9f9h" podUID="83658999-b2b6-4ac2-883b-8d8bae85d445" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Jul 27 18:23:17.672465 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:17.672421    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-c58edb-predictor-6844999f8f-g9f9h" podUID="83658999-b2b6-4ac2-883b-8d8bae85d445" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Jul 27 18:23:22.677336 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:22.677302    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-c58edb-predictor-6844999f8f-g9f9h"
Jul 27 18:23:22.677977 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:22.677946    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-c58edb-predictor-6844999f8f-g9f9h" podUID="83658999-b2b6-4ac2-883b-8d8bae85d445" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Jul 27 18:23:32.678413 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:32.678365    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-c58edb-predictor-6844999f8f-g9f9h" podUID="83658999-b2b6-4ac2-883b-8d8bae85d445" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Jul 27 18:23:42.678914 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:42.678877    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-c58edb-predictor-6844999f8f-g9f9h" podUID="83658999-b2b6-4ac2-883b-8d8bae85d445" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Jul 27 18:23:52.678099 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:23:52.678058    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-c58edb-predictor-6844999f8f-g9f9h" podUID="83658999-b2b6-4ac2-883b-8d8bae85d445" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Jul 27 18:24:02.678313 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:24:02.678271    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-c58edb-predictor-6844999f8f-g9f9h" podUID="83658999-b2b6-4ac2-883b-8d8bae85d445" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Jul 27 18:24:12.678159 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:24:12.678110    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-c58edb-predictor-6844999f8f-g9f9h" podUID="83658999-b2b6-4ac2-883b-8d8bae85d445" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Jul 27 18:24:22.678553 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:24:22.678512    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-c58edb-predictor-6844999f8f-g9f9h" podUID="83658999-b2b6-4ac2-883b-8d8bae85d445" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Jul 27 18:24:32.678479 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:24:32.678400    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-c58edb-predictor-6844999f8f-g9f9h"
Jul 27 18:24:39.301670 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:24:39.301629    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:24:47.993211 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:24:47.993149    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:24:54.439358 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:24:54.439320    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:24:54.515671 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:24:54.515640    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-c58edb-predictor-7bf565bf7d-n8wcw"]
Jul 27 18:24:54.519055 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:24:54.519036    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-c58edb-predictor-7bf565bf7d-n8wcw"
Jul 27 18:24:54.523912 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:24:54.523885    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-c58edb-kube-rbac-proxy-sar-config\""
Jul 27 18:24:54.524071 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:24:54.523985    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-c58edb-predictor-serving-cert\""
Jul 27 18:24:54.524071 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:24:54.524043    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-c58edb\""
Jul 27 18:24:54.524755 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:24:54.524734    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-c58edb-dockercfg-r8ll8\""
Jul 27 18:24:54.540409 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:24:54.540376    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-c58edb-predictor-7bf565bf7d-n8wcw"]
Jul 27 18:24:54.639171 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:24:54.639133    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b1103da-c8f0-4c54-bd4b-0a8488b37171-proxy-tls\") pod \"isvc-secondary-c58edb-predictor-7bf565bf7d-n8wcw\" (UID: \"1b1103da-c8f0-4c54-bd4b-0a8488b37171\") " pod="kserve-ci-e2e-test/isvc-secondary-c58edb-predictor-7bf565bf7d-n8wcw"
Jul 27 18:24:54.639412 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:24:54.639210    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b1103da-c8f0-4c54-bd4b-0a8488b37171-cabundle-cert\") pod \"isvc-secondary-c58edb-predictor-7bf565bf7d-n8wcw\" (UID: \"1b1103da-c8f0-4c54-bd4b-0a8488b37171\") " pod="kserve-ci-e2e-test/isvc-secondary-c58edb-predictor-7bf565bf7d-n8wcw"
Jul 27 18:24:54.639412 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:24:54.639278    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg9j2\" (UniqueName: \"kubernetes.io/projected/1b1103da-c8f0-4c54-bd4b-0a8488b37171-kube-api-access-lg9j2\") pod \"isvc-secondary-c58edb-predictor-7bf565bf7d-n8wcw\" (UID: \"1b1103da-c8f0-4c54-bd4b-0a8488b37171\") " pod="kserve-ci-e2e-test/isvc-secondary-c58edb-predictor-7bf565bf7d-n8wcw"
Jul 27 18:24:54.639412 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:24:54.639299    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b1103da-c8f0-4c54-bd4b-0a8488b37171-kserve-provision-location\") pod \"isvc-secondary-c58edb-predictor-7bf565bf7d-n8wcw\" (UID: \"1b1103da-c8f0-4c54-bd4b-0a8488b37171\") " pod="kserve-ci-e2e-test/isvc-secondary-c58edb-predictor-7bf565bf7d-n8wcw"
Jul 27 18:24:54.639412 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:24:54.639342    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b1103da-c8f0-4c54-bd4b-0a8488b37171-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-c58edb-predictor-7bf565bf7d-n8wcw\" (UID: \"1b1103da-c8f0-4c54-bd4b-0a8488b37171\") " pod="kserve-ci-e2e-test/isvc-secondary-c58edb-predictor-7bf565bf7d-n8wcw"
Jul 27 18:24:54.739991 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:24:54.739888    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b1103da-c8f0-4c54-bd4b-0a8488b37171-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-c58edb-predictor-7bf565bf7d-n8wcw\" (UID: \"1b1103da-c8f0-4c54-bd4b-0a8488b37171\") " pod="kserve-ci-e2e-test/isvc-secondary-c58edb-predictor-7bf565bf7d-n8wcw"
Jul 27 18:24:54.739991 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:24:54.739931    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b1103da-c8f0-4c54-bd4b-0a8488b37171-proxy-tls\") pod \"isvc-secondary-c58edb-predictor-7bf565bf7d-n8wcw\" (UID: \"1b1103da-c8f0-4c54-bd4b-0a8488b37171\") " pod="kserve-ci-e2e-test/isvc-secondary-c58edb-predictor-7bf565bf7d-n8wcw"
Jul 27 18:24:54.739991 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:24:54.739982    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b1103da-c8f0-4c54-bd4b-0a8488b37171-cabundle-cert\") pod \"isvc-secondary-c58edb-predictor-7bf565bf7d-n8wcw\" (UID: \"1b1103da-c8f0-4c54-bd4b-0a8488b37171\") " pod="kserve-ci-e2e-test/isvc-secondary-c58edb-predictor-7bf565bf7d-n8wcw"
Jul 27 18:24:54.740335 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:24:54.740003    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg9j2\" (UniqueName: \"kubernetes.io/projected/1b1103da-c8f0-4c54-bd4b-0a8488b37171-kube-api-access-lg9j2\") pod \"isvc-secondary-c58edb-predictor-7bf565bf7d-n8wcw\" (UID: \"1b1103da-c8f0-4c54-bd4b-0a8488b37171\") " pod="kserve-ci-e2e-test/isvc-secondary-c58edb-predictor-7bf565bf7d-n8wcw"
Jul 27 18:24:54.740335 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:24:54.740020    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b1103da-c8f0-4c54-bd4b-0a8488b37171-kserve-provision-location\") pod \"isvc-secondary-c58edb-predictor-7bf565bf7d-n8wcw\" (UID: \"1b1103da-c8f0-4c54-bd4b-0a8488b37171\") " pod="kserve-ci-e2e-test/isvc-secondary-c58edb-predictor-7bf565bf7d-n8wcw"
Jul 27 18:24:54.740335 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:24:54.740154    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-c58edb-predictor-serving-cert: secret "isvc-secondary-c58edb-predictor-serving-cert" not found
Jul 27 18:24:54.740335 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:24:54.740240    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b1103da-c8f0-4c54-bd4b-0a8488b37171-proxy-tls podName:1b1103da-c8f0-4c54-bd4b-0a8488b37171 nodeName:}" failed. No retries permitted until 2026-07-27 18:24:55.240218077 +0000 UTC m=+3437.383620114 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1b1103da-c8f0-4c54-bd4b-0a8488b37171-proxy-tls") pod "isvc-secondary-c58edb-predictor-7bf565bf7d-n8wcw" (UID: "1b1103da-c8f0-4c54-bd4b-0a8488b37171") : secret "isvc-secondary-c58edb-predictor-serving-cert" not found
Jul 27 18:24:54.740596 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:24:54.740436    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b1103da-c8f0-4c54-bd4b-0a8488b37171-kserve-provision-location\") pod \"isvc-secondary-c58edb-predictor-7bf565bf7d-n8wcw\" (UID: \"1b1103da-c8f0-4c54-bd4b-0a8488b37171\") " pod="kserve-ci-e2e-test/isvc-secondary-c58edb-predictor-7bf565bf7d-n8wcw"
Jul 27 18:24:54.740719 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:24:54.740697    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b1103da-c8f0-4c54-bd4b-0a8488b37171-cabundle-cert\") pod \"isvc-secondary-c58edb-predictor-7bf565bf7d-n8wcw\" (UID: \"1b1103da-c8f0-4c54-bd4b-0a8488b37171\") " pod="kserve-ci-e2e-test/isvc-secondary-c58edb-predictor-7bf565bf7d-n8wcw"
Jul 27 18:24:54.740780 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:24:54.740707    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b1103da-c8f0-4c54-bd4b-0a8488b37171-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-c58edb-predictor-7bf565bf7d-n8wcw\" (UID: \"1b1103da-c8f0-4c54-bd4b-0a8488b37171\") " pod="kserve-ci-e2e-test/isvc-secondary-c58edb-predictor-7bf565bf7d-n8wcw"
Jul 27 18:24:54.749498 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:24:54.749476    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg9j2\" (UniqueName: \"kubernetes.io/projected/1b1103da-c8f0-4c54-bd4b-0a8488b37171-kube-api-access-lg9j2\") pod \"isvc-secondary-c58edb-predictor-7bf565bf7d-n8wcw\" (UID: \"1b1103da-c8f0-4c54-bd4b-0a8488b37171\") " pod="kserve-ci-e2e-test/isvc-secondary-c58edb-predictor-7bf565bf7d-n8wcw"
Jul 27 18:24:54.799927 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:24:54.799889    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:24:55.245349 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:24:55.245311    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b1103da-c8f0-4c54-bd4b-0a8488b37171-proxy-tls\") pod \"isvc-secondary-c58edb-predictor-7bf565bf7d-n8wcw\" (UID: \"1b1103da-c8f0-4c54-bd4b-0a8488b37171\") " pod="kserve-ci-e2e-test/isvc-secondary-c58edb-predictor-7bf565bf7d-n8wcw"
Jul 27 18:24:55.247763 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:24:55.247738    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b1103da-c8f0-4c54-bd4b-0a8488b37171-proxy-tls\") pod \"isvc-secondary-c58edb-predictor-7bf565bf7d-n8wcw\" (UID: \"1b1103da-c8f0-4c54-bd4b-0a8488b37171\") " pod="kserve-ci-e2e-test/isvc-secondary-c58edb-predictor-7bf565bf7d-n8wcw"
Jul 27 18:24:55.429482 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:24:55.429427    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-c58edb-predictor-7bf565bf7d-n8wcw"
Jul 27 18:24:55.566356 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:24:55.566329    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-c58edb-predictor-7bf565bf7d-n8wcw"]
Jul 27 18:24:55.568398 ip-10-0-140-0 kubenswrapper[2573]: W0727 18:24:55.568362    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b1103da_c8f0_4c54_bd4b_0a8488b37171.slice/crio-45d401fb06d7fbf21f29d481738ef518db6ab881a001e3846eb41e53e72c62e8 WatchSource:0}: Error finding container 45d401fb06d7fbf21f29d481738ef518db6ab881a001e3846eb41e53e72c62e8: Status 404 returned error can't find the container with id 45d401fb06d7fbf21f29d481738ef518db6ab881a001e3846eb41e53e72c62e8
Jul 27 18:24:56.092396 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:24:56.092357    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-c58edb-predictor-7bf565bf7d-n8wcw" event={"ID":"1b1103da-c8f0-4c54-bd4b-0a8488b37171","Type":"ContainerStarted","Data":"0ad8c333cf50b44c4da53aa41299728b0cf39a69e7840ba53396f1deb2532a49"}
Jul 27 18:24:56.092585 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:24:56.092417    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-c58edb-predictor-7bf565bf7d-n8wcw" event={"ID":"1b1103da-c8f0-4c54-bd4b-0a8488b37171","Type":"ContainerStarted","Data":"45d401fb06d7fbf21f29d481738ef518db6ab881a001e3846eb41e53e72c62e8"}
Jul 27 18:25:01.116300 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:01.116269    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-c58edb-predictor-7bf565bf7d-n8wcw_1b1103da-c8f0-4c54-bd4b-0a8488b37171/storage-initializer/0.log"
Jul 27 18:25:01.116757 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:01.116306    2573 generic.go:356] "Generic (PLEG): container finished" podID="1b1103da-c8f0-4c54-bd4b-0a8488b37171" containerID="0ad8c333cf50b44c4da53aa41299728b0cf39a69e7840ba53396f1deb2532a49" exitCode=1
Jul 27 18:25:01.116757 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:01.116360    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-c58edb-predictor-7bf565bf7d-n8wcw" event={"ID":"1b1103da-c8f0-4c54-bd4b-0a8488b37171","Type":"ContainerDied","Data":"0ad8c333cf50b44c4da53aa41299728b0cf39a69e7840ba53396f1deb2532a49"}
Jul 27 18:25:02.123116 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:02.123088    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-c58edb-predictor-7bf565bf7d-n8wcw_1b1103da-c8f0-4c54-bd4b-0a8488b37171/storage-initializer/0.log"
Jul 27 18:25:02.123523 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:02.123210    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-c58edb-predictor-7bf565bf7d-n8wcw" event={"ID":"1b1103da-c8f0-4c54-bd4b-0a8488b37171","Type":"ContainerStarted","Data":"a73bd1c6baf8c77d87f992eae80d0f9aafb207e01709833a57e3b79a1cfa39ce"}
Jul 27 18:25:07.147539 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:07.147511    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-c58edb-predictor-7bf565bf7d-n8wcw_1b1103da-c8f0-4c54-bd4b-0a8488b37171/storage-initializer/1.log"
Jul 27 18:25:07.147966 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:07.147915    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-c58edb-predictor-7bf565bf7d-n8wcw_1b1103da-c8f0-4c54-bd4b-0a8488b37171/storage-initializer/0.log"
Jul 27 18:25:07.147966 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:07.147949    2573 generic.go:356] "Generic (PLEG): container finished" podID="1b1103da-c8f0-4c54-bd4b-0a8488b37171" containerID="a73bd1c6baf8c77d87f992eae80d0f9aafb207e01709833a57e3b79a1cfa39ce" exitCode=1
Jul 27 18:25:07.148050 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:07.148019    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-c58edb-predictor-7bf565bf7d-n8wcw" event={"ID":"1b1103da-c8f0-4c54-bd4b-0a8488b37171","Type":"ContainerDied","Data":"a73bd1c6baf8c77d87f992eae80d0f9aafb207e01709833a57e3b79a1cfa39ce"}
Jul 27 18:25:07.148122 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:07.148110    2573 scope.go:117] "RemoveContainer" containerID="0ad8c333cf50b44c4da53aa41299728b0cf39a69e7840ba53396f1deb2532a49"
Jul 27 18:25:07.148528 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:07.148510    2573 scope.go:117] "RemoveContainer" containerID="0ad8c333cf50b44c4da53aa41299728b0cf39a69e7840ba53396f1deb2532a49"
Jul 27 18:25:07.159412 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:25:07.159389    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-c58edb-predictor-7bf565bf7d-n8wcw_kserve-ci-e2e-test(1b1103da-c8f0-4c54-bd4b-0a8488b37171)\"" pod="kserve-ci-e2e-test/isvc-secondary-c58edb-predictor-7bf565bf7d-n8wcw" podUID="1b1103da-c8f0-4c54-bd4b-0a8488b37171"
Jul 27 18:25:07.159688 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:25:07.159659    2573 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-c58edb-predictor-7bf565bf7d-n8wcw_kserve-ci-e2e-test_1b1103da-c8f0-4c54-bd4b-0a8488b37171_0 in pod sandbox 45d401fb06d7fbf21f29d481738ef518db6ab881a001e3846eb41e53e72c62e8 from index: no such id: '0ad8c333cf50b44c4da53aa41299728b0cf39a69e7840ba53396f1deb2532a49'" containerID="0ad8c333cf50b44c4da53aa41299728b0cf39a69e7840ba53396f1deb2532a49"
Jul 27 18:25:07.159738 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:07.159700    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ad8c333cf50b44c4da53aa41299728b0cf39a69e7840ba53396f1deb2532a49"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-c58edb-predictor-7bf565bf7d-n8wcw_kserve-ci-e2e-test_1b1103da-c8f0-4c54-bd4b-0a8488b37171_0 in pod sandbox 45d401fb06d7fbf21f29d481738ef518db6ab881a001e3846eb41e53e72c62e8 from index: no such id: '0ad8c333cf50b44c4da53aa41299728b0cf39a69e7840ba53396f1deb2532a49'"
Jul 27 18:25:08.153582 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:08.153554    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-c58edb-predictor-7bf565bf7d-n8wcw_1b1103da-c8f0-4c54-bd4b-0a8488b37171/storage-initializer/1.log"
Jul 27 18:25:12.560365 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:12.560333    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-c58edb-predictor-7bf565bf7d-n8wcw"]
Jul 27 18:25:12.576763 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:12.576727    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:25:12.632363 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:12.632323    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-c58edb-predictor-6844999f8f-g9f9h"]
Jul 27 18:25:12.632686 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:12.632656    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-c58edb-predictor-6844999f8f-g9f9h" podUID="83658999-b2b6-4ac2-883b-8d8bae85d445" containerName="kserve-container" containerID="cri-o://0e621d421c426591cec041e972a17050fcfa105b212ce551679ac9ac0270ad31" gracePeriod=30
Jul 27 18:25:12.632776 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:12.632681    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-c58edb-predictor-6844999f8f-g9f9h" podUID="83658999-b2b6-4ac2-883b-8d8bae85d445" containerName="kube-rbac-proxy" containerID="cri-o://b50347c3130689498d238ebefde644c7a9b5791ef4d981571e24b736732bd0d9" gracePeriod=30
Jul 27 18:25:12.651856 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:12.651830    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:25:12.672686 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:12.672640    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-c58edb-predictor-6844999f8f-g9f9h" podUID="83658999-b2b6-4ac2-883b-8d8bae85d445" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.57:8643/healthz\": dial tcp 10.133.0.57:8643: connect: connection refused"
Jul 27 18:25:12.753725 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:12.753704    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-c58edb-predictor-7bf565bf7d-n8wcw_1b1103da-c8f0-4c54-bd4b-0a8488b37171/storage-initializer/1.log"
Jul 27 18:25:12.755854 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:12.755838    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-c58edb-predictor-7bf565bf7d-n8wcw"
Jul 27 18:25:12.896456 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:12.896418    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lg9j2\" (UniqueName: \"kubernetes.io/projected/1b1103da-c8f0-4c54-bd4b-0a8488b37171-kube-api-access-lg9j2\") pod \"1b1103da-c8f0-4c54-bd4b-0a8488b37171\" (UID: \"1b1103da-c8f0-4c54-bd4b-0a8488b37171\") "
Jul 27 18:25:12.896641 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:12.896469    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b1103da-c8f0-4c54-bd4b-0a8488b37171-cabundle-cert\") pod \"1b1103da-c8f0-4c54-bd4b-0a8488b37171\" (UID: \"1b1103da-c8f0-4c54-bd4b-0a8488b37171\") "
Jul 27 18:25:12.896641 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:12.896598    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b1103da-c8f0-4c54-bd4b-0a8488b37171-proxy-tls\") pod \"1b1103da-c8f0-4c54-bd4b-0a8488b37171\" (UID: \"1b1103da-c8f0-4c54-bd4b-0a8488b37171\") "
Jul 27 18:25:12.896718 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:12.896658    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b1103da-c8f0-4c54-bd4b-0a8488b37171-kube-rbac-proxy-sar-config\") pod \"1b1103da-c8f0-4c54-bd4b-0a8488b37171\" (UID: \"1b1103da-c8f0-4c54-bd4b-0a8488b37171\") "
Jul 27 18:25:12.896718 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:12.896703    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b1103da-c8f0-4c54-bd4b-0a8488b37171-kserve-provision-location\") pod \"1b1103da-c8f0-4c54-bd4b-0a8488b37171\" (UID: \"1b1103da-c8f0-4c54-bd4b-0a8488b37171\") "
Jul 27 18:25:12.896826 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:12.896796    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b1103da-c8f0-4c54-bd4b-0a8488b37171-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1b1103da-c8f0-4c54-bd4b-0a8488b37171" (UID: "1b1103da-c8f0-4c54-bd4b-0a8488b37171"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:25:12.897005 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:12.896984    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b1103da-c8f0-4c54-bd4b-0a8488b37171-cabundle-cert\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:25:12.897005 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:12.896991    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b1103da-c8f0-4c54-bd4b-0a8488b37171-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1b1103da-c8f0-4c54-bd4b-0a8488b37171" (UID: "1b1103da-c8f0-4c54-bd4b-0a8488b37171"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:25:12.897148 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:12.897028    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b1103da-c8f0-4c54-bd4b-0a8488b37171-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1b1103da-c8f0-4c54-bd4b-0a8488b37171" (UID: "1b1103da-c8f0-4c54-bd4b-0a8488b37171"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:25:12.898616 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:12.898590    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b1103da-c8f0-4c54-bd4b-0a8488b37171-kube-api-access-lg9j2" (OuterVolumeSpecName: "kube-api-access-lg9j2") pod "1b1103da-c8f0-4c54-bd4b-0a8488b37171" (UID: "1b1103da-c8f0-4c54-bd4b-0a8488b37171"). InnerVolumeSpecName "kube-api-access-lg9j2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:25:12.898743 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:12.898670    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b1103da-c8f0-4c54-bd4b-0a8488b37171-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1b1103da-c8f0-4c54-bd4b-0a8488b37171" (UID: "1b1103da-c8f0-4c54-bd4b-0a8488b37171"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:25:12.997951 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:12.997910    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg9j2\" (UniqueName: \"kubernetes.io/projected/1b1103da-c8f0-4c54-bd4b-0a8488b37171-kube-api-access-lg9j2\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:25:12.997951 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:12.997941    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b1103da-c8f0-4c54-bd4b-0a8488b37171-proxy-tls\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:25:12.997951 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:12.997950    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b1103da-c8f0-4c54-bd4b-0a8488b37171-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:25:12.998192 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:12.997965    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b1103da-c8f0-4c54-bd4b-0a8488b37171-kserve-provision-location\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:25:13.180539 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:13.180457    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-c58edb-predictor-7bf565bf7d-n8wcw_1b1103da-c8f0-4c54-bd4b-0a8488b37171/storage-initializer/1.log"
Jul 27 18:25:13.180724 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:13.180565    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-c58edb-predictor-7bf565bf7d-n8wcw" event={"ID":"1b1103da-c8f0-4c54-bd4b-0a8488b37171","Type":"ContainerDied","Data":"45d401fb06d7fbf21f29d481738ef518db6ab881a001e3846eb41e53e72c62e8"}
Jul 27 18:25:13.180724 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:13.180570    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-c58edb-predictor-7bf565bf7d-n8wcw"
Jul 27 18:25:13.180724 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:13.180651    2573 scope.go:117] "RemoveContainer" containerID="a73bd1c6baf8c77d87f992eae80d0f9aafb207e01709833a57e3b79a1cfa39ce"
Jul 27 18:25:13.182946 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:13.182923    2573 generic.go:356] "Generic (PLEG): container finished" podID="83658999-b2b6-4ac2-883b-8d8bae85d445" containerID="b50347c3130689498d238ebefde644c7a9b5791ef4d981571e24b736732bd0d9" exitCode=2
Jul 27 18:25:13.183092 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:13.182962    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-c58edb-predictor-6844999f8f-g9f9h" event={"ID":"83658999-b2b6-4ac2-883b-8d8bae85d445","Type":"ContainerDied","Data":"b50347c3130689498d238ebefde644c7a9b5791ef4d981571e24b736732bd0d9"}
Jul 27 18:25:13.270581 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:13.270537    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-c58edb-predictor-7bf565bf7d-n8wcw"]
Jul 27 18:25:13.277716 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:13.277686    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-c58edb-predictor-7bf565bf7d-n8wcw"]
Jul 27 18:25:14.469612 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:14.469575    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b1103da-c8f0-4c54-bd4b-0a8488b37171" path="/var/lib/kubelet/pods/1b1103da-c8f0-4c54-bd4b-0a8488b37171/volumes"
Jul 27 18:25:17.672674 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:17.672630    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-c58edb-predictor-6844999f8f-g9f9h" podUID="83658999-b2b6-4ac2-883b-8d8bae85d445" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.57:8643/healthz\": dial tcp 10.133.0.57:8643: connect: connection refused"
Jul 27 18:25:17.782069 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:17.782043    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-c58edb-predictor-6844999f8f-g9f9h"
Jul 27 18:25:17.800804 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:17.800775    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:25:17.851155 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:17.851074    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-a01701-predictor-5454df6644-pw524"]
Jul 27 18:25:17.851438 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:17.851424    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b1103da-c8f0-4c54-bd4b-0a8488b37171" containerName="storage-initializer"
Jul 27 18:25:17.851494 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:17.851439    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b1103da-c8f0-4c54-bd4b-0a8488b37171" containerName="storage-initializer"
Jul 27 18:25:17.851494 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:17.851448    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83658999-b2b6-4ac2-883b-8d8bae85d445" containerName="kube-rbac-proxy"
Jul 27 18:25:17.851494 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:17.851454    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="83658999-b2b6-4ac2-883b-8d8bae85d445" containerName="kube-rbac-proxy"
Jul 27 18:25:17.851494 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:17.851469    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83658999-b2b6-4ac2-883b-8d8bae85d445" containerName="storage-initializer"
Jul 27 18:25:17.851494 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:17.851476    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="83658999-b2b6-4ac2-883b-8d8bae85d445" containerName="storage-initializer"
Jul 27 18:25:17.851494 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:17.851482    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83658999-b2b6-4ac2-883b-8d8bae85d445" containerName="kserve-container"
Jul 27 18:25:17.851494 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:17.851487    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="83658999-b2b6-4ac2-883b-8d8bae85d445" containerName="kserve-container"
Jul 27 18:25:17.851720 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:17.851546    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b1103da-c8f0-4c54-bd4b-0a8488b37171" containerName="storage-initializer"
Jul 27 18:25:17.851720 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:17.851555    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="83658999-b2b6-4ac2-883b-8d8bae85d445" containerName="kube-rbac-proxy"
Jul 27 18:25:17.851720 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:17.851561    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b1103da-c8f0-4c54-bd4b-0a8488b37171" containerName="storage-initializer"
Jul 27 18:25:17.851720 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:17.851568    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="83658999-b2b6-4ac2-883b-8d8bae85d445" containerName="kserve-container"
Jul 27 18:25:17.851720 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:17.851638    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b1103da-c8f0-4c54-bd4b-0a8488b37171" containerName="storage-initializer"
Jul 27 18:25:17.851720 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:17.851644    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b1103da-c8f0-4c54-bd4b-0a8488b37171" containerName="storage-initializer"
Jul 27 18:25:17.854662 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:17.854625    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-a01701-predictor-5454df6644-pw524"
Jul 27 18:25:17.862380 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:17.862350    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-a01701\""
Jul 27 18:25:17.862503 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:17.862445    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-a01701-dockercfg-65spg\""
Jul 27 18:25:17.863058 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:17.863031    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-a01701-predictor-serving-cert\""
Jul 27 18:25:17.863170 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:17.863071    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-a01701-kube-rbac-proxy-sar-config\""
Jul 27 18:25:17.873826 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:17.873798    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-a01701-predictor-5454df6644-pw524"]
Jul 27 18:25:17.937834 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:17.937798    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83658999-b2b6-4ac2-883b-8d8bae85d445-kube-rbac-proxy-sar-config\") pod \"83658999-b2b6-4ac2-883b-8d8bae85d445\" (UID: \"83658999-b2b6-4ac2-883b-8d8bae85d445\") "
Jul 27 18:25:17.938016 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:17.937889    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83658999-b2b6-4ac2-883b-8d8bae85d445-cabundle-cert\") pod \"83658999-b2b6-4ac2-883b-8d8bae85d445\" (UID: \"83658999-b2b6-4ac2-883b-8d8bae85d445\") "
Jul 27 18:25:17.938016 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:17.937924    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83658999-b2b6-4ac2-883b-8d8bae85d445-proxy-tls\") pod \"83658999-b2b6-4ac2-883b-8d8bae85d445\" (UID: \"83658999-b2b6-4ac2-883b-8d8bae85d445\") "
Jul 27 18:25:17.938163 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:17.938139    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83658999-b2b6-4ac2-883b-8d8bae85d445-kserve-provision-location\") pod \"83658999-b2b6-4ac2-883b-8d8bae85d445\" (UID: \"83658999-b2b6-4ac2-883b-8d8bae85d445\") "
Jul 27 18:25:17.938237 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:17.938175    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2p5t5\" (UniqueName: \"kubernetes.io/projected/83658999-b2b6-4ac2-883b-8d8bae85d445-kube-api-access-2p5t5\") pod \"83658999-b2b6-4ac2-883b-8d8bae85d445\" (UID: \"83658999-b2b6-4ac2-883b-8d8bae85d445\") "
Jul 27 18:25:17.938237 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:17.938220    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83658999-b2b6-4ac2-883b-8d8bae85d445-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "83658999-b2b6-4ac2-883b-8d8bae85d445" (UID: "83658999-b2b6-4ac2-883b-8d8bae85d445"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:25:17.938378 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:17.938266    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83658999-b2b6-4ac2-883b-8d8bae85d445-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "83658999-b2b6-4ac2-883b-8d8bae85d445" (UID: "83658999-b2b6-4ac2-883b-8d8bae85d445"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:25:17.940451 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:17.938836    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3df339c7-6e53-41a1-b029-c18b4791b927-kserve-provision-location\") pod \"isvc-init-fail-a01701-predictor-5454df6644-pw524\" (UID: \"3df339c7-6e53-41a1-b029-c18b4791b927\") " pod="kserve-ci-e2e-test/isvc-init-fail-a01701-predictor-5454df6644-pw524"
Jul 27 18:25:17.940451 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:17.938886    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83658999-b2b6-4ac2-883b-8d8bae85d445-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "83658999-b2b6-4ac2-883b-8d8bae85d445" (UID: "83658999-b2b6-4ac2-883b-8d8bae85d445"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:25:17.940451 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:17.938971    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3df339c7-6e53-41a1-b029-c18b4791b927-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-a01701-predictor-5454df6644-pw524\" (UID: \"3df339c7-6e53-41a1-b029-c18b4791b927\") " pod="kserve-ci-e2e-test/isvc-init-fail-a01701-predictor-5454df6644-pw524"
Jul 27 18:25:17.940451 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:17.939011    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3df339c7-6e53-41a1-b029-c18b4791b927-cabundle-cert\") pod \"isvc-init-fail-a01701-predictor-5454df6644-pw524\" (UID: \"3df339c7-6e53-41a1-b029-c18b4791b927\") " pod="kserve-ci-e2e-test/isvc-init-fail-a01701-predictor-5454df6644-pw524"
Jul 27 18:25:17.940451 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:17.939079    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lrb7\" (UniqueName: \"kubernetes.io/projected/3df339c7-6e53-41a1-b029-c18b4791b927-kube-api-access-9lrb7\") pod \"isvc-init-fail-a01701-predictor-5454df6644-pw524\" (UID: \"3df339c7-6e53-41a1-b029-c18b4791b927\") " pod="kserve-ci-e2e-test/isvc-init-fail-a01701-predictor-5454df6644-pw524"
Jul 27 18:25:17.940451 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:17.939148    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3df339c7-6e53-41a1-b029-c18b4791b927-proxy-tls\") pod \"isvc-init-fail-a01701-predictor-5454df6644-pw524\" (UID: \"3df339c7-6e53-41a1-b029-c18b4791b927\") " pod="kserve-ci-e2e-test/isvc-init-fail-a01701-predictor-5454df6644-pw524"
Jul 27 18:25:17.945722 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:17.940500    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83658999-b2b6-4ac2-883b-8d8bae85d445-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "83658999-b2b6-4ac2-883b-8d8bae85d445" (UID: "83658999-b2b6-4ac2-883b-8d8bae85d445"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:25:17.945722 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:17.940636    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83658999-b2b6-4ac2-883b-8d8bae85d445-cabundle-cert\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:25:17.945722 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:17.940661    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83658999-b2b6-4ac2-883b-8d8bae85d445-kserve-provision-location\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:25:17.945722 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:17.940674    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/83658999-b2b6-4ac2-883b-8d8bae85d445-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:25:17.945722 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:17.941096    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83658999-b2b6-4ac2-883b-8d8bae85d445-kube-api-access-2p5t5" (OuterVolumeSpecName: "kube-api-access-2p5t5") pod "83658999-b2b6-4ac2-883b-8d8bae85d445" (UID: "83658999-b2b6-4ac2-883b-8d8bae85d445"). InnerVolumeSpecName "kube-api-access-2p5t5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:25:18.041922 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:18.041866    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3df339c7-6e53-41a1-b029-c18b4791b927-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-a01701-predictor-5454df6644-pw524\" (UID: \"3df339c7-6e53-41a1-b029-c18b4791b927\") " pod="kserve-ci-e2e-test/isvc-init-fail-a01701-predictor-5454df6644-pw524"
Jul 27 18:25:18.041922 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:18.041930    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3df339c7-6e53-41a1-b029-c18b4791b927-cabundle-cert\") pod \"isvc-init-fail-a01701-predictor-5454df6644-pw524\" (UID: \"3df339c7-6e53-41a1-b029-c18b4791b927\") " pod="kserve-ci-e2e-test/isvc-init-fail-a01701-predictor-5454df6644-pw524"
Jul 27 18:25:18.042203 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:18.041979    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lrb7\" (UniqueName: \"kubernetes.io/projected/3df339c7-6e53-41a1-b029-c18b4791b927-kube-api-access-9lrb7\") pod \"isvc-init-fail-a01701-predictor-5454df6644-pw524\" (UID: \"3df339c7-6e53-41a1-b029-c18b4791b927\") " pod="kserve-ci-e2e-test/isvc-init-fail-a01701-predictor-5454df6644-pw524"
Jul 27 18:25:18.042203 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:18.042029    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3df339c7-6e53-41a1-b029-c18b4791b927-proxy-tls\") pod \"isvc-init-fail-a01701-predictor-5454df6644-pw524\" (UID: \"3df339c7-6e53-41a1-b029-c18b4791b927\") " pod="kserve-ci-e2e-test/isvc-init-fail-a01701-predictor-5454df6644-pw524"
Jul 27 18:25:18.042203 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:18.042071    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3df339c7-6e53-41a1-b029-c18b4791b927-kserve-provision-location\") pod \"isvc-init-fail-a01701-predictor-5454df6644-pw524\" (UID: \"3df339c7-6e53-41a1-b029-c18b4791b927\") " pod="kserve-ci-e2e-test/isvc-init-fail-a01701-predictor-5454df6644-pw524"
Jul 27 18:25:18.042203 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:18.042116    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/83658999-b2b6-4ac2-883b-8d8bae85d445-proxy-tls\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:25:18.042203 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:18.042131    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p5t5\" (UniqueName: \"kubernetes.io/projected/83658999-b2b6-4ac2-883b-8d8bae85d445-kube-api-access-2p5t5\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:25:18.042203 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:25:18.042170    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-a01701-predictor-serving-cert: secret "isvc-init-fail-a01701-predictor-serving-cert" not found
Jul 27 18:25:18.042472 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:25:18.042289    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3df339c7-6e53-41a1-b029-c18b4791b927-proxy-tls podName:3df339c7-6e53-41a1-b029-c18b4791b927 nodeName:}" failed. No retries permitted until 2026-07-27 18:25:18.542243903 +0000 UTC m=+3460.685645951 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3df339c7-6e53-41a1-b029-c18b4791b927-proxy-tls") pod "isvc-init-fail-a01701-predictor-5454df6644-pw524" (UID: "3df339c7-6e53-41a1-b029-c18b4791b927") : secret "isvc-init-fail-a01701-predictor-serving-cert" not found
Jul 27 18:25:18.042520 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:18.042472    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3df339c7-6e53-41a1-b029-c18b4791b927-kserve-provision-location\") pod \"isvc-init-fail-a01701-predictor-5454df6644-pw524\" (UID: \"3df339c7-6e53-41a1-b029-c18b4791b927\") " pod="kserve-ci-e2e-test/isvc-init-fail-a01701-predictor-5454df6644-pw524"
Jul 27 18:25:18.042715 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:18.042695    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3df339c7-6e53-41a1-b029-c18b4791b927-cabundle-cert\") pod \"isvc-init-fail-a01701-predictor-5454df6644-pw524\" (UID: \"3df339c7-6e53-41a1-b029-c18b4791b927\") " pod="kserve-ci-e2e-test/isvc-init-fail-a01701-predictor-5454df6644-pw524"
Jul 27 18:25:18.042763 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:18.042702    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3df339c7-6e53-41a1-b029-c18b4791b927-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-a01701-predictor-5454df6644-pw524\" (UID: \"3df339c7-6e53-41a1-b029-c18b4791b927\") " pod="kserve-ci-e2e-test/isvc-init-fail-a01701-predictor-5454df6644-pw524"
Jul 27 18:25:18.057062 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:18.057029    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lrb7\" (UniqueName: \"kubernetes.io/projected/3df339c7-6e53-41a1-b029-c18b4791b927-kube-api-access-9lrb7\") pod \"isvc-init-fail-a01701-predictor-5454df6644-pw524\" (UID: \"3df339c7-6e53-41a1-b029-c18b4791b927\") " pod="kserve-ci-e2e-test/isvc-init-fail-a01701-predictor-5454df6644-pw524"
Jul 27 18:25:18.155860 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:18.155826    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:25:18.208494 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:18.208461    2573 generic.go:356] "Generic (PLEG): container finished" podID="83658999-b2b6-4ac2-883b-8d8bae85d445" containerID="0e621d421c426591cec041e972a17050fcfa105b212ce551679ac9ac0270ad31" exitCode=0
Jul 27 18:25:18.208639 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:18.208535    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-c58edb-predictor-6844999f8f-g9f9h"
Jul 27 18:25:18.208639 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:18.208541    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-c58edb-predictor-6844999f8f-g9f9h" event={"ID":"83658999-b2b6-4ac2-883b-8d8bae85d445","Type":"ContainerDied","Data":"0e621d421c426591cec041e972a17050fcfa105b212ce551679ac9ac0270ad31"}
Jul 27 18:25:18.208639 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:18.208625    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-c58edb-predictor-6844999f8f-g9f9h" event={"ID":"83658999-b2b6-4ac2-883b-8d8bae85d445","Type":"ContainerDied","Data":"589c084b05f4f3e6a4f29b8a95bf020ba28c1a4ab45ce9ca4b034c6267e2359c"}
Jul 27 18:25:18.208778 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:18.208655    2573 scope.go:117] "RemoveContainer" containerID="b50347c3130689498d238ebefde644c7a9b5791ef4d981571e24b736732bd0d9"
Jul 27 18:25:18.217609 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:18.217585    2573 scope.go:117] "RemoveContainer" containerID="0e621d421c426591cec041e972a17050fcfa105b212ce551679ac9ac0270ad31"
Jul 27 18:25:18.225503 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:18.225477    2573 scope.go:117] "RemoveContainer" containerID="122dc99245374497556267238b683c29f02cd4c937679d7b8fb2c3a3eb655596"
Jul 27 18:25:18.233567 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:18.233539    2573 scope.go:117] "RemoveContainer" containerID="b50347c3130689498d238ebefde644c7a9b5791ef4d981571e24b736732bd0d9"
Jul 27 18:25:18.233851 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:25:18.233831    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b50347c3130689498d238ebefde644c7a9b5791ef4d981571e24b736732bd0d9\": container with ID starting with b50347c3130689498d238ebefde644c7a9b5791ef4d981571e24b736732bd0d9 not found: ID does not exist" containerID="b50347c3130689498d238ebefde644c7a9b5791ef4d981571e24b736732bd0d9"
Jul 27 18:25:18.233901 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:18.233862    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b50347c3130689498d238ebefde644c7a9b5791ef4d981571e24b736732bd0d9"} err="failed to get container status \"b50347c3130689498d238ebefde644c7a9b5791ef4d981571e24b736732bd0d9\": rpc error: code = NotFound desc = could not find container \"b50347c3130689498d238ebefde644c7a9b5791ef4d981571e24b736732bd0d9\": container with ID starting with b50347c3130689498d238ebefde644c7a9b5791ef4d981571e24b736732bd0d9 not found: ID does not exist"
Jul 27 18:25:18.233901 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:18.233882    2573 scope.go:117] "RemoveContainer" containerID="0e621d421c426591cec041e972a17050fcfa105b212ce551679ac9ac0270ad31"
Jul 27 18:25:18.234158 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:25:18.234136    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e621d421c426591cec041e972a17050fcfa105b212ce551679ac9ac0270ad31\": container with ID starting with 0e621d421c426591cec041e972a17050fcfa105b212ce551679ac9ac0270ad31 not found: ID does not exist" containerID="0e621d421c426591cec041e972a17050fcfa105b212ce551679ac9ac0270ad31"
Jul 27 18:25:18.234213 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:18.234166    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e621d421c426591cec041e972a17050fcfa105b212ce551679ac9ac0270ad31"} err="failed to get container status \"0e621d421c426591cec041e972a17050fcfa105b212ce551679ac9ac0270ad31\": rpc error: code = NotFound desc = could not find container \"0e621d421c426591cec041e972a17050fcfa105b212ce551679ac9ac0270ad31\": container with ID starting with 0e621d421c426591cec041e972a17050fcfa105b212ce551679ac9ac0270ad31 not found: ID does not exist"
Jul 27 18:25:18.234213 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:18.234187    2573 scope.go:117] "RemoveContainer" containerID="122dc99245374497556267238b683c29f02cd4c937679d7b8fb2c3a3eb655596"
Jul 27 18:25:18.234559 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:25:18.234539    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"122dc99245374497556267238b683c29f02cd4c937679d7b8fb2c3a3eb655596\": container with ID starting with 122dc99245374497556267238b683c29f02cd4c937679d7b8fb2c3a3eb655596 not found: ID does not exist" containerID="122dc99245374497556267238b683c29f02cd4c937679d7b8fb2c3a3eb655596"
Jul 27 18:25:18.234602 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:18.234567    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"122dc99245374497556267238b683c29f02cd4c937679d7b8fb2c3a3eb655596"} err="failed to get container status \"122dc99245374497556267238b683c29f02cd4c937679d7b8fb2c3a3eb655596\": rpc error: code = NotFound desc = could not find container \"122dc99245374497556267238b683c29f02cd4c937679d7b8fb2c3a3eb655596\": container with ID starting with 122dc99245374497556267238b683c29f02cd4c937679d7b8fb2c3a3eb655596 not found: ID does not exist"
Jul 27 18:25:18.235358 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:18.235337    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-c58edb-predictor-6844999f8f-g9f9h"]
Jul 27 18:25:18.238996 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:18.238975    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-c58edb-predictor-6844999f8f-g9f9h"]
Jul 27 18:25:18.469935 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:18.469854    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83658999-b2b6-4ac2-883b-8d8bae85d445" path="/var/lib/kubelet/pods/83658999-b2b6-4ac2-883b-8d8bae85d445/volumes"
Jul 27 18:25:18.546410 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:18.546379    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3df339c7-6e53-41a1-b029-c18b4791b927-proxy-tls\") pod \"isvc-init-fail-a01701-predictor-5454df6644-pw524\" (UID: \"3df339c7-6e53-41a1-b029-c18b4791b927\") " pod="kserve-ci-e2e-test/isvc-init-fail-a01701-predictor-5454df6644-pw524"
Jul 27 18:25:18.548838 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:18.548814    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3df339c7-6e53-41a1-b029-c18b4791b927-proxy-tls\") pod \"isvc-init-fail-a01701-predictor-5454df6644-pw524\" (UID: \"3df339c7-6e53-41a1-b029-c18b4791b927\") " pod="kserve-ci-e2e-test/isvc-init-fail-a01701-predictor-5454df6644-pw524"
Jul 27 18:25:18.764969 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:18.764878    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-a01701-predictor-5454df6644-pw524"
Jul 27 18:25:18.910822 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:18.910791    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-a01701-predictor-5454df6644-pw524"]
Jul 27 18:25:18.913396 ip-10-0-140-0 kubenswrapper[2573]: W0727 18:25:18.913352    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3df339c7_6e53_41a1_b029_c18b4791b927.slice/crio-e407b25c79936fd1ddb7dd2d6ca29ef8969a47da534e9e6d716b32c22d61a4ff WatchSource:0}: Error finding container e407b25c79936fd1ddb7dd2d6ca29ef8969a47da534e9e6d716b32c22d61a4ff: Status 404 returned error can't find the container with id e407b25c79936fd1ddb7dd2d6ca29ef8969a47da534e9e6d716b32c22d61a4ff
Jul 27 18:25:19.215397 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:19.215351    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-a01701-predictor-5454df6644-pw524" event={"ID":"3df339c7-6e53-41a1-b029-c18b4791b927","Type":"ContainerStarted","Data":"a5eed26a338d0549a90dd2700cc54455fe1b370ab8768707c2de150d5ae844c7"}
Jul 27 18:25:19.215568 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:19.215427    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-a01701-predictor-5454df6644-pw524" event={"ID":"3df339c7-6e53-41a1-b029-c18b4791b927","Type":"ContainerStarted","Data":"e407b25c79936fd1ddb7dd2d6ca29ef8969a47da534e9e6d716b32c22d61a4ff"}
Jul 27 18:25:24.239487 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:24.239460    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-a01701-predictor-5454df6644-pw524_3df339c7-6e53-41a1-b029-c18b4791b927/storage-initializer/0.log"
Jul 27 18:25:24.239872 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:24.239499    2573 generic.go:356] "Generic (PLEG): container finished" podID="3df339c7-6e53-41a1-b029-c18b4791b927" containerID="a5eed26a338d0549a90dd2700cc54455fe1b370ab8768707c2de150d5ae844c7" exitCode=1
Jul 27 18:25:24.239872 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:24.239525    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-a01701-predictor-5454df6644-pw524" event={"ID":"3df339c7-6e53-41a1-b029-c18b4791b927","Type":"ContainerDied","Data":"a5eed26a338d0549a90dd2700cc54455fe1b370ab8768707c2de150d5ae844c7"}
Jul 27 18:25:25.245687 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:25.245660    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-a01701-predictor-5454df6644-pw524_3df339c7-6e53-41a1-b029-c18b4791b927/storage-initializer/0.log"
Jul 27 18:25:25.246075 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:25.245728    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-a01701-predictor-5454df6644-pw524" event={"ID":"3df339c7-6e53-41a1-b029-c18b4791b927","Type":"ContainerStarted","Data":"8a33293ce5d9ecdcbc904399ebc2b31ebdf4f8f0803d791095f64da8d2e44743"}
Jul 27 18:25:27.783800 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:27.783760    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-a01701-predictor-5454df6644-pw524"]
Jul 27 18:25:27.784300 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:27.784024    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-a01701-predictor-5454df6644-pw524" podUID="3df339c7-6e53-41a1-b029-c18b4791b927" containerName="storage-initializer" containerID="cri-o://8a33293ce5d9ecdcbc904399ebc2b31ebdf4f8f0803d791095f64da8d2e44743" gracePeriod=30
Jul 27 18:25:27.815580 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:27.815547    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:25:30.535702 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:30.535669    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-a01701-predictor-5454df6644-pw524_3df339c7-6e53-41a1-b029-c18b4791b927/storage-initializer/1.log"
Jul 27 18:25:30.536130 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:30.536114    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-a01701-predictor-5454df6644-pw524_3df339c7-6e53-41a1-b029-c18b4791b927/storage-initializer/0.log"
Jul 27 18:25:30.538412 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:30.538398    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-a01701-predictor-5454df6644-pw524"
Jul 27 18:25:30.648975 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:30.648938    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3df339c7-6e53-41a1-b029-c18b4791b927-kserve-provision-location\") pod \"3df339c7-6e53-41a1-b029-c18b4791b927\" (UID: \"3df339c7-6e53-41a1-b029-c18b4791b927\") "
Jul 27 18:25:30.649178 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:30.648993    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9lrb7\" (UniqueName: \"kubernetes.io/projected/3df339c7-6e53-41a1-b029-c18b4791b927-kube-api-access-9lrb7\") pod \"3df339c7-6e53-41a1-b029-c18b4791b927\" (UID: \"3df339c7-6e53-41a1-b029-c18b4791b927\") "
Jul 27 18:25:30.649178 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:30.649018    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3df339c7-6e53-41a1-b029-c18b4791b927-kube-rbac-proxy-sar-config\") pod \"3df339c7-6e53-41a1-b029-c18b4791b927\" (UID: \"3df339c7-6e53-41a1-b029-c18b4791b927\") "
Jul 27 18:25:30.649178 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:30.649081    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3df339c7-6e53-41a1-b029-c18b4791b927-cabundle-cert\") pod \"3df339c7-6e53-41a1-b029-c18b4791b927\" (UID: \"3df339c7-6e53-41a1-b029-c18b4791b927\") "
Jul 27 18:25:30.649178 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:30.649101    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3df339c7-6e53-41a1-b029-c18b4791b927-proxy-tls\") pod \"3df339c7-6e53-41a1-b029-c18b4791b927\" (UID: \"3df339c7-6e53-41a1-b029-c18b4791b927\") "
Jul 27 18:25:30.649419 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:30.649338    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3df339c7-6e53-41a1-b029-c18b4791b927-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3df339c7-6e53-41a1-b029-c18b4791b927" (UID: "3df339c7-6e53-41a1-b029-c18b4791b927"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:25:30.649494 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:30.649455    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3df339c7-6e53-41a1-b029-c18b4791b927-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3df339c7-6e53-41a1-b029-c18b4791b927" (UID: "3df339c7-6e53-41a1-b029-c18b4791b927"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:25:30.649531 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:30.649478    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3df339c7-6e53-41a1-b029-c18b4791b927-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3df339c7-6e53-41a1-b029-c18b4791b927" (UID: "3df339c7-6e53-41a1-b029-c18b4791b927"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:25:30.651269 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:30.651175    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3df339c7-6e53-41a1-b029-c18b4791b927-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3df339c7-6e53-41a1-b029-c18b4791b927" (UID: "3df339c7-6e53-41a1-b029-c18b4791b927"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:25:30.651269 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:30.651259    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3df339c7-6e53-41a1-b029-c18b4791b927-kube-api-access-9lrb7" (OuterVolumeSpecName: "kube-api-access-9lrb7") pod "3df339c7-6e53-41a1-b029-c18b4791b927" (UID: "3df339c7-6e53-41a1-b029-c18b4791b927"). InnerVolumeSpecName "kube-api-access-9lrb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:25:30.750214 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:30.750175    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3df339c7-6e53-41a1-b029-c18b4791b927-kserve-provision-location\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:25:30.750214 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:30.750206    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lrb7\" (UniqueName: \"kubernetes.io/projected/3df339c7-6e53-41a1-b029-c18b4791b927-kube-api-access-9lrb7\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:25:30.750214 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:30.750215    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3df339c7-6e53-41a1-b029-c18b4791b927-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:25:30.750214 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:30.750224    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3df339c7-6e53-41a1-b029-c18b4791b927-cabundle-cert\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:25:30.750516 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:30.750233    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3df339c7-6e53-41a1-b029-c18b4791b927-proxy-tls\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:25:31.273425 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:31.273398    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-a01701-predictor-5454df6644-pw524_3df339c7-6e53-41a1-b029-c18b4791b927/storage-initializer/1.log"
Jul 27 18:25:31.273822 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:31.273803    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-a01701-predictor-5454df6644-pw524_3df339c7-6e53-41a1-b029-c18b4791b927/storage-initializer/0.log"
Jul 27 18:25:31.273891 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:31.273844    2573 generic.go:356] "Generic (PLEG): container finished" podID="3df339c7-6e53-41a1-b029-c18b4791b927" containerID="8a33293ce5d9ecdcbc904399ebc2b31ebdf4f8f0803d791095f64da8d2e44743" exitCode=1
Jul 27 18:25:31.273931 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:31.273908    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-a01701-predictor-5454df6644-pw524"
Jul 27 18:25:31.273974 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:31.273925    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-a01701-predictor-5454df6644-pw524" event={"ID":"3df339c7-6e53-41a1-b029-c18b4791b927","Type":"ContainerDied","Data":"8a33293ce5d9ecdcbc904399ebc2b31ebdf4f8f0803d791095f64da8d2e44743"}
Jul 27 18:25:31.274011 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:31.273974    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-a01701-predictor-5454df6644-pw524" event={"ID":"3df339c7-6e53-41a1-b029-c18b4791b927","Type":"ContainerDied","Data":"e407b25c79936fd1ddb7dd2d6ca29ef8969a47da534e9e6d716b32c22d61a4ff"}
Jul 27 18:25:31.274011 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:31.274002    2573 scope.go:117] "RemoveContainer" containerID="8a33293ce5d9ecdcbc904399ebc2b31ebdf4f8f0803d791095f64da8d2e44743"
Jul 27 18:25:31.282630 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:31.282609    2573 scope.go:117] "RemoveContainer" containerID="a5eed26a338d0549a90dd2700cc54455fe1b370ab8768707c2de150d5ae844c7"
Jul 27 18:25:31.290606 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:31.290589    2573 scope.go:117] "RemoveContainer" containerID="8a33293ce5d9ecdcbc904399ebc2b31ebdf4f8f0803d791095f64da8d2e44743"
Jul 27 18:25:31.290861 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:25:31.290842    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a33293ce5d9ecdcbc904399ebc2b31ebdf4f8f0803d791095f64da8d2e44743\": container with ID starting with 8a33293ce5d9ecdcbc904399ebc2b31ebdf4f8f0803d791095f64da8d2e44743 not found: ID does not exist" containerID="8a33293ce5d9ecdcbc904399ebc2b31ebdf4f8f0803d791095f64da8d2e44743"
Jul 27 18:25:31.290935 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:31.290874    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a33293ce5d9ecdcbc904399ebc2b31ebdf4f8f0803d791095f64da8d2e44743"} err="failed to get container status \"8a33293ce5d9ecdcbc904399ebc2b31ebdf4f8f0803d791095f64da8d2e44743\": rpc error: code = NotFound desc = could not find container \"8a33293ce5d9ecdcbc904399ebc2b31ebdf4f8f0803d791095f64da8d2e44743\": container with ID starting with 8a33293ce5d9ecdcbc904399ebc2b31ebdf4f8f0803d791095f64da8d2e44743 not found: ID does not exist"
Jul 27 18:25:31.290935 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:31.290900    2573 scope.go:117] "RemoveContainer" containerID="a5eed26a338d0549a90dd2700cc54455fe1b370ab8768707c2de150d5ae844c7"
Jul 27 18:25:31.291145 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:25:31.291127    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5eed26a338d0549a90dd2700cc54455fe1b370ab8768707c2de150d5ae844c7\": container with ID starting with a5eed26a338d0549a90dd2700cc54455fe1b370ab8768707c2de150d5ae844c7 not found: ID does not exist" containerID="a5eed26a338d0549a90dd2700cc54455fe1b370ab8768707c2de150d5ae844c7"
Jul 27 18:25:31.291181 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:31.291154    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5eed26a338d0549a90dd2700cc54455fe1b370ab8768707c2de150d5ae844c7"} err="failed to get container status \"a5eed26a338d0549a90dd2700cc54455fe1b370ab8768707c2de150d5ae844c7\": rpc error: code = NotFound desc = could not find container \"a5eed26a338d0549a90dd2700cc54455fe1b370ab8768707c2de150d5ae844c7\": container with ID starting with a5eed26a338d0549a90dd2700cc54455fe1b370ab8768707c2de150d5ae844c7 not found: ID does not exist"
Jul 27 18:25:31.312093 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:31.312054    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-a01701-predictor-5454df6644-pw524"]
Jul 27 18:25:31.316389 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:31.316359    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-a01701-predictor-5454df6644-pw524"]
Jul 27 18:25:32.142081 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:32.142047    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:25:32.469444 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:32.469361    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3df339c7-6e53-41a1-b029-c18b4791b927" path="/var/lib/kubelet/pods/3df339c7-6e53-41a1-b029-c18b4791b927/volumes"
Jul 27 18:25:32.982352 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:32.982321    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:25:33.026827 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:33.026792    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-nz7pm"]
Jul 27 18:25:33.027138 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:33.027126    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3df339c7-6e53-41a1-b029-c18b4791b927" containerName="storage-initializer"
Jul 27 18:25:33.027179 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:33.027139    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3df339c7-6e53-41a1-b029-c18b4791b927" containerName="storage-initializer"
Jul 27 18:25:33.027221 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:33.027200    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3df339c7-6e53-41a1-b029-c18b4791b927" containerName="storage-initializer"
Jul 27 18:25:33.027303 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:33.027292    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3df339c7-6e53-41a1-b029-c18b4791b927" containerName="storage-initializer"
Jul 27 18:25:33.027303 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:33.027302    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3df339c7-6e53-41a1-b029-c18b4791b927" containerName="storage-initializer"
Jul 27 18:25:33.027373 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:33.027352    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3df339c7-6e53-41a1-b029-c18b4791b927" containerName="storage-initializer"
Jul 27 18:25:33.031768 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:33.031749    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-nz7pm"
Jul 27 18:25:33.035675 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:33.035655    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
Jul 27 18:25:33.035795 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:33.035705    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 18:25:33.036783 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:33.036765    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 18:25:33.037379 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:33.037354    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
Jul 27 18:25:33.037468 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:33.037382    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z6ld7\""
Jul 27 18:25:33.037764 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:33.037746    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 18:25:33.038108 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:33.038091    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 18:25:33.042162 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:33.042140    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-nz7pm"]
Jul 27 18:25:33.070425 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:33.070402    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94df01d5-07a3-49fe-9014-c4b394e04aaa-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-nz7pm\" (UID: \"94df01d5-07a3-49fe-9014-c4b394e04aaa\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-nz7pm"
Jul 27 18:25:33.070542 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:33.070453    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94df01d5-07a3-49fe-9014-c4b394e04aaa-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-nz7pm\" (UID: \"94df01d5-07a3-49fe-9014-c4b394e04aaa\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-nz7pm"
Jul 27 18:25:33.070542 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:33.070475    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94df01d5-07a3-49fe-9014-c4b394e04aaa-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-nz7pm\" (UID: \"94df01d5-07a3-49fe-9014-c4b394e04aaa\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-nz7pm"
Jul 27 18:25:33.070542 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:33.070524    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwv66\" (UniqueName: \"kubernetes.io/projected/94df01d5-07a3-49fe-9014-c4b394e04aaa-kube-api-access-bwv66\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-nz7pm\" (UID: \"94df01d5-07a3-49fe-9014-c4b394e04aaa\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-nz7pm"
Jul 27 18:25:33.070679 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:33.070574    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94df01d5-07a3-49fe-9014-c4b394e04aaa-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-nz7pm\" (UID: \"94df01d5-07a3-49fe-9014-c4b394e04aaa\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-nz7pm"
Jul 27 18:25:33.171053 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:33.171017    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94df01d5-07a3-49fe-9014-c4b394e04aaa-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-nz7pm\" (UID: \"94df01d5-07a3-49fe-9014-c4b394e04aaa\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-nz7pm"
Jul 27 18:25:33.171053 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:33.171055    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94df01d5-07a3-49fe-9014-c4b394e04aaa-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-nz7pm\" (UID: \"94df01d5-07a3-49fe-9014-c4b394e04aaa\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-nz7pm"
Jul 27 18:25:33.171541 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:33.171092    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwv66\" (UniqueName: \"kubernetes.io/projected/94df01d5-07a3-49fe-9014-c4b394e04aaa-kube-api-access-bwv66\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-nz7pm\" (UID: \"94df01d5-07a3-49fe-9014-c4b394e04aaa\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-nz7pm"
Jul 27 18:25:33.171541 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:33.171131    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94df01d5-07a3-49fe-9014-c4b394e04aaa-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-nz7pm\" (UID: \"94df01d5-07a3-49fe-9014-c4b394e04aaa\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-nz7pm"
Jul 27 18:25:33.171541 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:33.171176    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94df01d5-07a3-49fe-9014-c4b394e04aaa-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-nz7pm\" (UID: \"94df01d5-07a3-49fe-9014-c4b394e04aaa\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-nz7pm"
Jul 27 18:25:33.171541 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:25:33.171307    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-serving-cert: secret "isvc-predictive-sklearn-predictor-serving-cert" not found
Jul 27 18:25:33.171541 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:25:33.171367    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94df01d5-07a3-49fe-9014-c4b394e04aaa-proxy-tls podName:94df01d5-07a3-49fe-9014-c4b394e04aaa nodeName:}" failed. No retries permitted until 2026-07-27 18:25:33.671346344 +0000 UTC m=+3475.814748379 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/94df01d5-07a3-49fe-9014-c4b394e04aaa-proxy-tls") pod "isvc-predictive-sklearn-predictor-5f849d69df-nz7pm" (UID: "94df01d5-07a3-49fe-9014-c4b394e04aaa") : secret "isvc-predictive-sklearn-predictor-serving-cert" not found
Jul 27 18:25:33.171709 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:33.171602    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94df01d5-07a3-49fe-9014-c4b394e04aaa-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-nz7pm\" (UID: \"94df01d5-07a3-49fe-9014-c4b394e04aaa\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-nz7pm"
Jul 27 18:25:33.171786 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:33.171769    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94df01d5-07a3-49fe-9014-c4b394e04aaa-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-nz7pm\" (UID: \"94df01d5-07a3-49fe-9014-c4b394e04aaa\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-nz7pm"
Jul 27 18:25:33.171824 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:33.171801    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94df01d5-07a3-49fe-9014-c4b394e04aaa-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-nz7pm\" (UID: \"94df01d5-07a3-49fe-9014-c4b394e04aaa\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-nz7pm"
Jul 27 18:25:33.186494 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:33.186459    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwv66\" (UniqueName: \"kubernetes.io/projected/94df01d5-07a3-49fe-9014-c4b394e04aaa-kube-api-access-bwv66\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-nz7pm\" (UID: \"94df01d5-07a3-49fe-9014-c4b394e04aaa\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-nz7pm"
Jul 27 18:25:33.233552 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:33.233462    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:25:33.676130 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:33.676095    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94df01d5-07a3-49fe-9014-c4b394e04aaa-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-nz7pm\" (UID: \"94df01d5-07a3-49fe-9014-c4b394e04aaa\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-nz7pm"
Jul 27 18:25:33.678605 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:33.678584    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94df01d5-07a3-49fe-9014-c4b394e04aaa-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-nz7pm\" (UID: \"94df01d5-07a3-49fe-9014-c4b394e04aaa\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-nz7pm"
Jul 27 18:25:33.943101 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:33.942997    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-nz7pm"
Jul 27 18:25:34.084798 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:34.084770    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-nz7pm"]
Jul 27 18:25:34.087395 ip-10-0-140-0 kubenswrapper[2573]: W0727 18:25:34.087355    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94df01d5_07a3_49fe_9014_c4b394e04aaa.slice/crio-00fd067e39fecce955ddd044db32af5d15bf104a766526002944f2632b02193b WatchSource:0}: Error finding container 00fd067e39fecce955ddd044db32af5d15bf104a766526002944f2632b02193b: Status 404 returned error can't find the container with id 00fd067e39fecce955ddd044db32af5d15bf104a766526002944f2632b02193b
Jul 27 18:25:34.290480 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:34.290395    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-nz7pm" event={"ID":"94df01d5-07a3-49fe-9014-c4b394e04aaa","Type":"ContainerStarted","Data":"c362e07959cd56130a1d0159746d889558ecee776a1aaed8f332fa5bf0241b20"}
Jul 27 18:25:34.290833 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:34.290464    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-nz7pm" event={"ID":"94df01d5-07a3-49fe-9014-c4b394e04aaa","Type":"ContainerStarted","Data":"00fd067e39fecce955ddd044db32af5d15bf104a766526002944f2632b02193b"}
Jul 27 18:25:38.309318 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:38.309203    2573 generic.go:356] "Generic (PLEG): container finished" podID="94df01d5-07a3-49fe-9014-c4b394e04aaa" containerID="c362e07959cd56130a1d0159746d889558ecee776a1aaed8f332fa5bf0241b20" exitCode=0
Jul 27 18:25:38.309318 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:38.309278    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-nz7pm" event={"ID":"94df01d5-07a3-49fe-9014-c4b394e04aaa","Type":"ContainerDied","Data":"c362e07959cd56130a1d0159746d889558ecee776a1aaed8f332fa5bf0241b20"}
Jul 27 18:25:38.310578 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:38.310562    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 18:25:42.907534 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:25:42.907455    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:26:02.449693 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:26:02.449601    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-nz7pm" event={"ID":"94df01d5-07a3-49fe-9014-c4b394e04aaa","Type":"ContainerStarted","Data":"897b36df73c4e4ebae2120ea531a87b54dc913f2ca94e386f5b5543e67d271ba"}
Jul 27 18:26:02.449693 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:26:02.449668    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-nz7pm" event={"ID":"94df01d5-07a3-49fe-9014-c4b394e04aaa","Type":"ContainerStarted","Data":"639d3e5ece5ad841aa40c47cf0e6681629a778b71223ec7959030f10ba67315a"}
Jul 27 18:26:02.450141 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:26:02.449863    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-nz7pm"
Jul 27 18:26:02.489216 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:26:02.489163    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-nz7pm" podStartSLOduration=1.875141595 podStartE2EDuration="29.48914628s" totalImagesPullingTime="23.849858454s" totalInitContainerRuntime="3.764146231s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:25:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:26:02.487705995 +0000 UTC m=+3504.631108048" watchObservedRunningTime="2026-07-27 18:26:02.48914628 +0000 UTC m=+3504.632548339"
Jul 27 18:26:03.454279 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:26:03.454238    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-nz7pm"
Jul 27 18:26:03.455713 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:26:03.455684    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-nz7pm" podUID="94df01d5-07a3-49fe-9014-c4b394e04aaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Jul 27 18:26:04.459499 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:26:04.459459    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-nz7pm" podUID="94df01d5-07a3-49fe-9014-c4b394e04aaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Jul 27 18:26:09.463879 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:26:09.463838    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-nz7pm"
Jul 27 18:26:09.464392 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:26:09.464366    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-nz7pm" podUID="94df01d5-07a3-49fe-9014-c4b394e04aaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Jul 27 18:26:19.464838 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:26:19.464800    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-nz7pm" podUID="94df01d5-07a3-49fe-9014-c4b394e04aaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Jul 27 18:26:29.465147 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:26:29.465102    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-nz7pm" podUID="94df01d5-07a3-49fe-9014-c4b394e04aaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Jul 27 18:26:39.464710 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:26:39.464669    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-nz7pm" podUID="94df01d5-07a3-49fe-9014-c4b394e04aaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Jul 27 18:26:49.464892 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:26:49.464854    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-nz7pm" podUID="94df01d5-07a3-49fe-9014-c4b394e04aaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Jul 27 18:26:59.464675 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:26:59.464636    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-nz7pm" podUID="94df01d5-07a3-49fe-9014-c4b394e04aaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Jul 27 18:27:09.464837 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:09.464793    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-nz7pm" podUID="94df01d5-07a3-49fe-9014-c4b394e04aaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Jul 27 18:27:19.465440 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:19.465412    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-nz7pm"
Jul 27 18:27:23.125912 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:23.125872    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-nz7pm"]
Jul 27 18:27:23.126359 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:23.126091    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-nz7pm" podUID="94df01d5-07a3-49fe-9014-c4b394e04aaa" containerName="kserve-container" containerID="cri-o://639d3e5ece5ad841aa40c47cf0e6681629a778b71223ec7959030f10ba67315a" gracePeriod=30
Jul 27 18:27:23.126359 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:23.126160    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-nz7pm" podUID="94df01d5-07a3-49fe-9014-c4b394e04aaa" containerName="kube-rbac-proxy" containerID="cri-o://897b36df73c4e4ebae2120ea531a87b54dc913f2ca94e386f5b5543e67d271ba" gracePeriod=30
Jul 27 18:27:23.161475 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:23.161445    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:27:23.808507 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:23.808472    2573 generic.go:356] "Generic (PLEG): container finished" podID="94df01d5-07a3-49fe-9014-c4b394e04aaa" containerID="897b36df73c4e4ebae2120ea531a87b54dc913f2ca94e386f5b5543e67d271ba" exitCode=2
Jul 27 18:27:23.808688 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:23.808544    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-nz7pm" event={"ID":"94df01d5-07a3-49fe-9014-c4b394e04aaa","Type":"ContainerDied","Data":"897b36df73c4e4ebae2120ea531a87b54dc913f2ca94e386f5b5543e67d271ba"}
Jul 27 18:27:24.460642 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:24.460601    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-nz7pm" podUID="94df01d5-07a3-49fe-9014-c4b394e04aaa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.60:8643/healthz\": dial tcp 10.133.0.60:8643: connect: connection refused"
Jul 27 18:27:28.203791 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:28.203760    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:27:28.278040 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:28.277979    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7s8hx"]
Jul 27 18:27:28.280746 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:28.280721    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7s8hx"
Jul 27 18:27:28.283502 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:28.283474    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
Jul 27 18:27:28.283660 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:28.283642    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
Jul 27 18:27:28.287621 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:28.287594    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-nz7pm"
Jul 27 18:27:28.289632 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:28.289605    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7s8hx"]
Jul 27 18:27:28.322300 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:28.322221    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:27:28.371284 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:28.371168    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94df01d5-07a3-49fe-9014-c4b394e04aaa-kube-rbac-proxy-sar-config\") pod \"94df01d5-07a3-49fe-9014-c4b394e04aaa\" (UID: \"94df01d5-07a3-49fe-9014-c4b394e04aaa\") "
Jul 27 18:27:28.371284 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:28.371202    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bwv66\" (UniqueName: \"kubernetes.io/projected/94df01d5-07a3-49fe-9014-c4b394e04aaa-kube-api-access-bwv66\") pod \"94df01d5-07a3-49fe-9014-c4b394e04aaa\" (UID: \"94df01d5-07a3-49fe-9014-c4b394e04aaa\") "
Jul 27 18:27:28.371531 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:28.371332    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94df01d5-07a3-49fe-9014-c4b394e04aaa-proxy-tls\") pod \"94df01d5-07a3-49fe-9014-c4b394e04aaa\" (UID: \"94df01d5-07a3-49fe-9014-c4b394e04aaa\") "
Jul 27 18:27:28.371531 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:28.371378    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94df01d5-07a3-49fe-9014-c4b394e04aaa-kserve-provision-location\") pod \"94df01d5-07a3-49fe-9014-c4b394e04aaa\" (UID: \"94df01d5-07a3-49fe-9014-c4b394e04aaa\") "
Jul 27 18:27:28.371531 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:28.371467    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94df01d5-07a3-49fe-9014-c4b394e04aaa-cabundle-cert\") pod \"94df01d5-07a3-49fe-9014-c4b394e04aaa\" (UID: \"94df01d5-07a3-49fe-9014-c4b394e04aaa\") "
Jul 27 18:27:28.371696 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:28.371571    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94df01d5-07a3-49fe-9014-c4b394e04aaa-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "94df01d5-07a3-49fe-9014-c4b394e04aaa" (UID: "94df01d5-07a3-49fe-9014-c4b394e04aaa"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:27:28.371696 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:28.371600    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b609a892-a10c-4859-b454-6e279e130f29-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-7s8hx\" (UID: \"b609a892-a10c-4859-b454-6e279e130f29\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7s8hx"
Jul 27 18:27:28.371791 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:28.371723    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b609a892-a10c-4859-b454-6e279e130f29-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-7s8hx\" (UID: \"b609a892-a10c-4859-b454-6e279e130f29\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7s8hx"
Jul 27 18:27:28.371791 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:28.371739    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94df01d5-07a3-49fe-9014-c4b394e04aaa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "94df01d5-07a3-49fe-9014-c4b394e04aaa" (UID: "94df01d5-07a3-49fe-9014-c4b394e04aaa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:27:28.371876 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:28.371802    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b609a892-a10c-4859-b454-6e279e130f29-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-7s8hx\" (UID: \"b609a892-a10c-4859-b454-6e279e130f29\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7s8hx"
Jul 27 18:27:28.371876 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:28.371824    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p9jx\" (UniqueName: \"kubernetes.io/projected/b609a892-a10c-4859-b454-6e279e130f29-kube-api-access-2p9jx\") pod \"isvc-predictive-xgboost-predictor-695f978978-7s8hx\" (UID: \"b609a892-a10c-4859-b454-6e279e130f29\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7s8hx"
Jul 27 18:27:28.371876 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:28.371848    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b609a892-a10c-4859-b454-6e279e130f29-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-7s8hx\" (UID: \"b609a892-a10c-4859-b454-6e279e130f29\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7s8hx"
Jul 27 18:27:28.371876 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:28.371854    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94df01d5-07a3-49fe-9014-c4b394e04aaa-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "94df01d5-07a3-49fe-9014-c4b394e04aaa" (UID: "94df01d5-07a3-49fe-9014-c4b394e04aaa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:27:28.372057 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:28.371945    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94df01d5-07a3-49fe-9014-c4b394e04aaa-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:27:28.372057 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:28.371966    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94df01d5-07a3-49fe-9014-c4b394e04aaa-kserve-provision-location\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:27:28.372057 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:28.371981    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94df01d5-07a3-49fe-9014-c4b394e04aaa-cabundle-cert\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:27:28.373434 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:28.373414    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94df01d5-07a3-49fe-9014-c4b394e04aaa-kube-api-access-bwv66" (OuterVolumeSpecName: "kube-api-access-bwv66") pod "94df01d5-07a3-49fe-9014-c4b394e04aaa" (UID: "94df01d5-07a3-49fe-9014-c4b394e04aaa"). InnerVolumeSpecName "kube-api-access-bwv66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:27:28.373505 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:28.373465    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94df01d5-07a3-49fe-9014-c4b394e04aaa-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "94df01d5-07a3-49fe-9014-c4b394e04aaa" (UID: "94df01d5-07a3-49fe-9014-c4b394e04aaa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:27:28.472290 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:28.472257    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b609a892-a10c-4859-b454-6e279e130f29-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-7s8hx\" (UID: \"b609a892-a10c-4859-b454-6e279e130f29\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7s8hx"
Jul 27 18:27:28.472476 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:28.472323    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b609a892-a10c-4859-b454-6e279e130f29-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-7s8hx\" (UID: \"b609a892-a10c-4859-b454-6e279e130f29\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7s8hx"
Jul 27 18:27:28.472476 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:28.472357    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b609a892-a10c-4859-b454-6e279e130f29-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-7s8hx\" (UID: \"b609a892-a10c-4859-b454-6e279e130f29\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7s8hx"
Jul 27 18:27:28.472476 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:28.472373    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p9jx\" (UniqueName: \"kubernetes.io/projected/b609a892-a10c-4859-b454-6e279e130f29-kube-api-access-2p9jx\") pod \"isvc-predictive-xgboost-predictor-695f978978-7s8hx\" (UID: \"b609a892-a10c-4859-b454-6e279e130f29\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7s8hx"
Jul 27 18:27:28.472476 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:28.472396    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b609a892-a10c-4859-b454-6e279e130f29-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-7s8hx\" (UID: \"b609a892-a10c-4859-b454-6e279e130f29\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7s8hx"
Jul 27 18:27:28.472476 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:28.472449    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwv66\" (UniqueName: \"kubernetes.io/projected/94df01d5-07a3-49fe-9014-c4b394e04aaa-kube-api-access-bwv66\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:27:28.472476 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:28.472466    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94df01d5-07a3-49fe-9014-c4b394e04aaa-proxy-tls\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:27:28.472805 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:28.472622    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b609a892-a10c-4859-b454-6e279e130f29-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-7s8hx\" (UID: \"b609a892-a10c-4859-b454-6e279e130f29\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7s8hx"
Jul 27 18:27:28.473108 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:28.473084    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b609a892-a10c-4859-b454-6e279e130f29-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-7s8hx\" (UID: \"b609a892-a10c-4859-b454-6e279e130f29\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7s8hx"
Jul 27 18:27:28.473190 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:28.473118    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b609a892-a10c-4859-b454-6e279e130f29-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-7s8hx\" (UID: \"b609a892-a10c-4859-b454-6e279e130f29\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7s8hx"
Jul 27 18:27:28.475012 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:28.474994    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b609a892-a10c-4859-b454-6e279e130f29-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-7s8hx\" (UID: \"b609a892-a10c-4859-b454-6e279e130f29\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7s8hx"
Jul 27 18:27:28.482307 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:28.482279    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p9jx\" (UniqueName: \"kubernetes.io/projected/b609a892-a10c-4859-b454-6e279e130f29-kube-api-access-2p9jx\") pod \"isvc-predictive-xgboost-predictor-695f978978-7s8hx\" (UID: \"b609a892-a10c-4859-b454-6e279e130f29\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7s8hx"
Jul 27 18:27:28.594479 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:28.594440    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7s8hx"
Jul 27 18:27:28.731325 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:28.731292    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7s8hx"]
Jul 27 18:27:28.733823 ip-10-0-140-0 kubenswrapper[2573]: W0727 18:27:28.733796    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb609a892_a10c_4859_b454_6e279e130f29.slice/crio-3e20c221fbb6c4ba243c36d504004506b0bb6dffdc94ad6276f96173dc662170 WatchSource:0}: Error finding container 3e20c221fbb6c4ba243c36d504004506b0bb6dffdc94ad6276f96173dc662170: Status 404 returned error can't find the container with id 3e20c221fbb6c4ba243c36d504004506b0bb6dffdc94ad6276f96173dc662170
Jul 27 18:27:28.832069 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:28.832031    2573 generic.go:356] "Generic (PLEG): container finished" podID="94df01d5-07a3-49fe-9014-c4b394e04aaa" containerID="639d3e5ece5ad841aa40c47cf0e6681629a778b71223ec7959030f10ba67315a" exitCode=0
Jul 27 18:27:28.832298 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:28.832085    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-nz7pm"
Jul 27 18:27:28.832298 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:28.832113    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-nz7pm" event={"ID":"94df01d5-07a3-49fe-9014-c4b394e04aaa","Type":"ContainerDied","Data":"639d3e5ece5ad841aa40c47cf0e6681629a778b71223ec7959030f10ba67315a"}
Jul 27 18:27:28.832298 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:28.832171    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-nz7pm" event={"ID":"94df01d5-07a3-49fe-9014-c4b394e04aaa","Type":"ContainerDied","Data":"00fd067e39fecce955ddd044db32af5d15bf104a766526002944f2632b02193b"}
Jul 27 18:27:28.832298 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:28.832185    2573 scope.go:117] "RemoveContainer" containerID="897b36df73c4e4ebae2120ea531a87b54dc913f2ca94e386f5b5543e67d271ba"
Jul 27 18:27:28.834027 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:28.833999    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7s8hx" event={"ID":"b609a892-a10c-4859-b454-6e279e130f29","Type":"ContainerStarted","Data":"52efa5fb727d76332329e282b31633d75e1fbc66df4dc32ffc3995879df0bf68"}
Jul 27 18:27:28.834151 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:28.834064    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7s8hx" event={"ID":"b609a892-a10c-4859-b454-6e279e130f29","Type":"ContainerStarted","Data":"3e20c221fbb6c4ba243c36d504004506b0bb6dffdc94ad6276f96173dc662170"}
Jul 27 18:27:28.840847 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:28.840827    2573 scope.go:117] "RemoveContainer" containerID="639d3e5ece5ad841aa40c47cf0e6681629a778b71223ec7959030f10ba67315a"
Jul 27 18:27:28.848840 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:28.848816    2573 scope.go:117] "RemoveContainer" containerID="c362e07959cd56130a1d0159746d889558ecee776a1aaed8f332fa5bf0241b20"
Jul 27 18:27:28.851508 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:28.851488    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-nz7pm"]
Jul 27 18:27:28.856785 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:28.856762    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-nz7pm"]
Jul 27 18:27:28.857468 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:28.857324    2573 scope.go:117] "RemoveContainer" containerID="897b36df73c4e4ebae2120ea531a87b54dc913f2ca94e386f5b5543e67d271ba"
Jul 27 18:27:28.857660 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:27:28.857640    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"897b36df73c4e4ebae2120ea531a87b54dc913f2ca94e386f5b5543e67d271ba\": container with ID starting with 897b36df73c4e4ebae2120ea531a87b54dc913f2ca94e386f5b5543e67d271ba not found: ID does not exist" containerID="897b36df73c4e4ebae2120ea531a87b54dc913f2ca94e386f5b5543e67d271ba"
Jul 27 18:27:28.857702 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:28.857670    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"897b36df73c4e4ebae2120ea531a87b54dc913f2ca94e386f5b5543e67d271ba"} err="failed to get container status \"897b36df73c4e4ebae2120ea531a87b54dc913f2ca94e386f5b5543e67d271ba\": rpc error: code = NotFound desc = could not find container \"897b36df73c4e4ebae2120ea531a87b54dc913f2ca94e386f5b5543e67d271ba\": container with ID starting with 897b36df73c4e4ebae2120ea531a87b54dc913f2ca94e386f5b5543e67d271ba not found: ID does not exist"
Jul 27 18:27:28.857702 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:28.857687    2573 scope.go:117] "RemoveContainer" containerID="639d3e5ece5ad841aa40c47cf0e6681629a778b71223ec7959030f10ba67315a"
Jul 27 18:27:28.857987 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:27:28.857967    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"639d3e5ece5ad841aa40c47cf0e6681629a778b71223ec7959030f10ba67315a\": container with ID starting with 639d3e5ece5ad841aa40c47cf0e6681629a778b71223ec7959030f10ba67315a not found: ID does not exist" containerID="639d3e5ece5ad841aa40c47cf0e6681629a778b71223ec7959030f10ba67315a"
Jul 27 18:27:28.858055 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:28.857993    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"639d3e5ece5ad841aa40c47cf0e6681629a778b71223ec7959030f10ba67315a"} err="failed to get container status \"639d3e5ece5ad841aa40c47cf0e6681629a778b71223ec7959030f10ba67315a\": rpc error: code = NotFound desc = could not find container \"639d3e5ece5ad841aa40c47cf0e6681629a778b71223ec7959030f10ba67315a\": container with ID starting with 639d3e5ece5ad841aa40c47cf0e6681629a778b71223ec7959030f10ba67315a not found: ID does not exist"
Jul 27 18:27:28.858055 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:28.858017    2573 scope.go:117] "RemoveContainer" containerID="c362e07959cd56130a1d0159746d889558ecee776a1aaed8f332fa5bf0241b20"
Jul 27 18:27:28.858408 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:27:28.858381    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c362e07959cd56130a1d0159746d889558ecee776a1aaed8f332fa5bf0241b20\": container with ID starting with c362e07959cd56130a1d0159746d889558ecee776a1aaed8f332fa5bf0241b20 not found: ID does not exist" containerID="c362e07959cd56130a1d0159746d889558ecee776a1aaed8f332fa5bf0241b20"
Jul 27 18:27:28.858506 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:28.858415    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c362e07959cd56130a1d0159746d889558ecee776a1aaed8f332fa5bf0241b20"} err="failed to get container status \"c362e07959cd56130a1d0159746d889558ecee776a1aaed8f332fa5bf0241b20\": rpc error: code = NotFound desc = could not find container \"c362e07959cd56130a1d0159746d889558ecee776a1aaed8f332fa5bf0241b20\": container with ID starting with c362e07959cd56130a1d0159746d889558ecee776a1aaed8f332fa5bf0241b20 not found: ID does not exist"
Jul 27 18:27:30.470545 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:30.470507    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94df01d5-07a3-49fe-9014-c4b394e04aaa" path="/var/lib/kubelet/pods/94df01d5-07a3-49fe-9014-c4b394e04aaa/volumes"
Jul 27 18:27:32.853571 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:32.853478    2573 generic.go:356] "Generic (PLEG): container finished" podID="b609a892-a10c-4859-b454-6e279e130f29" containerID="52efa5fb727d76332329e282b31633d75e1fbc66df4dc32ffc3995879df0bf68" exitCode=0
Jul 27 18:27:32.853571 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:32.853559    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7s8hx" event={"ID":"b609a892-a10c-4859-b454-6e279e130f29","Type":"ContainerDied","Data":"52efa5fb727d76332329e282b31633d75e1fbc66df4dc32ffc3995879df0bf68"}
Jul 27 18:27:33.860468 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:33.860425    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7s8hx" event={"ID":"b609a892-a10c-4859-b454-6e279e130f29","Type":"ContainerStarted","Data":"35ea51c6fe75a774f141787ada3b20583eec96969446a5f70aa91c6b41a54c9c"}
Jul 27 18:27:33.860869 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:33.860526    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7s8hx" event={"ID":"b609a892-a10c-4859-b454-6e279e130f29","Type":"ContainerStarted","Data":"111196c73b1a0b37e9202157ea2243252c198aa82c52744bfa62f68b6a4c9424"}
Jul 27 18:27:33.860869 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:33.860725    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7s8hx"
Jul 27 18:27:33.883699 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:33.883646    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7s8hx" podStartSLOduration=2.191780898 podStartE2EDuration="5.883627651s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.691846753s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:27:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:27:33.882164795 +0000 UTC m=+3596.025566867" watchObservedRunningTime="2026-07-27 18:27:33.883627651 +0000 UTC m=+3596.027029776"
Jul 27 18:27:34.864952 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:34.864921    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7s8hx"
Jul 27 18:27:34.866380 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:34.866356    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7s8hx" podUID="b609a892-a10c-4859-b454-6e279e130f29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Jul 27 18:27:35.870490 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:35.870454    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7s8hx" podUID="b609a892-a10c-4859-b454-6e279e130f29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Jul 27 18:27:38.544312 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:38.544279    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vc2tz_87dac797-f8d2-4d36-999f-f8ce855dbeff/ovn-acl-logging/0.log"
Jul 27 18:27:38.551882 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:38.551862    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vc2tz_87dac797-f8d2-4d36-999f-f8ce855dbeff/ovn-acl-logging/0.log"
Jul 27 18:27:40.875017 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:40.874990    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7s8hx"
Jul 27 18:27:40.875494 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:40.875467    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7s8hx" podUID="b609a892-a10c-4859-b454-6e279e130f29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Jul 27 18:27:50.876120 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:27:50.876084    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7s8hx" podUID="b609a892-a10c-4859-b454-6e279e130f29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Jul 27 18:28:00.875877 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:28:00.875839    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7s8hx" podUID="b609a892-a10c-4859-b454-6e279e130f29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Jul 27 18:28:10.876266 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:28:10.876220    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7s8hx" podUID="b609a892-a10c-4859-b454-6e279e130f29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Jul 27 18:28:20.875935 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:28:20.875897    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7s8hx" podUID="b609a892-a10c-4859-b454-6e279e130f29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Jul 27 18:28:30.875648 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:28:30.875600    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7s8hx" podUID="b609a892-a10c-4859-b454-6e279e130f29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Jul 27 18:28:40.875523 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:28:40.875477    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7s8hx" podUID="b609a892-a10c-4859-b454-6e279e130f29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Jul 27 18:28:50.469604 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:28:50.469576    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7s8hx"
Jul 27 18:28:58.350652 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:28:58.350620    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7s8hx"]
Jul 27 18:28:58.351263 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:28:58.350866    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7s8hx" podUID="b609a892-a10c-4859-b454-6e279e130f29" containerName="kserve-container" containerID="cri-o://111196c73b1a0b37e9202157ea2243252c198aa82c52744bfa62f68b6a4c9424" gracePeriod=30
Jul 27 18:28:58.351263 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:28:58.350908    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7s8hx" podUID="b609a892-a10c-4859-b454-6e279e130f29" containerName="kube-rbac-proxy" containerID="cri-o://35ea51c6fe75a774f141787ada3b20583eec96969446a5f70aa91c6b41a54c9c" gracePeriod=30
Jul 27 18:28:58.393937 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:28:58.393904    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:28:59.225990 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:28:59.225958    2573 generic.go:356] "Generic (PLEG): container finished" podID="b609a892-a10c-4859-b454-6e279e130f29" containerID="35ea51c6fe75a774f141787ada3b20583eec96969446a5f70aa91c6b41a54c9c" exitCode=2
Jul 27 18:28:59.226190 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:28:59.226040    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7s8hx" event={"ID":"b609a892-a10c-4859-b454-6e279e130f29","Type":"ContainerDied","Data":"35ea51c6fe75a774f141787ada3b20583eec96969446a5f70aa91c6b41a54c9c"}
Jul 27 18:28:59.760503 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:28:59.760468    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:29:00.465301 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:00.465233    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7s8hx" podUID="b609a892-a10c-4859-b454-6e279e130f29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.61:8080: connect: connection refused"
Jul 27 18:29:00.871714 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:00.871617    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7s8hx" podUID="b609a892-a10c-4859-b454-6e279e130f29" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.61:8643/healthz\": dial tcp 10.133.0.61:8643: connect: connection refused"
Jul 27 18:29:03.493661 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:03.493634    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7s8hx"
Jul 27 18:29:03.523673 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:03.523639    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:29:03.551901 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:03.551822    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck"]
Jul 27 18:29:03.552282 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:03.552232    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94df01d5-07a3-49fe-9014-c4b394e04aaa" containerName="kube-rbac-proxy"
Jul 27 18:29:03.552282 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:03.552272    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="94df01d5-07a3-49fe-9014-c4b394e04aaa" containerName="kube-rbac-proxy"
Jul 27 18:29:03.552440 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:03.552287    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94df01d5-07a3-49fe-9014-c4b394e04aaa" containerName="kserve-container"
Jul 27 18:29:03.552440 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:03.552296    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="94df01d5-07a3-49fe-9014-c4b394e04aaa" containerName="kserve-container"
Jul 27 18:29:03.552440 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:03.552305    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b609a892-a10c-4859-b454-6e279e130f29" containerName="kube-rbac-proxy"
Jul 27 18:29:03.552440 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:03.552312    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b609a892-a10c-4859-b454-6e279e130f29" containerName="kube-rbac-proxy"
Jul 27 18:29:03.552440 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:03.552325    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b609a892-a10c-4859-b454-6e279e130f29" containerName="kserve-container"
Jul 27 18:29:03.552440 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:03.552332    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b609a892-a10c-4859-b454-6e279e130f29" containerName="kserve-container"
Jul 27 18:29:03.552440 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:03.552356    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94df01d5-07a3-49fe-9014-c4b394e04aaa" containerName="storage-initializer"
Jul 27 18:29:03.552440 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:03.552364    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="94df01d5-07a3-49fe-9014-c4b394e04aaa" containerName="storage-initializer"
Jul 27 18:29:03.552440 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:03.552391    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b609a892-a10c-4859-b454-6e279e130f29" containerName="storage-initializer"
Jul 27 18:29:03.552440 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:03.552399    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b609a892-a10c-4859-b454-6e279e130f29" containerName="storage-initializer"
Jul 27 18:29:03.552929 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:03.552496    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="94df01d5-07a3-49fe-9014-c4b394e04aaa" containerName="kube-rbac-proxy"
Jul 27 18:29:03.552929 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:03.552511    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b609a892-a10c-4859-b454-6e279e130f29" containerName="kserve-container"
Jul 27 18:29:03.552929 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:03.552523    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="94df01d5-07a3-49fe-9014-c4b394e04aaa" containerName="kserve-container"
Jul 27 18:29:03.552929 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:03.552534    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b609a892-a10c-4859-b454-6e279e130f29" containerName="kube-rbac-proxy"
Jul 27 18:29:03.554866 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:03.554846    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck"
Jul 27 18:29:03.557905 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:03.557883    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
Jul 27 18:29:03.563557 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:03.563528    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b609a892-a10c-4859-b454-6e279e130f29-kube-rbac-proxy-sar-config\") pod \"b609a892-a10c-4859-b454-6e279e130f29\" (UID: \"b609a892-a10c-4859-b454-6e279e130f29\") "
Jul 27 18:29:03.563665 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:03.563615    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b609a892-a10c-4859-b454-6e279e130f29-cabundle-cert\") pod \"b609a892-a10c-4859-b454-6e279e130f29\" (UID: \"b609a892-a10c-4859-b454-6e279e130f29\") "
Jul 27 18:29:03.563665 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:03.563651    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b609a892-a10c-4859-b454-6e279e130f29-proxy-tls\") pod \"b609a892-a10c-4859-b454-6e279e130f29\" (UID: \"b609a892-a10c-4859-b454-6e279e130f29\") "
Jul 27 18:29:03.563772 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:03.563673    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b609a892-a10c-4859-b454-6e279e130f29-kserve-provision-location\") pod \"b609a892-a10c-4859-b454-6e279e130f29\" (UID: \"b609a892-a10c-4859-b454-6e279e130f29\") "
Jul 27 18:29:03.563772 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:03.563700    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2p9jx\" (UniqueName: \"kubernetes.io/projected/b609a892-a10c-4859-b454-6e279e130f29-kube-api-access-2p9jx\") pod \"b609a892-a10c-4859-b454-6e279e130f29\" (UID: \"b609a892-a10c-4859-b454-6e279e130f29\") "
Jul 27 18:29:03.563885 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:03.563818    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b609a892-a10c-4859-b454-6e279e130f29-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b609a892-a10c-4859-b454-6e279e130f29" (UID: "b609a892-a10c-4859-b454-6e279e130f29"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:29:03.564041 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:03.564016    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b609a892-a10c-4859-b454-6e279e130f29-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b609a892-a10c-4859-b454-6e279e130f29" (UID: "b609a892-a10c-4859-b454-6e279e130f29"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:29:03.564101 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:03.564054    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b609a892-a10c-4859-b454-6e279e130f29-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b609a892-a10c-4859-b454-6e279e130f29" (UID: "b609a892-a10c-4859-b454-6e279e130f29"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:29:03.564101 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:03.564078    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b609a892-a10c-4859-b454-6e279e130f29-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:29:03.565836 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:03.565805    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b609a892-a10c-4859-b454-6e279e130f29-kube-api-access-2p9jx" (OuterVolumeSpecName: "kube-api-access-2p9jx") pod "b609a892-a10c-4859-b454-6e279e130f29" (UID: "b609a892-a10c-4859-b454-6e279e130f29"). InnerVolumeSpecName "kube-api-access-2p9jx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:29:03.565929 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:03.565866    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b609a892-a10c-4859-b454-6e279e130f29-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b609a892-a10c-4859-b454-6e279e130f29" (UID: "b609a892-a10c-4859-b454-6e279e130f29"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:29:03.566144 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:03.566126    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
Jul 27 18:29:03.572307 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:03.572285    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck"]
Jul 27 18:29:03.665071 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:03.665036    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f35e3a2-9f78-4e4d-b60b-508668b4539a-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck\" (UID: \"0f35e3a2-9f78-4e4d-b60b-508668b4539a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck"
Jul 27 18:29:03.665311 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:03.665095    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f35e3a2-9f78-4e4d-b60b-508668b4539a-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck\" (UID: \"0f35e3a2-9f78-4e4d-b60b-508668b4539a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck"
Jul 27 18:29:03.665311 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:03.665214    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f35e3a2-9f78-4e4d-b60b-508668b4539a-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck\" (UID: \"0f35e3a2-9f78-4e4d-b60b-508668b4539a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck"
Jul 27 18:29:03.665311 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:03.665287    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f35e3a2-9f78-4e4d-b60b-508668b4539a-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck\" (UID: \"0f35e3a2-9f78-4e4d-b60b-508668b4539a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck"
Jul 27 18:29:03.665460 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:03.665328    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbrk2\" (UniqueName: \"kubernetes.io/projected/0f35e3a2-9f78-4e4d-b60b-508668b4539a-kube-api-access-wbrk2\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck\" (UID: \"0f35e3a2-9f78-4e4d-b60b-508668b4539a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck"
Jul 27 18:29:03.665460 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:03.665389    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b609a892-a10c-4859-b454-6e279e130f29-cabundle-cert\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:29:03.665460 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:03.665400    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b609a892-a10c-4859-b454-6e279e130f29-proxy-tls\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:29:03.665460 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:03.665409    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b609a892-a10c-4859-b454-6e279e130f29-kserve-provision-location\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:29:03.665460 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:03.665419    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p9jx\" (UniqueName: \"kubernetes.io/projected/b609a892-a10c-4859-b454-6e279e130f29-kube-api-access-2p9jx\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:29:03.766115 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:03.766079    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f35e3a2-9f78-4e4d-b60b-508668b4539a-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck\" (UID: \"0f35e3a2-9f78-4e4d-b60b-508668b4539a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck"
Jul 27 18:29:03.766115 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:03.766119    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f35e3a2-9f78-4e4d-b60b-508668b4539a-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck\" (UID: \"0f35e3a2-9f78-4e4d-b60b-508668b4539a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck"
Jul 27 18:29:03.766388 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:03.766144    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbrk2\" (UniqueName: \"kubernetes.io/projected/0f35e3a2-9f78-4e4d-b60b-508668b4539a-kube-api-access-wbrk2\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck\" (UID: \"0f35e3a2-9f78-4e4d-b60b-508668b4539a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck"
Jul 27 18:29:03.766388 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:29:03.766241    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-serving-cert: secret "isvc-predictive-lightgbm-predictor-serving-cert" not found
Jul 27 18:29:03.766388 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:03.766280    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f35e3a2-9f78-4e4d-b60b-508668b4539a-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck\" (UID: \"0f35e3a2-9f78-4e4d-b60b-508668b4539a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck"
Jul 27 18:29:03.766388 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:29:03.766345    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f35e3a2-9f78-4e4d-b60b-508668b4539a-proxy-tls podName:0f35e3a2-9f78-4e4d-b60b-508668b4539a nodeName:}" failed. No retries permitted until 2026-07-27 18:29:04.266323016 +0000 UTC m=+3686.409725045 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0f35e3a2-9f78-4e4d-b60b-508668b4539a-proxy-tls") pod "isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck" (UID: "0f35e3a2-9f78-4e4d-b60b-508668b4539a") : secret "isvc-predictive-lightgbm-predictor-serving-cert" not found
Jul 27 18:29:03.766388 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:03.766363    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f35e3a2-9f78-4e4d-b60b-508668b4539a-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck\" (UID: \"0f35e3a2-9f78-4e4d-b60b-508668b4539a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck"
Jul 27 18:29:03.766721 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:03.766702    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f35e3a2-9f78-4e4d-b60b-508668b4539a-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck\" (UID: \"0f35e3a2-9f78-4e4d-b60b-508668b4539a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck"
Jul 27 18:29:03.766943 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:03.766923    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f35e3a2-9f78-4e4d-b60b-508668b4539a-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck\" (UID: \"0f35e3a2-9f78-4e4d-b60b-508668b4539a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck"
Jul 27 18:29:03.767015 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:03.766973    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f35e3a2-9f78-4e4d-b60b-508668b4539a-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck\" (UID: \"0f35e3a2-9f78-4e4d-b60b-508668b4539a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck"
Jul 27 18:29:03.775548 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:03.775528    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbrk2\" (UniqueName: \"kubernetes.io/projected/0f35e3a2-9f78-4e4d-b60b-508668b4539a-kube-api-access-wbrk2\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck\" (UID: \"0f35e3a2-9f78-4e4d-b60b-508668b4539a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck"
Jul 27 18:29:03.919118 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:03.919082    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:29:04.248642 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:04.248544    2573 generic.go:356] "Generic (PLEG): container finished" podID="b609a892-a10c-4859-b454-6e279e130f29" containerID="111196c73b1a0b37e9202157ea2243252c198aa82c52744bfa62f68b6a4c9424" exitCode=0
Jul 27 18:29:04.248642 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:04.248601    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7s8hx"
Jul 27 18:29:04.248642 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:04.248617    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7s8hx" event={"ID":"b609a892-a10c-4859-b454-6e279e130f29","Type":"ContainerDied","Data":"111196c73b1a0b37e9202157ea2243252c198aa82c52744bfa62f68b6a4c9424"}
Jul 27 18:29:04.248876 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:04.248681    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7s8hx" event={"ID":"b609a892-a10c-4859-b454-6e279e130f29","Type":"ContainerDied","Data":"3e20c221fbb6c4ba243c36d504004506b0bb6dffdc94ad6276f96173dc662170"}
Jul 27 18:29:04.248876 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:04.248709    2573 scope.go:117] "RemoveContainer" containerID="35ea51c6fe75a774f141787ada3b20583eec96969446a5f70aa91c6b41a54c9c"
Jul 27 18:29:04.257334 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:04.257314    2573 scope.go:117] "RemoveContainer" containerID="111196c73b1a0b37e9202157ea2243252c198aa82c52744bfa62f68b6a4c9424"
Jul 27 18:29:04.265072 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:04.265055    2573 scope.go:117] "RemoveContainer" containerID="52efa5fb727d76332329e282b31633d75e1fbc66df4dc32ffc3995879df0bf68"
Jul 27 18:29:04.270364 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:04.270332    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f35e3a2-9f78-4e4d-b60b-508668b4539a-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck\" (UID: \"0f35e3a2-9f78-4e4d-b60b-508668b4539a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck"
Jul 27 18:29:04.272550 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:04.272524    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7s8hx"]
Jul 27 18:29:04.272999 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:04.272984    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f35e3a2-9f78-4e4d-b60b-508668b4539a-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck\" (UID: \"0f35e3a2-9f78-4e4d-b60b-508668b4539a\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck"
Jul 27 18:29:04.273757 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:04.273742    2573 scope.go:117] "RemoveContainer" containerID="35ea51c6fe75a774f141787ada3b20583eec96969446a5f70aa91c6b41a54c9c"
Jul 27 18:29:04.273989 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:04.273968    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-7s8hx"]
Jul 27 18:29:04.274080 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:29:04.274062    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35ea51c6fe75a774f141787ada3b20583eec96969446a5f70aa91c6b41a54c9c\": container with ID starting with 35ea51c6fe75a774f141787ada3b20583eec96969446a5f70aa91c6b41a54c9c not found: ID does not exist" containerID="35ea51c6fe75a774f141787ada3b20583eec96969446a5f70aa91c6b41a54c9c"
Jul 27 18:29:04.274122 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:04.274090    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35ea51c6fe75a774f141787ada3b20583eec96969446a5f70aa91c6b41a54c9c"} err="failed to get container status \"35ea51c6fe75a774f141787ada3b20583eec96969446a5f70aa91c6b41a54c9c\": rpc error: code = NotFound desc = could not find container \"35ea51c6fe75a774f141787ada3b20583eec96969446a5f70aa91c6b41a54c9c\": container with ID starting with 35ea51c6fe75a774f141787ada3b20583eec96969446a5f70aa91c6b41a54c9c not found: ID does not exist"
Jul 27 18:29:04.274122 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:04.274109    2573 scope.go:117] "RemoveContainer" containerID="111196c73b1a0b37e9202157ea2243252c198aa82c52744bfa62f68b6a4c9424"
Jul 27 18:29:04.274450 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:29:04.274431    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"111196c73b1a0b37e9202157ea2243252c198aa82c52744bfa62f68b6a4c9424\": container with ID starting with 111196c73b1a0b37e9202157ea2243252c198aa82c52744bfa62f68b6a4c9424 not found: ID does not exist" containerID="111196c73b1a0b37e9202157ea2243252c198aa82c52744bfa62f68b6a4c9424"
Jul 27 18:29:04.274515 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:04.274456    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"111196c73b1a0b37e9202157ea2243252c198aa82c52744bfa62f68b6a4c9424"} err="failed to get container status \"111196c73b1a0b37e9202157ea2243252c198aa82c52744bfa62f68b6a4c9424\": rpc error: code = NotFound desc = could not find container \"111196c73b1a0b37e9202157ea2243252c198aa82c52744bfa62f68b6a4c9424\": container with ID starting with 111196c73b1a0b37e9202157ea2243252c198aa82c52744bfa62f68b6a4c9424 not found: ID does not exist"
Jul 27 18:29:04.274515 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:04.274472    2573 scope.go:117] "RemoveContainer" containerID="52efa5fb727d76332329e282b31633d75e1fbc66df4dc32ffc3995879df0bf68"
Jul 27 18:29:04.274722 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:29:04.274703    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52efa5fb727d76332329e282b31633d75e1fbc66df4dc32ffc3995879df0bf68\": container with ID starting with 52efa5fb727d76332329e282b31633d75e1fbc66df4dc32ffc3995879df0bf68 not found: ID does not exist" containerID="52efa5fb727d76332329e282b31633d75e1fbc66df4dc32ffc3995879df0bf68"
Jul 27 18:29:04.274768 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:04.274726    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52efa5fb727d76332329e282b31633d75e1fbc66df4dc32ffc3995879df0bf68"} err="failed to get container status \"52efa5fb727d76332329e282b31633d75e1fbc66df4dc32ffc3995879df0bf68\": rpc error: code = NotFound desc = could not find container \"52efa5fb727d76332329e282b31633d75e1fbc66df4dc32ffc3995879df0bf68\": container with ID starting with 52efa5fb727d76332329e282b31633d75e1fbc66df4dc32ffc3995879df0bf68 not found: ID does not exist"
Jul 27 18:29:04.465164 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:04.465130    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck"
Jul 27 18:29:04.469294 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:04.469272    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b609a892-a10c-4859-b454-6e279e130f29" path="/var/lib/kubelet/pods/b609a892-a10c-4859-b454-6e279e130f29/volumes"
Jul 27 18:29:04.604705 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:04.604684    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck"]
Jul 27 18:29:04.607355 ip-10-0-140-0 kubenswrapper[2573]: W0727 18:29:04.607326    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f35e3a2_9f78_4e4d_b60b_508668b4539a.slice/crio-546a6da7f6a3b1e68ab0ff9ee686e28cd48dd2de84679f58415b0fde9fd9f07b WatchSource:0}: Error finding container 546a6da7f6a3b1e68ab0ff9ee686e28cd48dd2de84679f58415b0fde9fd9f07b: Status 404 returned error can't find the container with id 546a6da7f6a3b1e68ab0ff9ee686e28cd48dd2de84679f58415b0fde9fd9f07b
Jul 27 18:29:05.253923 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:05.253888    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck" event={"ID":"0f35e3a2-9f78-4e4d-b60b-508668b4539a","Type":"ContainerStarted","Data":"bf7388ade1918881ccb4ded7a2e204a635dc995b104d18e3ac1a093754870ea3"}
Jul 27 18:29:05.254123 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:05.253957    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck" event={"ID":"0f35e3a2-9f78-4e4d-b60b-508668b4539a","Type":"ContainerStarted","Data":"546a6da7f6a3b1e68ab0ff9ee686e28cd48dd2de84679f58415b0fde9fd9f07b"}
Jul 27 18:29:09.274112 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:09.274077    2573 generic.go:356] "Generic (PLEG): container finished" podID="0f35e3a2-9f78-4e4d-b60b-508668b4539a" containerID="bf7388ade1918881ccb4ded7a2e204a635dc995b104d18e3ac1a093754870ea3" exitCode=0
Jul 27 18:29:09.274634 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:09.274149    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck" event={"ID":"0f35e3a2-9f78-4e4d-b60b-508668b4539a","Type":"ContainerDied","Data":"bf7388ade1918881ccb4ded7a2e204a635dc995b104d18e3ac1a093754870ea3"}
Jul 27 18:29:10.280918 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:10.280885    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck" event={"ID":"0f35e3a2-9f78-4e4d-b60b-508668b4539a","Type":"ContainerStarted","Data":"90c2458f944c19c0d70a630ccbe09eaca82bdb54b1337389687ffb67efb28ba0"}
Jul 27 18:29:10.281350 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:10.280950    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck" event={"ID":"0f35e3a2-9f78-4e4d-b60b-508668b4539a","Type":"ContainerStarted","Data":"d98615041536b9c6f9b78e67573ba35a69f0163cdc05cd4b3c16bb8b8fbf947c"}
Jul 27 18:29:10.281350 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:10.281281    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck"
Jul 27 18:29:10.281442 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:10.281424    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck"
Jul 27 18:29:10.282910 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:10.282885    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck" podUID="0f35e3a2-9f78-4e4d-b60b-508668b4539a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Jul 27 18:29:10.305190 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:10.305131    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck" podStartSLOduration=3.611699294 podStartE2EDuration="7.305114633s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.693415339s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:29:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:29:10.304072828 +0000 UTC m=+3692.447474880" watchObservedRunningTime="2026-07-27 18:29:10.305114633 +0000 UTC m=+3692.448516680"
Jul 27 18:29:11.286048 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:11.286006    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck" podUID="0f35e3a2-9f78-4e4d-b60b-508668b4539a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Jul 27 18:29:16.290573 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:16.290543    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck"
Jul 27 18:29:16.291078 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:16.291050    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck" podUID="0f35e3a2-9f78-4e4d-b60b-508668b4539a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Jul 27 18:29:26.291371 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:26.291331    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck" podUID="0f35e3a2-9f78-4e4d-b60b-508668b4539a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Jul 27 18:29:36.291579 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:36.291536    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck" podUID="0f35e3a2-9f78-4e4d-b60b-508668b4539a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Jul 27 18:29:46.291505 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:46.291464    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck" podUID="0f35e3a2-9f78-4e4d-b60b-508668b4539a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Jul 27 18:29:56.291043 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:29:56.291005    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck" podUID="0f35e3a2-9f78-4e4d-b60b-508668b4539a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Jul 27 18:30:06.291346 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:06.291300    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck" podUID="0f35e3a2-9f78-4e4d-b60b-508668b4539a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Jul 27 18:30:16.291764 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:16.291723    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck" podUID="0f35e3a2-9f78-4e4d-b60b-508668b4539a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Jul 27 18:30:24.469427 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:24.469395    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck"
Jul 27 18:30:33.640013 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:33.639917    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck"]
Jul 27 18:30:33.640594 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:33.640188    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck" podUID="0f35e3a2-9f78-4e4d-b60b-508668b4539a" containerName="kserve-container" containerID="cri-o://d98615041536b9c6f9b78e67573ba35a69f0163cdc05cd4b3c16bb8b8fbf947c" gracePeriod=30
Jul 27 18:30:33.640594 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:33.640353    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck" podUID="0f35e3a2-9f78-4e4d-b60b-508668b4539a" containerName="kube-rbac-proxy" containerID="cri-o://90c2458f944c19c0d70a630ccbe09eaca82bdb54b1337389687ffb67efb28ba0" gracePeriod=30
Jul 27 18:30:33.667138 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:33.667105    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:30:34.469501 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:34.469465    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck" podUID="0f35e3a2-9f78-4e4d-b60b-508668b4539a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.62:8080: connect: connection refused"
Jul 27 18:30:34.661187 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:34.661148    2573 generic.go:356] "Generic (PLEG): container finished" podID="0f35e3a2-9f78-4e4d-b60b-508668b4539a" containerID="90c2458f944c19c0d70a630ccbe09eaca82bdb54b1337389687ffb67efb28ba0" exitCode=2
Jul 27 18:30:34.661600 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:34.661220    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck" event={"ID":"0f35e3a2-9f78-4e4d-b60b-508668b4539a","Type":"ContainerDied","Data":"90c2458f944c19c0d70a630ccbe09eaca82bdb54b1337389687ffb67efb28ba0"}
Jul 27 18:30:36.286880 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:36.286832    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck" podUID="0f35e3a2-9f78-4e4d-b60b-508668b4539a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.62:8643/healthz\": dial tcp 10.133.0.62:8643: connect: connection refused"
Jul 27 18:30:38.680420 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:38.680382    2573 generic.go:356] "Generic (PLEG): container finished" podID="0f35e3a2-9f78-4e4d-b60b-508668b4539a" containerID="d98615041536b9c6f9b78e67573ba35a69f0163cdc05cd4b3c16bb8b8fbf947c" exitCode=0
Jul 27 18:30:38.680871 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:38.680449    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck" event={"ID":"0f35e3a2-9f78-4e4d-b60b-508668b4539a","Type":"ContainerDied","Data":"d98615041536b9c6f9b78e67573ba35a69f0163cdc05cd4b3c16bb8b8fbf947c"}
Jul 27 18:30:38.773790 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:38.773682    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:30:38.846837 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:38.846805    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j"]
Jul 27 18:30:38.850505 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:38.850480    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j"
Jul 27 18:30:38.853399 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:38.853378    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\""
Jul 27 18:30:38.853928 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:38.853909    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\""
Jul 27 18:30:38.863518 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:38.863491    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j"]
Jul 27 18:30:38.871747 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:38.871719    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/996961a9-cceb-4096-ba61-157534de65fa-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j\" (UID: \"996961a9-cceb-4096-ba61-157534de65fa\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j"
Jul 27 18:30:38.871889 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:38.871763    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pd4s\" (UniqueName: \"kubernetes.io/projected/996961a9-cceb-4096-ba61-157534de65fa-kube-api-access-2pd4s\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j\" (UID: \"996961a9-cceb-4096-ba61-157534de65fa\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j"
Jul 27 18:30:38.871889 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:38.871788    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/996961a9-cceb-4096-ba61-157534de65fa-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j\" (UID: \"996961a9-cceb-4096-ba61-157534de65fa\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j"
Jul 27 18:30:38.871889 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:38.871815    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/996961a9-cceb-4096-ba61-157534de65fa-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j\" (UID: \"996961a9-cceb-4096-ba61-157534de65fa\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j"
Jul 27 18:30:38.871889 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:38.871855    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/996961a9-cceb-4096-ba61-157534de65fa-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j\" (UID: \"996961a9-cceb-4096-ba61-157534de65fa\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j"
Jul 27 18:30:38.972993 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:38.972950    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pd4s\" (UniqueName: \"kubernetes.io/projected/996961a9-cceb-4096-ba61-157534de65fa-kube-api-access-2pd4s\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j\" (UID: \"996961a9-cceb-4096-ba61-157534de65fa\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j"
Jul 27 18:30:38.972993 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:38.972991    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/996961a9-cceb-4096-ba61-157534de65fa-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j\" (UID: \"996961a9-cceb-4096-ba61-157534de65fa\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j"
Jul 27 18:30:38.973276 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:38.973119    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/996961a9-cceb-4096-ba61-157534de65fa-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j\" (UID: \"996961a9-cceb-4096-ba61-157534de65fa\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j"
Jul 27 18:30:38.973276 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:38.973206    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/996961a9-cceb-4096-ba61-157534de65fa-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j\" (UID: \"996961a9-cceb-4096-ba61-157534de65fa\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j"
Jul 27 18:30:38.973276 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:38.973268    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/996961a9-cceb-4096-ba61-157534de65fa-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j\" (UID: \"996961a9-cceb-4096-ba61-157534de65fa\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j"
Jul 27 18:30:38.973453 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:30:38.973273    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-serving-cert: secret "isvc-predictive-sklearn-v2-predictor-serving-cert" not found
Jul 27 18:30:38.973453 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:38.973395    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/996961a9-cceb-4096-ba61-157534de65fa-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j\" (UID: \"996961a9-cceb-4096-ba61-157534de65fa\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j"
Jul 27 18:30:38.973453 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:30:38.973413    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/996961a9-cceb-4096-ba61-157534de65fa-proxy-tls podName:996961a9-cceb-4096-ba61-157534de65fa nodeName:}" failed. No retries permitted until 2026-07-27 18:30:39.47339132 +0000 UTC m=+3781.616793355 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/996961a9-cceb-4096-ba61-157534de65fa-proxy-tls") pod "isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j" (UID: "996961a9-cceb-4096-ba61-157534de65fa") : secret "isvc-predictive-sklearn-v2-predictor-serving-cert" not found
Jul 27 18:30:38.973888 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:38.973864    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/996961a9-cceb-4096-ba61-157534de65fa-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j\" (UID: \"996961a9-cceb-4096-ba61-157534de65fa\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j"
Jul 27 18:30:38.973975 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:38.973882    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/996961a9-cceb-4096-ba61-157534de65fa-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j\" (UID: \"996961a9-cceb-4096-ba61-157534de65fa\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j"
Jul 27 18:30:38.982940 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:38.982894    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pd4s\" (UniqueName: \"kubernetes.io/projected/996961a9-cceb-4096-ba61-157534de65fa-kube-api-access-2pd4s\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j\" (UID: \"996961a9-cceb-4096-ba61-157534de65fa\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j"
Jul 27 18:30:39.089309 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:39.089282    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck"
Jul 27 18:30:39.100504 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:39.100473    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:30:39.175277 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:39.175213    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f35e3a2-9f78-4e4d-b60b-508668b4539a-cabundle-cert\") pod \"0f35e3a2-9f78-4e4d-b60b-508668b4539a\" (UID: \"0f35e3a2-9f78-4e4d-b60b-508668b4539a\") "
Jul 27 18:30:39.175482 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:39.175297    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f35e3a2-9f78-4e4d-b60b-508668b4539a-kserve-provision-location\") pod \"0f35e3a2-9f78-4e4d-b60b-508668b4539a\" (UID: \"0f35e3a2-9f78-4e4d-b60b-508668b4539a\") "
Jul 27 18:30:39.175482 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:39.175330    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f35e3a2-9f78-4e4d-b60b-508668b4539a-kube-rbac-proxy-sar-config\") pod \"0f35e3a2-9f78-4e4d-b60b-508668b4539a\" (UID: \"0f35e3a2-9f78-4e4d-b60b-508668b4539a\") "
Jul 27 18:30:39.175482 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:39.175354    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wbrk2\" (UniqueName: \"kubernetes.io/projected/0f35e3a2-9f78-4e4d-b60b-508668b4539a-kube-api-access-wbrk2\") pod \"0f35e3a2-9f78-4e4d-b60b-508668b4539a\" (UID: \"0f35e3a2-9f78-4e4d-b60b-508668b4539a\") "
Jul 27 18:30:39.175482 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:39.175387    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f35e3a2-9f78-4e4d-b60b-508668b4539a-proxy-tls\") pod \"0f35e3a2-9f78-4e4d-b60b-508668b4539a\" (UID: \"0f35e3a2-9f78-4e4d-b60b-508668b4539a\") "
Jul 27 18:30:39.175704 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:39.175623    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f35e3a2-9f78-4e4d-b60b-508668b4539a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0f35e3a2-9f78-4e4d-b60b-508668b4539a" (UID: "0f35e3a2-9f78-4e4d-b60b-508668b4539a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:30:39.175704 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:39.175652    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f35e3a2-9f78-4e4d-b60b-508668b4539a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0f35e3a2-9f78-4e4d-b60b-508668b4539a" (UID: "0f35e3a2-9f78-4e4d-b60b-508668b4539a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:30:39.175793 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:39.175704    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f35e3a2-9f78-4e4d-b60b-508668b4539a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0f35e3a2-9f78-4e4d-b60b-508668b4539a" (UID: "0f35e3a2-9f78-4e4d-b60b-508668b4539a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:30:39.177679 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:39.177637    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f35e3a2-9f78-4e4d-b60b-508668b4539a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0f35e3a2-9f78-4e4d-b60b-508668b4539a" (UID: "0f35e3a2-9f78-4e4d-b60b-508668b4539a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:30:39.177679 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:39.177646    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f35e3a2-9f78-4e4d-b60b-508668b4539a-kube-api-access-wbrk2" (OuterVolumeSpecName: "kube-api-access-wbrk2") pod "0f35e3a2-9f78-4e4d-b60b-508668b4539a" (UID: "0f35e3a2-9f78-4e4d-b60b-508668b4539a"). InnerVolumeSpecName "kube-api-access-wbrk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:30:39.276225 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:39.276162    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f35e3a2-9f78-4e4d-b60b-508668b4539a-proxy-tls\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:30:39.276225 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:39.276215    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f35e3a2-9f78-4e4d-b60b-508668b4539a-cabundle-cert\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:30:39.276225 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:39.276230    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f35e3a2-9f78-4e4d-b60b-508668b4539a-kserve-provision-location\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:30:39.276225 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:39.276244    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f35e3a2-9f78-4e4d-b60b-508668b4539a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:30:39.276544 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:39.276279    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbrk2\" (UniqueName: \"kubernetes.io/projected/0f35e3a2-9f78-4e4d-b60b-508668b4539a-kube-api-access-wbrk2\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:30:39.477870 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:39.477814    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/996961a9-cceb-4096-ba61-157534de65fa-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j\" (UID: \"996961a9-cceb-4096-ba61-157534de65fa\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j"
Jul 27 18:30:39.480310 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:39.480286    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/996961a9-cceb-4096-ba61-157534de65fa-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j\" (UID: \"996961a9-cceb-4096-ba61-157534de65fa\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j"
Jul 27 18:30:39.687507 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:39.687474    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck"
Jul 27 18:30:39.687507 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:39.687489    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck" event={"ID":"0f35e3a2-9f78-4e4d-b60b-508668b4539a","Type":"ContainerDied","Data":"546a6da7f6a3b1e68ab0ff9ee686e28cd48dd2de84679f58415b0fde9fd9f07b"}
Jul 27 18:30:39.687993 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:39.687559    2573 scope.go:117] "RemoveContainer" containerID="90c2458f944c19c0d70a630ccbe09eaca82bdb54b1337389687ffb67efb28ba0"
Jul 27 18:30:39.696662 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:39.696637    2573 scope.go:117] "RemoveContainer" containerID="d98615041536b9c6f9b78e67573ba35a69f0163cdc05cd4b3c16bb8b8fbf947c"
Jul 27 18:30:39.704366 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:39.704346    2573 scope.go:117] "RemoveContainer" containerID="bf7388ade1918881ccb4ded7a2e204a635dc995b104d18e3ac1a093754870ea3"
Jul 27 18:30:39.713300 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:39.713274    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck"]
Jul 27 18:30:39.717730 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:39.717708    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hrnck"]
Jul 27 18:30:39.761448 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:39.761355    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j"
Jul 27 18:30:39.904197 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:39.904166    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j"]
Jul 27 18:30:39.907407 ip-10-0-140-0 kubenswrapper[2573]: W0727 18:30:39.907380    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod996961a9_cceb_4096_ba61_157534de65fa.slice/crio-4e61603b2cacb09b5a86361f3245b1d31179a6785cac270113149d73dc78db35 WatchSource:0}: Error finding container 4e61603b2cacb09b5a86361f3245b1d31179a6785cac270113149d73dc78db35: Status 404 returned error can't find the container with id 4e61603b2cacb09b5a86361f3245b1d31179a6785cac270113149d73dc78db35
Jul 27 18:30:40.469883 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:40.469840    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f35e3a2-9f78-4e4d-b60b-508668b4539a" path="/var/lib/kubelet/pods/0f35e3a2-9f78-4e4d-b60b-508668b4539a/volumes"
Jul 27 18:30:40.694485 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:40.694452    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j" event={"ID":"996961a9-cceb-4096-ba61-157534de65fa","Type":"ContainerStarted","Data":"287b613ea6dff2a403b968f56e5623f5d28afd47f5db3281e1cf340c989e813e"}
Jul 27 18:30:40.694881 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:40.694508    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j" event={"ID":"996961a9-cceb-4096-ba61-157534de65fa","Type":"ContainerStarted","Data":"4e61603b2cacb09b5a86361f3245b1d31179a6785cac270113149d73dc78db35"}
Jul 27 18:30:44.713897 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:44.713861    2573 generic.go:356] "Generic (PLEG): container finished" podID="996961a9-cceb-4096-ba61-157534de65fa" containerID="287b613ea6dff2a403b968f56e5623f5d28afd47f5db3281e1cf340c989e813e" exitCode=0
Jul 27 18:30:44.714295 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:44.713901    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j" event={"ID":"996961a9-cceb-4096-ba61-157534de65fa","Type":"ContainerDied","Data":"287b613ea6dff2a403b968f56e5623f5d28afd47f5db3281e1cf340c989e813e"}
Jul 27 18:30:45.719959 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:45.719921    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j" event={"ID":"996961a9-cceb-4096-ba61-157534de65fa","Type":"ContainerStarted","Data":"9dfa52e50e4bc9285f300cbf082b2ffc19c2bcbaa2cb5f2157740f14c773aea1"}
Jul 27 18:30:45.720464 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:45.720029    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j" event={"ID":"996961a9-cceb-4096-ba61-157534de65fa","Type":"ContainerStarted","Data":"32d6c6bc7bca29392aab168f530a0eec367bca4c3ffb9f4ff554720e448235a1"}
Jul 27 18:30:45.720464 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:45.720262    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j"
Jul 27 18:30:45.762514 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:45.762456    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j" podStartSLOduration=3.997048728 podStartE2EDuration="7.762440008s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.76539128s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:30:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:30:45.760377745 +0000 UTC m=+3787.903779811" watchObservedRunningTime="2026-07-27 18:30:45.762440008 +0000 UTC m=+3787.905842059"
Jul 27 18:30:46.724941 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:46.724914    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j"
Jul 27 18:30:52.735206 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:30:52.735175    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j"
Jul 27 18:31:22.736437 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:31:22.736397    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j" podUID="996961a9-cceb-4096-ba61-157534de65fa" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.63:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.63:8080: connect: connection refused"
Jul 27 18:31:32.736039 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:31:32.736000    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j" podUID="996961a9-cceb-4096-ba61-157534de65fa" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.63:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.63:8080: connect: connection refused"
Jul 27 18:31:42.736376 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:31:42.736336    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j" podUID="996961a9-cceb-4096-ba61-157534de65fa" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.63:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.63:8080: connect: connection refused"
Jul 27 18:31:52.736502 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:31:52.736460    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j" podUID="996961a9-cceb-4096-ba61-157534de65fa" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.63:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.63:8080: connect: connection refused"
Jul 27 18:31:55.991279 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:31:55.991206    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:32:02.793135 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:02.793033    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j"
Jul 27 18:32:09.777568 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:09.777534    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j"]
Jul 27 18:32:09.777988 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:09.777765    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j" podUID="996961a9-cceb-4096-ba61-157534de65fa" containerName="kserve-container" containerID="cri-o://32d6c6bc7bca29392aab168f530a0eec367bca4c3ffb9f4ff554720e448235a1" gracePeriod=30
Jul 27 18:32:09.777988 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:09.777837    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j" podUID="996961a9-cceb-4096-ba61-157534de65fa" containerName="kube-rbac-proxy" containerID="cri-o://9dfa52e50e4bc9285f300cbf082b2ffc19c2bcbaa2cb5f2157740f14c773aea1" gracePeriod=30
Jul 27 18:32:09.815955 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:09.815921    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:32:10.092487 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:10.092394    2573 generic.go:356] "Generic (PLEG): container finished" podID="996961a9-cceb-4096-ba61-157534de65fa" containerID="9dfa52e50e4bc9285f300cbf082b2ffc19c2bcbaa2cb5f2157740f14c773aea1" exitCode=2
Jul 27 18:32:10.092487 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:10.092452    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j" event={"ID":"996961a9-cceb-4096-ba61-157534de65fa","Type":"ContainerDied","Data":"9dfa52e50e4bc9285f300cbf082b2ffc19c2bcbaa2cb5f2157740f14c773aea1"}
Jul 27 18:32:12.730031 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:12.729991    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j" podUID="996961a9-cceb-4096-ba61-157534de65fa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.63:8643/healthz\": dial tcp 10.133.0.63:8643: connect: connection refused"
Jul 27 18:32:12.736385 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:12.736349    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j" podUID="996961a9-cceb-4096-ba61-157534de65fa" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.63:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.63:8080: connect: connection refused"
Jul 27 18:32:14.729132 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:14.729104    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j"
Jul 27 18:32:14.858593 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:14.858507    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/996961a9-cceb-4096-ba61-157534de65fa-kserve-provision-location\") pod \"996961a9-cceb-4096-ba61-157534de65fa\" (UID: \"996961a9-cceb-4096-ba61-157534de65fa\") "
Jul 27 18:32:14.858593 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:14.858583    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/996961a9-cceb-4096-ba61-157534de65fa-proxy-tls\") pod \"996961a9-cceb-4096-ba61-157534de65fa\" (UID: \"996961a9-cceb-4096-ba61-157534de65fa\") "
Jul 27 18:32:14.858839 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:14.858604    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/996961a9-cceb-4096-ba61-157534de65fa-kube-rbac-proxy-sar-config\") pod \"996961a9-cceb-4096-ba61-157534de65fa\" (UID: \"996961a9-cceb-4096-ba61-157534de65fa\") "
Jul 27 18:32:14.858839 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:14.858627    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/996961a9-cceb-4096-ba61-157534de65fa-cabundle-cert\") pod \"996961a9-cceb-4096-ba61-157534de65fa\" (UID: \"996961a9-cceb-4096-ba61-157534de65fa\") "
Jul 27 18:32:14.858839 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:14.858662    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2pd4s\" (UniqueName: \"kubernetes.io/projected/996961a9-cceb-4096-ba61-157534de65fa-kube-api-access-2pd4s\") pod \"996961a9-cceb-4096-ba61-157534de65fa\" (UID: \"996961a9-cceb-4096-ba61-157534de65fa\") "
Jul 27 18:32:14.858997 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:14.858934    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/996961a9-cceb-4096-ba61-157534de65fa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "996961a9-cceb-4096-ba61-157534de65fa" (UID: "996961a9-cceb-4096-ba61-157534de65fa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:32:14.859071 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:14.859043    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/996961a9-cceb-4096-ba61-157534de65fa-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "996961a9-cceb-4096-ba61-157534de65fa" (UID: "996961a9-cceb-4096-ba61-157534de65fa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:32:14.859126 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:14.859052    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/996961a9-cceb-4096-ba61-157534de65fa-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "996961a9-cceb-4096-ba61-157534de65fa" (UID: "996961a9-cceb-4096-ba61-157534de65fa"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:32:14.860838 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:14.860773    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/996961a9-cceb-4096-ba61-157534de65fa-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "996961a9-cceb-4096-ba61-157534de65fa" (UID: "996961a9-cceb-4096-ba61-157534de65fa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:32:14.860975 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:14.860860    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/996961a9-cceb-4096-ba61-157534de65fa-kube-api-access-2pd4s" (OuterVolumeSpecName: "kube-api-access-2pd4s") pod "996961a9-cceb-4096-ba61-157534de65fa" (UID: "996961a9-cceb-4096-ba61-157534de65fa"). InnerVolumeSpecName "kube-api-access-2pd4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:32:14.879215 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:14.879187    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:32:14.940084 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:14.940045    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4"]
Jul 27 18:32:14.940468 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:14.940450    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="996961a9-cceb-4096-ba61-157534de65fa" containerName="storage-initializer"
Jul 27 18:32:14.940568 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:14.940470    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="996961a9-cceb-4096-ba61-157534de65fa" containerName="storage-initializer"
Jul 27 18:32:14.940568 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:14.940484    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f35e3a2-9f78-4e4d-b60b-508668b4539a" containerName="kube-rbac-proxy"
Jul 27 18:32:14.940568 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:14.940492    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f35e3a2-9f78-4e4d-b60b-508668b4539a" containerName="kube-rbac-proxy"
Jul 27 18:32:14.940568 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:14.940509    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f35e3a2-9f78-4e4d-b60b-508668b4539a" containerName="kserve-container"
Jul 27 18:32:14.940568 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:14.940519    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f35e3a2-9f78-4e4d-b60b-508668b4539a" containerName="kserve-container"
Jul 27 18:32:14.940568 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:14.940529    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="996961a9-cceb-4096-ba61-157534de65fa" containerName="kserve-container"
Jul 27 18:32:14.940568 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:14.940537    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="996961a9-cceb-4096-ba61-157534de65fa" containerName="kserve-container"
Jul 27 18:32:14.940568 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:14.940561    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="996961a9-cceb-4096-ba61-157534de65fa" containerName="kube-rbac-proxy"
Jul 27 18:32:14.940568 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:14.940570    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="996961a9-cceb-4096-ba61-157534de65fa" containerName="kube-rbac-proxy"
Jul 27 18:32:14.941016 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:14.940587    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f35e3a2-9f78-4e4d-b60b-508668b4539a" containerName="storage-initializer"
Jul 27 18:32:14.941016 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:14.940597    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f35e3a2-9f78-4e4d-b60b-508668b4539a" containerName="storage-initializer"
Jul 27 18:32:14.941016 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:14.940676    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f35e3a2-9f78-4e4d-b60b-508668b4539a" containerName="kube-rbac-proxy"
Jul 27 18:32:14.941016 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:14.940689    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f35e3a2-9f78-4e4d-b60b-508668b4539a" containerName="kserve-container"
Jul 27 18:32:14.941016 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:14.940700    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="996961a9-cceb-4096-ba61-157534de65fa" containerName="kube-rbac-proxy"
Jul 27 18:32:14.941016 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:14.940711    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="996961a9-cceb-4096-ba61-157534de65fa" containerName="kserve-container"
Jul 27 18:32:14.944163 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:14.944141    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4"
Jul 27 18:32:14.946920 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:14.946896    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\""
Jul 27 18:32:14.946920 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:14.946902    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\""
Jul 27 18:32:14.953022 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:14.953000    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4"]
Jul 27 18:32:14.959862 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:14.959828    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/996961a9-cceb-4096-ba61-157534de65fa-proxy-tls\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:32:14.959862 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:14.959862    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/996961a9-cceb-4096-ba61-157534de65fa-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:32:14.960017 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:14.959877    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/996961a9-cceb-4096-ba61-157534de65fa-cabundle-cert\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:32:14.960017 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:14.959892    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pd4s\" (UniqueName: \"kubernetes.io/projected/996961a9-cceb-4096-ba61-157534de65fa-kube-api-access-2pd4s\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:32:14.960017 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:14.959906    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/996961a9-cceb-4096-ba61-157534de65fa-kserve-provision-location\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:32:15.060783 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:15.060739    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/61e53394-dc2c-4b0c-9b14-d458c3272311-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4\" (UID: \"61e53394-dc2c-4b0c-9b14-d458c3272311\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4"
Jul 27 18:32:15.060965 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:15.060797    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61e53394-dc2c-4b0c-9b14-d458c3272311-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4\" (UID: \"61e53394-dc2c-4b0c-9b14-d458c3272311\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4"
Jul 27 18:32:15.060965 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:15.060847    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/61e53394-dc2c-4b0c-9b14-d458c3272311-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4\" (UID: \"61e53394-dc2c-4b0c-9b14-d458c3272311\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4"
Jul 27 18:32:15.060965 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:15.060874    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61e53394-dc2c-4b0c-9b14-d458c3272311-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4\" (UID: \"61e53394-dc2c-4b0c-9b14-d458c3272311\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4"
Jul 27 18:32:15.061079 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:15.060962    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5dw4\" (UniqueName: \"kubernetes.io/projected/61e53394-dc2c-4b0c-9b14-d458c3272311-kube-api-access-l5dw4\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4\" (UID: \"61e53394-dc2c-4b0c-9b14-d458c3272311\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4"
Jul 27 18:32:15.117048 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:15.116949    2573 generic.go:356] "Generic (PLEG): container finished" podID="996961a9-cceb-4096-ba61-157534de65fa" containerID="32d6c6bc7bca29392aab168f530a0eec367bca4c3ffb9f4ff554720e448235a1" exitCode=0
Jul 27 18:32:15.117048 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:15.117032    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j"
Jul 27 18:32:15.117048 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:15.117040    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j" event={"ID":"996961a9-cceb-4096-ba61-157534de65fa","Type":"ContainerDied","Data":"32d6c6bc7bca29392aab168f530a0eec367bca4c3ffb9f4ff554720e448235a1"}
Jul 27 18:32:15.117340 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:15.117100    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j" event={"ID":"996961a9-cceb-4096-ba61-157534de65fa","Type":"ContainerDied","Data":"4e61603b2cacb09b5a86361f3245b1d31179a6785cac270113149d73dc78db35"}
Jul 27 18:32:15.117340 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:15.117129    2573 scope.go:117] "RemoveContainer" containerID="9dfa52e50e4bc9285f300cbf082b2ffc19c2bcbaa2cb5f2157740f14c773aea1"
Jul 27 18:32:15.127844 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:15.127820    2573 scope.go:117] "RemoveContainer" containerID="32d6c6bc7bca29392aab168f530a0eec367bca4c3ffb9f4ff554720e448235a1"
Jul 27 18:32:15.135924 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:15.135905    2573 scope.go:117] "RemoveContainer" containerID="287b613ea6dff2a403b968f56e5623f5d28afd47f5db3281e1cf340c989e813e"
Jul 27 18:32:15.142374 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:15.142351    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j"]
Jul 27 18:32:15.144463 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:15.144441    2573 scope.go:117] "RemoveContainer" containerID="9dfa52e50e4bc9285f300cbf082b2ffc19c2bcbaa2cb5f2157740f14c773aea1"
Jul 27 18:32:15.144725 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:32:15.144706    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dfa52e50e4bc9285f300cbf082b2ffc19c2bcbaa2cb5f2157740f14c773aea1\": container with ID starting with 9dfa52e50e4bc9285f300cbf082b2ffc19c2bcbaa2cb5f2157740f14c773aea1 not found: ID does not exist" containerID="9dfa52e50e4bc9285f300cbf082b2ffc19c2bcbaa2cb5f2157740f14c773aea1"
Jul 27 18:32:15.144779 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:15.144735    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dfa52e50e4bc9285f300cbf082b2ffc19c2bcbaa2cb5f2157740f14c773aea1"} err="failed to get container status \"9dfa52e50e4bc9285f300cbf082b2ffc19c2bcbaa2cb5f2157740f14c773aea1\": rpc error: code = NotFound desc = could not find container \"9dfa52e50e4bc9285f300cbf082b2ffc19c2bcbaa2cb5f2157740f14c773aea1\": container with ID starting with 9dfa52e50e4bc9285f300cbf082b2ffc19c2bcbaa2cb5f2157740f14c773aea1 not found: ID does not exist"
Jul 27 18:32:15.144779 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:15.144755    2573 scope.go:117] "RemoveContainer" containerID="32d6c6bc7bca29392aab168f530a0eec367bca4c3ffb9f4ff554720e448235a1"
Jul 27 18:32:15.145017 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:32:15.145002    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32d6c6bc7bca29392aab168f530a0eec367bca4c3ffb9f4ff554720e448235a1\": container with ID starting with 32d6c6bc7bca29392aab168f530a0eec367bca4c3ffb9f4ff554720e448235a1 not found: ID does not exist" containerID="32d6c6bc7bca29392aab168f530a0eec367bca4c3ffb9f4ff554720e448235a1"
Jul 27 18:32:15.145066 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:15.145022    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32d6c6bc7bca29392aab168f530a0eec367bca4c3ffb9f4ff554720e448235a1"} err="failed to get container status \"32d6c6bc7bca29392aab168f530a0eec367bca4c3ffb9f4ff554720e448235a1\": rpc error: code = NotFound desc = could not find container \"32d6c6bc7bca29392aab168f530a0eec367bca4c3ffb9f4ff554720e448235a1\": container with ID starting with 32d6c6bc7bca29392aab168f530a0eec367bca4c3ffb9f4ff554720e448235a1 not found: ID does not exist"
Jul 27 18:32:15.145066 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:15.145036    2573 scope.go:117] "RemoveContainer" containerID="287b613ea6dff2a403b968f56e5623f5d28afd47f5db3281e1cf340c989e813e"
Jul 27 18:32:15.145309 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:32:15.145286    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"287b613ea6dff2a403b968f56e5623f5d28afd47f5db3281e1cf340c989e813e\": container with ID starting with 287b613ea6dff2a403b968f56e5623f5d28afd47f5db3281e1cf340c989e813e not found: ID does not exist" containerID="287b613ea6dff2a403b968f56e5623f5d28afd47f5db3281e1cf340c989e813e"
Jul 27 18:32:15.145426 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:15.145316    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"287b613ea6dff2a403b968f56e5623f5d28afd47f5db3281e1cf340c989e813e"} err="failed to get container status \"287b613ea6dff2a403b968f56e5623f5d28afd47f5db3281e1cf340c989e813e\": rpc error: code = NotFound desc = could not find container \"287b613ea6dff2a403b968f56e5623f5d28afd47f5db3281e1cf340c989e813e\": container with ID starting with 287b613ea6dff2a403b968f56e5623f5d28afd47f5db3281e1cf340c989e813e not found: ID does not exist"
Jul 27 18:32:15.145892 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:15.145874    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-kqj7j"]
Jul 27 18:32:15.161777 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:15.161739    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/61e53394-dc2c-4b0c-9b14-d458c3272311-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4\" (UID: \"61e53394-dc2c-4b0c-9b14-d458c3272311\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4"
Jul 27 18:32:15.161962 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:15.161790    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61e53394-dc2c-4b0c-9b14-d458c3272311-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4\" (UID: \"61e53394-dc2c-4b0c-9b14-d458c3272311\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4"
Jul 27 18:32:15.161962 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:15.161823    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/61e53394-dc2c-4b0c-9b14-d458c3272311-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4\" (UID: \"61e53394-dc2c-4b0c-9b14-d458c3272311\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4"
Jul 27 18:32:15.161962 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:15.161849    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61e53394-dc2c-4b0c-9b14-d458c3272311-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4\" (UID: \"61e53394-dc2c-4b0c-9b14-d458c3272311\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4"
Jul 27 18:32:15.161962 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:15.161896    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5dw4\" (UniqueName: \"kubernetes.io/projected/61e53394-dc2c-4b0c-9b14-d458c3272311-kube-api-access-l5dw4\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4\" (UID: \"61e53394-dc2c-4b0c-9b14-d458c3272311\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4"
Jul 27 18:32:15.162157 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:32:15.161996    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-serving-cert: secret "isvc-predictive-xgboost-v2-predictor-serving-cert" not found
Jul 27 18:32:15.162157 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:32:15.162070    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61e53394-dc2c-4b0c-9b14-d458c3272311-proxy-tls podName:61e53394-dc2c-4b0c-9b14-d458c3272311 nodeName:}" failed. No retries permitted until 2026-07-27 18:32:15.662048338 +0000 UTC m=+3877.805450368 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/61e53394-dc2c-4b0c-9b14-d458c3272311-proxy-tls") pod "isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4" (UID: "61e53394-dc2c-4b0c-9b14-d458c3272311") : secret "isvc-predictive-xgboost-v2-predictor-serving-cert" not found
Jul 27 18:32:15.162331 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:15.162310    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61e53394-dc2c-4b0c-9b14-d458c3272311-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4\" (UID: \"61e53394-dc2c-4b0c-9b14-d458c3272311\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4"
Jul 27 18:32:15.162565 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:15.162548    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/61e53394-dc2c-4b0c-9b14-d458c3272311-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4\" (UID: \"61e53394-dc2c-4b0c-9b14-d458c3272311\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4"
Jul 27 18:32:15.162605 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:15.162547    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/61e53394-dc2c-4b0c-9b14-d458c3272311-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4\" (UID: \"61e53394-dc2c-4b0c-9b14-d458c3272311\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4"
Jul 27 18:32:15.170916 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:15.170894    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5dw4\" (UniqueName: \"kubernetes.io/projected/61e53394-dc2c-4b0c-9b14-d458c3272311-kube-api-access-l5dw4\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4\" (UID: \"61e53394-dc2c-4b0c-9b14-d458c3272311\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4"
Jul 27 18:32:15.279457 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:15.279423    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:32:15.665878 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:15.665838    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61e53394-dc2c-4b0c-9b14-d458c3272311-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4\" (UID: \"61e53394-dc2c-4b0c-9b14-d458c3272311\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4"
Jul 27 18:32:15.668228 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:15.668195    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61e53394-dc2c-4b0c-9b14-d458c3272311-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4\" (UID: \"61e53394-dc2c-4b0c-9b14-d458c3272311\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4"
Jul 27 18:32:15.855409 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:15.855373    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4"
Jul 27 18:32:15.992688 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:15.992662    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4"]
Jul 27 18:32:15.995392 ip-10-0-140-0 kubenswrapper[2573]: W0727 18:32:15.995359    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61e53394_dc2c_4b0c_9b14_d458c3272311.slice/crio-79a7b85c89b129b3da246d61596c20257751c5c864187588de9a8db045df3d33 WatchSource:0}: Error finding container 79a7b85c89b129b3da246d61596c20257751c5c864187588de9a8db045df3d33: Status 404 returned error can't find the container with id 79a7b85c89b129b3da246d61596c20257751c5c864187588de9a8db045df3d33
Jul 27 18:32:16.124461 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:16.124426    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4" event={"ID":"61e53394-dc2c-4b0c-9b14-d458c3272311","Type":"ContainerStarted","Data":"492d5a881ca78e17c92e9a0906deda27a4e11ae14265b473595c5289eab61a28"}
Jul 27 18:32:16.124675 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:16.124486    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4" event={"ID":"61e53394-dc2c-4b0c-9b14-d458c3272311","Type":"ContainerStarted","Data":"79a7b85c89b129b3da246d61596c20257751c5c864187588de9a8db045df3d33"}
Jul 27 18:32:16.470159 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:16.470120    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="996961a9-cceb-4096-ba61-157534de65fa" path="/var/lib/kubelet/pods/996961a9-cceb-4096-ba61-157534de65fa/volumes"
Jul 27 18:32:20.145578 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:20.145543    2573 generic.go:356] "Generic (PLEG): container finished" podID="61e53394-dc2c-4b0c-9b14-d458c3272311" containerID="492d5a881ca78e17c92e9a0906deda27a4e11ae14265b473595c5289eab61a28" exitCode=0
Jul 27 18:32:20.145969 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:20.145622    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4" event={"ID":"61e53394-dc2c-4b0c-9b14-d458c3272311","Type":"ContainerDied","Data":"492d5a881ca78e17c92e9a0906deda27a4e11ae14265b473595c5289eab61a28"}
Jul 27 18:32:21.152595 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:21.152559    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4" event={"ID":"61e53394-dc2c-4b0c-9b14-d458c3272311","Type":"ContainerStarted","Data":"abf673c87c91f3e1d57d0167a8defb99c192f6e44859a3e60de78d70c873e5b0"}
Jul 27 18:32:21.153049 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:21.152627    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4" event={"ID":"61e53394-dc2c-4b0c-9b14-d458c3272311","Type":"ContainerStarted","Data":"b8099f73ec32eb458af7faf1ca8f9d7a4b343c2f55696ae494921ae52d3c9952"}
Jul 27 18:32:21.153049 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:21.152822    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4"
Jul 27 18:32:21.174685 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:21.174632    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4" podStartSLOduration=3.4377046780000002 podStartE2EDuration="7.174615669s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.736910991s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:32:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:32:21.173874239 +0000 UTC m=+3883.317276292" watchObservedRunningTime="2026-07-27 18:32:21.174615669 +0000 UTC m=+3883.318017720"
Jul 27 18:32:22.156795 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:22.156767    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4"
Jul 27 18:32:28.166373 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:28.166342    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4"
Jul 27 18:32:38.569798 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:38.569768    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vc2tz_87dac797-f8d2-4d36-999f-f8ce855dbeff/ovn-acl-logging/0.log"
Jul 27 18:32:38.578681 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:38.578657    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vc2tz_87dac797-f8d2-4d36-999f-f8ce855dbeff/ovn-acl-logging/0.log"
Jul 27 18:32:56.845784 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:56.845726    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:32:58.167524 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:32:58.167487    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4" podUID="61e53394-dc2c-4b0c-9b14-d458c3272311" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.64:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.64:8080: connect: connection refused"
Jul 27 18:33:08.167283 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:08.167220    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4" podUID="61e53394-dc2c-4b0c-9b14-d458c3272311" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.64:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.64:8080: connect: connection refused"
Jul 27 18:33:18.167635 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:18.167595    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4" podUID="61e53394-dc2c-4b0c-9b14-d458c3272311" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.64:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.64:8080: connect: connection refused"
Jul 27 18:33:28.167023 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:28.166978    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4" podUID="61e53394-dc2c-4b0c-9b14-d458c3272311" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.64:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.64:8080: connect: connection refused"
Jul 27 18:33:38.191491 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:38.191393    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4"
Jul 27 18:33:45.140330 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:45.140297    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4"]
Jul 27 18:33:45.140794 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:45.140591    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4" podUID="61e53394-dc2c-4b0c-9b14-d458c3272311" containerName="kserve-container" containerID="cri-o://b8099f73ec32eb458af7faf1ca8f9d7a4b343c2f55696ae494921ae52d3c9952" gracePeriod=30
Jul 27 18:33:45.140794 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:45.140591    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4" podUID="61e53394-dc2c-4b0c-9b14-d458c3272311" containerName="kube-rbac-proxy" containerID="cri-o://abf673c87c91f3e1d57d0167a8defb99c192f6e44859a3e60de78d70c873e5b0" gracePeriod=30
Jul 27 18:33:45.168279 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:45.168229    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:33:45.524675 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:45.524579    2573 generic.go:356] "Generic (PLEG): container finished" podID="61e53394-dc2c-4b0c-9b14-d458c3272311" containerID="abf673c87c91f3e1d57d0167a8defb99c192f6e44859a3e60de78d70c873e5b0" exitCode=2
Jul 27 18:33:45.524675 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:45.524651    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4" event={"ID":"61e53394-dc2c-4b0c-9b14-d458c3272311","Type":"ContainerDied","Data":"abf673c87c91f3e1d57d0167a8defb99c192f6e44859a3e60de78d70c873e5b0"}
Jul 27 18:33:48.161294 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:48.161230    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4" podUID="61e53394-dc2c-4b0c-9b14-d458c3272311" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.64:8643/healthz\": dial tcp 10.133.0.64:8643: connect: connection refused"
Jul 27 18:33:48.166982 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:48.166948    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4" podUID="61e53394-dc2c-4b0c-9b14-d458c3272311" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.64:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.64:8080: connect: connection refused"
Jul 27 18:33:49.993079 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:49.993052    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4"
Jul 27 18:33:50.052921 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.052830    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61e53394-dc2c-4b0c-9b14-d458c3272311-kserve-provision-location\") pod \"61e53394-dc2c-4b0c-9b14-d458c3272311\" (UID: \"61e53394-dc2c-4b0c-9b14-d458c3272311\") "
Jul 27 18:33:50.053090 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.052930    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/61e53394-dc2c-4b0c-9b14-d458c3272311-cabundle-cert\") pod \"61e53394-dc2c-4b0c-9b14-d458c3272311\" (UID: \"61e53394-dc2c-4b0c-9b14-d458c3272311\") "
Jul 27 18:33:50.053090 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.052965    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61e53394-dc2c-4b0c-9b14-d458c3272311-proxy-tls\") pod \"61e53394-dc2c-4b0c-9b14-d458c3272311\" (UID: \"61e53394-dc2c-4b0c-9b14-d458c3272311\") "
Jul 27 18:33:50.053090 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.053005    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l5dw4\" (UniqueName: \"kubernetes.io/projected/61e53394-dc2c-4b0c-9b14-d458c3272311-kube-api-access-l5dw4\") pod \"61e53394-dc2c-4b0c-9b14-d458c3272311\" (UID: \"61e53394-dc2c-4b0c-9b14-d458c3272311\") "
Jul 27 18:33:50.053300 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.053140    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/61e53394-dc2c-4b0c-9b14-d458c3272311-kube-rbac-proxy-sar-config\") pod \"61e53394-dc2c-4b0c-9b14-d458c3272311\" (UID: \"61e53394-dc2c-4b0c-9b14-d458c3272311\") "
Jul 27 18:33:50.053300 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.053209    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61e53394-dc2c-4b0c-9b14-d458c3272311-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "61e53394-dc2c-4b0c-9b14-d458c3272311" (UID: "61e53394-dc2c-4b0c-9b14-d458c3272311"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:33:50.053419 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.053395    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/61e53394-dc2c-4b0c-9b14-d458c3272311-kserve-provision-location\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:33:50.053477 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.053358    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61e53394-dc2c-4b0c-9b14-d458c3272311-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "61e53394-dc2c-4b0c-9b14-d458c3272311" (UID: "61e53394-dc2c-4b0c-9b14-d458c3272311"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:33:50.053526 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.053506    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61e53394-dc2c-4b0c-9b14-d458c3272311-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "61e53394-dc2c-4b0c-9b14-d458c3272311" (UID: "61e53394-dc2c-4b0c-9b14-d458c3272311"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:33:50.055140 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.055118    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61e53394-dc2c-4b0c-9b14-d458c3272311-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "61e53394-dc2c-4b0c-9b14-d458c3272311" (UID: "61e53394-dc2c-4b0c-9b14-d458c3272311"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:33:50.055223 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.055159    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61e53394-dc2c-4b0c-9b14-d458c3272311-kube-api-access-l5dw4" (OuterVolumeSpecName: "kube-api-access-l5dw4") pod "61e53394-dc2c-4b0c-9b14-d458c3272311" (UID: "61e53394-dc2c-4b0c-9b14-d458c3272311"). InnerVolumeSpecName "kube-api-access-l5dw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:33:50.153888 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.153855    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5dw4\" (UniqueName: \"kubernetes.io/projected/61e53394-dc2c-4b0c-9b14-d458c3272311-kube-api-access-l5dw4\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:33:50.153888 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.153884    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/61e53394-dc2c-4b0c-9b14-d458c3272311-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:33:50.153888 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.153893    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/61e53394-dc2c-4b0c-9b14-d458c3272311-cabundle-cert\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:33:50.154130 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.153902    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/61e53394-dc2c-4b0c-9b14-d458c3272311-proxy-tls\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:33:50.201227 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.201192    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:33:50.260147 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.257069    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-bf492"]
Jul 27 18:33:50.260147 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.257794    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61e53394-dc2c-4b0c-9b14-d458c3272311" containerName="kube-rbac-proxy"
Jul 27 18:33:50.260147 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.257815    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="61e53394-dc2c-4b0c-9b14-d458c3272311" containerName="kube-rbac-proxy"
Jul 27 18:33:50.260147 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.257835    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61e53394-dc2c-4b0c-9b14-d458c3272311" containerName="kserve-container"
Jul 27 18:33:50.260147 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.257843    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="61e53394-dc2c-4b0c-9b14-d458c3272311" containerName="kserve-container"
Jul 27 18:33:50.260147 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.257878    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61e53394-dc2c-4b0c-9b14-d458c3272311" containerName="storage-initializer"
Jul 27 18:33:50.260147 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.257893    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="61e53394-dc2c-4b0c-9b14-d458c3272311" containerName="storage-initializer"
Jul 27 18:33:50.260147 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.258027    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="61e53394-dc2c-4b0c-9b14-d458c3272311" containerName="kube-rbac-proxy"
Jul 27 18:33:50.260147 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.258044    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="61e53394-dc2c-4b0c-9b14-d458c3272311" containerName="kserve-container"
Jul 27 18:33:50.262604 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.262574    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-bf492"
Jul 27 18:33:50.266028 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.265686    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
Jul 27 18:33:50.266028 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.265691    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
Jul 27 18:33:50.268125 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.268098    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-bf492"]
Jul 27 18:33:50.356504 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.356407    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djqn8\" (UniqueName: \"kubernetes.io/projected/ec78b987-9ea1-46da-bb08-c95fdceadc6d-kube-api-access-djqn8\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-bf492\" (UID: \"ec78b987-9ea1-46da-bb08-c95fdceadc6d\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-bf492"
Jul 27 18:33:50.356504 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.356454    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec78b987-9ea1-46da-bb08-c95fdceadc6d-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-bf492\" (UID: \"ec78b987-9ea1-46da-bb08-c95fdceadc6d\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-bf492"
Jul 27 18:33:50.356504 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.356481    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec78b987-9ea1-46da-bb08-c95fdceadc6d-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-bf492\" (UID: \"ec78b987-9ea1-46da-bb08-c95fdceadc6d\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-bf492"
Jul 27 18:33:50.356738 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.356569    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec78b987-9ea1-46da-bb08-c95fdceadc6d-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-bf492\" (UID: \"ec78b987-9ea1-46da-bb08-c95fdceadc6d\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-bf492"
Jul 27 18:33:50.356738 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.356615    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec78b987-9ea1-46da-bb08-c95fdceadc6d-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-bf492\" (UID: \"ec78b987-9ea1-46da-bb08-c95fdceadc6d\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-bf492"
Jul 27 18:33:50.457881 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.457849    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec78b987-9ea1-46da-bb08-c95fdceadc6d-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-bf492\" (UID: \"ec78b987-9ea1-46da-bb08-c95fdceadc6d\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-bf492"
Jul 27 18:33:50.458074 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.457894    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec78b987-9ea1-46da-bb08-c95fdceadc6d-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-bf492\" (UID: \"ec78b987-9ea1-46da-bb08-c95fdceadc6d\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-bf492"
Jul 27 18:33:50.458074 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.457930    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec78b987-9ea1-46da-bb08-c95fdceadc6d-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-bf492\" (UID: \"ec78b987-9ea1-46da-bb08-c95fdceadc6d\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-bf492"
Jul 27 18:33:50.458074 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.457980    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djqn8\" (UniqueName: \"kubernetes.io/projected/ec78b987-9ea1-46da-bb08-c95fdceadc6d-kube-api-access-djqn8\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-bf492\" (UID: \"ec78b987-9ea1-46da-bb08-c95fdceadc6d\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-bf492"
Jul 27 18:33:50.458074 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.458024    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec78b987-9ea1-46da-bb08-c95fdceadc6d-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-bf492\" (UID: \"ec78b987-9ea1-46da-bb08-c95fdceadc6d\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-bf492"
Jul 27 18:33:50.458455 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.458435    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec78b987-9ea1-46da-bb08-c95fdceadc6d-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-bf492\" (UID: \"ec78b987-9ea1-46da-bb08-c95fdceadc6d\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-bf492"
Jul 27 18:33:50.458756 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.458735    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec78b987-9ea1-46da-bb08-c95fdceadc6d-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-bf492\" (UID: \"ec78b987-9ea1-46da-bb08-c95fdceadc6d\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-bf492"
Jul 27 18:33:50.458803 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.458735    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec78b987-9ea1-46da-bb08-c95fdceadc6d-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-bf492\" (UID: \"ec78b987-9ea1-46da-bb08-c95fdceadc6d\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-bf492"
Jul 27 18:33:50.460731 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.460598    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec78b987-9ea1-46da-bb08-c95fdceadc6d-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-bf492\" (UID: \"ec78b987-9ea1-46da-bb08-c95fdceadc6d\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-bf492"
Jul 27 18:33:50.467587 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.467564    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djqn8\" (UniqueName: \"kubernetes.io/projected/ec78b987-9ea1-46da-bb08-c95fdceadc6d-kube-api-access-djqn8\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-bf492\" (UID: \"ec78b987-9ea1-46da-bb08-c95fdceadc6d\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-bf492"
Jul 27 18:33:50.507485 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.507457    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:33:50.547735 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.547703    2573 generic.go:356] "Generic (PLEG): container finished" podID="61e53394-dc2c-4b0c-9b14-d458c3272311" containerID="b8099f73ec32eb458af7faf1ca8f9d7a4b343c2f55696ae494921ae52d3c9952" exitCode=0
Jul 27 18:33:50.547922 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.547779    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4"
Jul 27 18:33:50.547922 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.547787    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4" event={"ID":"61e53394-dc2c-4b0c-9b14-d458c3272311","Type":"ContainerDied","Data":"b8099f73ec32eb458af7faf1ca8f9d7a4b343c2f55696ae494921ae52d3c9952"}
Jul 27 18:33:50.547922 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.547858    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4" event={"ID":"61e53394-dc2c-4b0c-9b14-d458c3272311","Type":"ContainerDied","Data":"79a7b85c89b129b3da246d61596c20257751c5c864187588de9a8db045df3d33"}
Jul 27 18:33:50.547922 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.547898    2573 scope.go:117] "RemoveContainer" containerID="abf673c87c91f3e1d57d0167a8defb99c192f6e44859a3e60de78d70c873e5b0"
Jul 27 18:33:50.556131 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.556115    2573 scope.go:117] "RemoveContainer" containerID="b8099f73ec32eb458af7faf1ca8f9d7a4b343c2f55696ae494921ae52d3c9952"
Jul 27 18:33:50.563579 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.563563    2573 scope.go:117] "RemoveContainer" containerID="492d5a881ca78e17c92e9a0906deda27a4e11ae14265b473595c5289eab61a28"
Jul 27 18:33:50.567714 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.567691    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4"]
Jul 27 18:33:50.570597 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.570570    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-xvhn4"]
Jul 27 18:33:50.571529 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.571513    2573 scope.go:117] "RemoveContainer" containerID="abf673c87c91f3e1d57d0167a8defb99c192f6e44859a3e60de78d70c873e5b0"
Jul 27 18:33:50.571812 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:33:50.571794    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abf673c87c91f3e1d57d0167a8defb99c192f6e44859a3e60de78d70c873e5b0\": container with ID starting with abf673c87c91f3e1d57d0167a8defb99c192f6e44859a3e60de78d70c873e5b0 not found: ID does not exist" containerID="abf673c87c91f3e1d57d0167a8defb99c192f6e44859a3e60de78d70c873e5b0"
Jul 27 18:33:50.571857 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.571823    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abf673c87c91f3e1d57d0167a8defb99c192f6e44859a3e60de78d70c873e5b0"} err="failed to get container status \"abf673c87c91f3e1d57d0167a8defb99c192f6e44859a3e60de78d70c873e5b0\": rpc error: code = NotFound desc = could not find container \"abf673c87c91f3e1d57d0167a8defb99c192f6e44859a3e60de78d70c873e5b0\": container with ID starting with abf673c87c91f3e1d57d0167a8defb99c192f6e44859a3e60de78d70c873e5b0 not found: ID does not exist"
Jul 27 18:33:50.571857 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.571845    2573 scope.go:117] "RemoveContainer" containerID="b8099f73ec32eb458af7faf1ca8f9d7a4b343c2f55696ae494921ae52d3c9952"
Jul 27 18:33:50.572097 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:33:50.572077    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8099f73ec32eb458af7faf1ca8f9d7a4b343c2f55696ae494921ae52d3c9952\": container with ID starting with b8099f73ec32eb458af7faf1ca8f9d7a4b343c2f55696ae494921ae52d3c9952 not found: ID does not exist" containerID="b8099f73ec32eb458af7faf1ca8f9d7a4b343c2f55696ae494921ae52d3c9952"
Jul 27 18:33:50.572156 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.572104    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8099f73ec32eb458af7faf1ca8f9d7a4b343c2f55696ae494921ae52d3c9952"} err="failed to get container status \"b8099f73ec32eb458af7faf1ca8f9d7a4b343c2f55696ae494921ae52d3c9952\": rpc error: code = NotFound desc = could not find container \"b8099f73ec32eb458af7faf1ca8f9d7a4b343c2f55696ae494921ae52d3c9952\": container with ID starting with b8099f73ec32eb458af7faf1ca8f9d7a4b343c2f55696ae494921ae52d3c9952 not found: ID does not exist"
Jul 27 18:33:50.572156 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.572120    2573 scope.go:117] "RemoveContainer" containerID="492d5a881ca78e17c92e9a0906deda27a4e11ae14265b473595c5289eab61a28"
Jul 27 18:33:50.572372 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:33:50.572354    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"492d5a881ca78e17c92e9a0906deda27a4e11ae14265b473595c5289eab61a28\": container with ID starting with 492d5a881ca78e17c92e9a0906deda27a4e11ae14265b473595c5289eab61a28 not found: ID does not exist" containerID="492d5a881ca78e17c92e9a0906deda27a4e11ae14265b473595c5289eab61a28"
Jul 27 18:33:50.572412 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.572378    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"492d5a881ca78e17c92e9a0906deda27a4e11ae14265b473595c5289eab61a28"} err="failed to get container status \"492d5a881ca78e17c92e9a0906deda27a4e11ae14265b473595c5289eab61a28\": rpc error: code = NotFound desc = could not find container \"492d5a881ca78e17c92e9a0906deda27a4e11ae14265b473595c5289eab61a28\": container with ID starting with 492d5a881ca78e17c92e9a0906deda27a4e11ae14265b473595c5289eab61a28 not found: ID does not exist"
Jul 27 18:33:50.573516 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.573498    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-bf492"
Jul 27 18:33:50.711903 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:50.711874    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-bf492"]
Jul 27 18:33:50.714663 ip-10-0-140-0 kubenswrapper[2573]: W0727 18:33:50.714629    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec78b987_9ea1_46da_bb08_c95fdceadc6d.slice/crio-c2747a2ca53cbf99d23c4232ab660997d2d1551bf39b533d4e388dc4046c93d7 WatchSource:0}: Error finding container c2747a2ca53cbf99d23c4232ab660997d2d1551bf39b533d4e388dc4046c93d7: Status 404 returned error can't find the container with id c2747a2ca53cbf99d23c4232ab660997d2d1551bf39b533d4e388dc4046c93d7
Jul 27 18:33:51.555482 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:51.555445    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-bf492" event={"ID":"ec78b987-9ea1-46da-bb08-c95fdceadc6d","Type":"ContainerStarted","Data":"b1ab930adf6301435bf1588da0e3b8ae3655b4c3fa6890d7c5bc73ba51ceb330"}
Jul 27 18:33:51.555995 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:51.555519    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-bf492" event={"ID":"ec78b987-9ea1-46da-bb08-c95fdceadc6d","Type":"ContainerStarted","Data":"c2747a2ca53cbf99d23c4232ab660997d2d1551bf39b533d4e388dc4046c93d7"}
Jul 27 18:33:52.469104 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:52.469069    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61e53394-dc2c-4b0c-9b14-d458c3272311" path="/var/lib/kubelet/pods/61e53394-dc2c-4b0c-9b14-d458c3272311/volumes"
Jul 27 18:33:54.569827 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:54.569796    2573 generic.go:356] "Generic (PLEG): container finished" podID="ec78b987-9ea1-46da-bb08-c95fdceadc6d" containerID="b1ab930adf6301435bf1588da0e3b8ae3655b4c3fa6890d7c5bc73ba51ceb330" exitCode=0
Jul 27 18:33:54.570243 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:54.569868    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-bf492" event={"ID":"ec78b987-9ea1-46da-bb08-c95fdceadc6d","Type":"ContainerDied","Data":"b1ab930adf6301435bf1588da0e3b8ae3655b4c3fa6890d7c5bc73ba51ceb330"}
Jul 27 18:33:55.575943 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:55.575909    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-bf492" event={"ID":"ec78b987-9ea1-46da-bb08-c95fdceadc6d","Type":"ContainerStarted","Data":"189c16730e2109ea178234b875201803231e49f89f34835a83e3248c24b20b35"}
Jul 27 18:33:55.576357 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:55.575991    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-bf492" event={"ID":"ec78b987-9ea1-46da-bb08-c95fdceadc6d","Type":"ContainerStarted","Data":"76b57755edf6be3d5deb3176310d3672f3abea4fbd463ca3f3d6adc2bfec222e"}
Jul 27 18:33:55.576357 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:55.576173    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-bf492"
Jul 27 18:33:55.596000 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:55.595950    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-bf492" podStartSLOduration=1.832923914 podStartE2EDuration="5.59593693s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.763013016s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:33:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:33:55.595159316 +0000 UTC m=+3977.738561370" watchObservedRunningTime="2026-07-27 18:33:55.59593693 +0000 UTC m=+3977.739338980"
Jul 27 18:33:56.580128 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:33:56.580098    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-bf492"
Jul 27 18:34:02.590920 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:34:02.590881    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-bf492"
Jul 27 18:34:32.592120 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:34:32.592080    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-bf492" podUID="ec78b987-9ea1-46da-bb08-c95fdceadc6d" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.65:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.65:8080: connect: connection refused"
Jul 27 18:34:42.591501 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:34:42.591460    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-bf492" podUID="ec78b987-9ea1-46da-bb08-c95fdceadc6d" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.65:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.65:8080: connect: connection refused"
Jul 27 18:34:52.592060 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:34:52.592014    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-bf492" podUID="ec78b987-9ea1-46da-bb08-c95fdceadc6d" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.65:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.65:8080: connect: connection refused"
Jul 27 18:35:02.592270 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:02.592154    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-bf492" podUID="ec78b987-9ea1-46da-bb08-c95fdceadc6d" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.65:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.65:8080: connect: connection refused"
Jul 27 18:35:07.464878 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:07.464832    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-bf492" podUID="ec78b987-9ea1-46da-bb08-c95fdceadc6d" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.65:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.65:8080: connect: connection refused"
Jul 27 18:35:17.492088 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:17.492053    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-bf492"
Jul 27 18:35:20.352561 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:20.352518    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-bf492"]
Jul 27 18:35:20.353046 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:20.352820    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-bf492" podUID="ec78b987-9ea1-46da-bb08-c95fdceadc6d" containerName="kserve-container" containerID="cri-o://76b57755edf6be3d5deb3176310d3672f3abea4fbd463ca3f3d6adc2bfec222e" gracePeriod=30
Jul 27 18:35:20.353046 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:20.352841    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-bf492" podUID="ec78b987-9ea1-46da-bb08-c95fdceadc6d" containerName="kube-rbac-proxy" containerID="cri-o://189c16730e2109ea178234b875201803231e49f89f34835a83e3248c24b20b35" gracePeriod=30
Jul 27 18:35:20.385916 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:20.385881    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:35:20.959728 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:20.959696    2573 generic.go:356] "Generic (PLEG): container finished" podID="ec78b987-9ea1-46da-bb08-c95fdceadc6d" containerID="189c16730e2109ea178234b875201803231e49f89f34835a83e3248c24b20b35" exitCode=2
Jul 27 18:35:20.959908 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:20.959767    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-bf492" event={"ID":"ec78b987-9ea1-46da-bb08-c95fdceadc6d","Type":"ContainerDied","Data":"189c16730e2109ea178234b875201803231e49f89f34835a83e3248c24b20b35"}
Jul 27 18:35:22.586078 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:22.586034    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-bf492" podUID="ec78b987-9ea1-46da-bb08-c95fdceadc6d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.65:8643/healthz\": dial tcp 10.133.0.65:8643: connect: connection refused"
Jul 27 18:35:25.557209 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:25.557177    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:35:25.607529 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:25.607507    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-bf492"
Jul 27 18:35:25.654222 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:25.654196    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec78b987-9ea1-46da-bb08-c95fdceadc6d-proxy-tls\") pod \"ec78b987-9ea1-46da-bb08-c95fdceadc6d\" (UID: \"ec78b987-9ea1-46da-bb08-c95fdceadc6d\") "
Jul 27 18:35:25.654394 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:25.654239    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djqn8\" (UniqueName: \"kubernetes.io/projected/ec78b987-9ea1-46da-bb08-c95fdceadc6d-kube-api-access-djqn8\") pod \"ec78b987-9ea1-46da-bb08-c95fdceadc6d\" (UID: \"ec78b987-9ea1-46da-bb08-c95fdceadc6d\") "
Jul 27 18:35:25.654394 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:25.654287    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec78b987-9ea1-46da-bb08-c95fdceadc6d-kube-rbac-proxy-sar-config\") pod \"ec78b987-9ea1-46da-bb08-c95fdceadc6d\" (UID: \"ec78b987-9ea1-46da-bb08-c95fdceadc6d\") "
Jul 27 18:35:25.654394 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:25.654313    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec78b987-9ea1-46da-bb08-c95fdceadc6d-cabundle-cert\") pod \"ec78b987-9ea1-46da-bb08-c95fdceadc6d\" (UID: \"ec78b987-9ea1-46da-bb08-c95fdceadc6d\") "
Jul 27 18:35:25.654394 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:25.654365    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec78b987-9ea1-46da-bb08-c95fdceadc6d-kserve-provision-location\") pod \"ec78b987-9ea1-46da-bb08-c95fdceadc6d\" (UID: \"ec78b987-9ea1-46da-bb08-c95fdceadc6d\") "
Jul 27 18:35:25.654730 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:25.654707    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec78b987-9ea1-46da-bb08-c95fdceadc6d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ec78b987-9ea1-46da-bb08-c95fdceadc6d" (UID: "ec78b987-9ea1-46da-bb08-c95fdceadc6d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:35:25.654795 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:25.654721    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec78b987-9ea1-46da-bb08-c95fdceadc6d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ec78b987-9ea1-46da-bb08-c95fdceadc6d" (UID: "ec78b987-9ea1-46da-bb08-c95fdceadc6d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:35:25.654795 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:25.654746    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec78b987-9ea1-46da-bb08-c95fdceadc6d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ec78b987-9ea1-46da-bb08-c95fdceadc6d" (UID: "ec78b987-9ea1-46da-bb08-c95fdceadc6d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:35:25.656412 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:25.656390    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec78b987-9ea1-46da-bb08-c95fdceadc6d-kube-api-access-djqn8" (OuterVolumeSpecName: "kube-api-access-djqn8") pod "ec78b987-9ea1-46da-bb08-c95fdceadc6d" (UID: "ec78b987-9ea1-46da-bb08-c95fdceadc6d"). InnerVolumeSpecName "kube-api-access-djqn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:35:25.656521 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:25.656498    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec78b987-9ea1-46da-bb08-c95fdceadc6d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ec78b987-9ea1-46da-bb08-c95fdceadc6d" (UID: "ec78b987-9ea1-46da-bb08-c95fdceadc6d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:35:25.675324 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:25.675290    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:35:25.755657 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:25.755575    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec78b987-9ea1-46da-bb08-c95fdceadc6d-proxy-tls\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:35:25.755657 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:25.755606    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djqn8\" (UniqueName: \"kubernetes.io/projected/ec78b987-9ea1-46da-bb08-c95fdceadc6d-kube-api-access-djqn8\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:35:25.755657 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:25.755617    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec78b987-9ea1-46da-bb08-c95fdceadc6d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:35:25.755657 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:25.755628    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec78b987-9ea1-46da-bb08-c95fdceadc6d-cabundle-cert\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:35:25.755657 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:25.755637    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec78b987-9ea1-46da-bb08-c95fdceadc6d-kserve-provision-location\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:35:25.981885 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:25.981848    2573 generic.go:356] "Generic (PLEG): container finished" podID="ec78b987-9ea1-46da-bb08-c95fdceadc6d" containerID="76b57755edf6be3d5deb3176310d3672f3abea4fbd463ca3f3d6adc2bfec222e" exitCode=0
Jul 27 18:35:25.982092 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:25.981914    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-bf492"
Jul 27 18:35:25.982092 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:25.981929    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-bf492" event={"ID":"ec78b987-9ea1-46da-bb08-c95fdceadc6d","Type":"ContainerDied","Data":"76b57755edf6be3d5deb3176310d3672f3abea4fbd463ca3f3d6adc2bfec222e"}
Jul 27 18:35:25.982092 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:25.982001    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-bf492" event={"ID":"ec78b987-9ea1-46da-bb08-c95fdceadc6d","Type":"ContainerDied","Data":"c2747a2ca53cbf99d23c4232ab660997d2d1551bf39b533d4e388dc4046c93d7"}
Jul 27 18:35:25.982092 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:25.982040    2573 scope.go:117] "RemoveContainer" containerID="189c16730e2109ea178234b875201803231e49f89f34835a83e3248c24b20b35"
Jul 27 18:35:25.991383 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:25.991366    2573 scope.go:117] "RemoveContainer" containerID="76b57755edf6be3d5deb3176310d3672f3abea4fbd463ca3f3d6adc2bfec222e"
Jul 27 18:35:25.998929 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:25.998911    2573 scope.go:117] "RemoveContainer" containerID="b1ab930adf6301435bf1588da0e3b8ae3655b4c3fa6890d7c5bc73ba51ceb330"
Jul 27 18:35:26.004648 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:26.004618    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-bf492"]
Jul 27 18:35:26.006503 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:26.006490    2573 scope.go:117] "RemoveContainer" containerID="189c16730e2109ea178234b875201803231e49f89f34835a83e3248c24b20b35"
Jul 27 18:35:26.006762 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:35:26.006742    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"189c16730e2109ea178234b875201803231e49f89f34835a83e3248c24b20b35\": container with ID starting with 189c16730e2109ea178234b875201803231e49f89f34835a83e3248c24b20b35 not found: ID does not exist" containerID="189c16730e2109ea178234b875201803231e49f89f34835a83e3248c24b20b35"
Jul 27 18:35:26.006809 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:26.006770    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"189c16730e2109ea178234b875201803231e49f89f34835a83e3248c24b20b35"} err="failed to get container status \"189c16730e2109ea178234b875201803231e49f89f34835a83e3248c24b20b35\": rpc error: code = NotFound desc = could not find container \"189c16730e2109ea178234b875201803231e49f89f34835a83e3248c24b20b35\": container with ID starting with 189c16730e2109ea178234b875201803231e49f89f34835a83e3248c24b20b35 not found: ID does not exist"
Jul 27 18:35:26.006809 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:26.006788    2573 scope.go:117] "RemoveContainer" containerID="76b57755edf6be3d5deb3176310d3672f3abea4fbd463ca3f3d6adc2bfec222e"
Jul 27 18:35:26.007012 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:35:26.006994    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76b57755edf6be3d5deb3176310d3672f3abea4fbd463ca3f3d6adc2bfec222e\": container with ID starting with 76b57755edf6be3d5deb3176310d3672f3abea4fbd463ca3f3d6adc2bfec222e not found: ID does not exist" containerID="76b57755edf6be3d5deb3176310d3672f3abea4fbd463ca3f3d6adc2bfec222e"
Jul 27 18:35:26.007049 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:26.007019    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76b57755edf6be3d5deb3176310d3672f3abea4fbd463ca3f3d6adc2bfec222e"} err="failed to get container status \"76b57755edf6be3d5deb3176310d3672f3abea4fbd463ca3f3d6adc2bfec222e\": rpc error: code = NotFound desc = could not find container \"76b57755edf6be3d5deb3176310d3672f3abea4fbd463ca3f3d6adc2bfec222e\": container with ID starting with 76b57755edf6be3d5deb3176310d3672f3abea4fbd463ca3f3d6adc2bfec222e not found: ID does not exist"
Jul 27 18:35:26.007049 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:26.007035    2573 scope.go:117] "RemoveContainer" containerID="b1ab930adf6301435bf1588da0e3b8ae3655b4c3fa6890d7c5bc73ba51ceb330"
Jul 27 18:35:26.007305 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:35:26.007289    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1ab930adf6301435bf1588da0e3b8ae3655b4c3fa6890d7c5bc73ba51ceb330\": container with ID starting with b1ab930adf6301435bf1588da0e3b8ae3655b4c3fa6890d7c5bc73ba51ceb330 not found: ID does not exist" containerID="b1ab930adf6301435bf1588da0e3b8ae3655b4c3fa6890d7c5bc73ba51ceb330"
Jul 27 18:35:26.007358 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:26.007312    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1ab930adf6301435bf1588da0e3b8ae3655b4c3fa6890d7c5bc73ba51ceb330"} err="failed to get container status \"b1ab930adf6301435bf1588da0e3b8ae3655b4c3fa6890d7c5bc73ba51ceb330\": rpc error: code = NotFound desc = could not find container \"b1ab930adf6301435bf1588da0e3b8ae3655b4c3fa6890d7c5bc73ba51ceb330\": container with ID starting with b1ab930adf6301435bf1588da0e3b8ae3655b4c3fa6890d7c5bc73ba51ceb330 not found: ID does not exist"
Jul 27 18:35:26.010768 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:26.010747    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-bf492"]
Jul 27 18:35:26.469375 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:26.469340    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec78b987-9ea1-46da-bb08-c95fdceadc6d" path="/var/lib/kubelet/pods/ec78b987-9ea1-46da-bb08-c95fdceadc6d/volumes"
Jul 27 18:35:27.530869 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:27.530836    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:35:32.636274 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:32.636214    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:35:32.670622 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:32.670590    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-tshtx"]
Jul 27 18:35:32.670929 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:32.670916    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec78b987-9ea1-46da-bb08-c95fdceadc6d" containerName="kube-rbac-proxy"
Jul 27 18:35:32.670974 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:32.670930    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec78b987-9ea1-46da-bb08-c95fdceadc6d" containerName="kube-rbac-proxy"
Jul 27 18:35:32.670974 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:32.670940    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec78b987-9ea1-46da-bb08-c95fdceadc6d" containerName="storage-initializer"
Jul 27 18:35:32.670974 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:32.670946    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec78b987-9ea1-46da-bb08-c95fdceadc6d" containerName="storage-initializer"
Jul 27 18:35:32.670974 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:32.670959    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec78b987-9ea1-46da-bb08-c95fdceadc6d" containerName="kserve-container"
Jul 27 18:35:32.670974 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:32.670963    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec78b987-9ea1-46da-bb08-c95fdceadc6d" containerName="kserve-container"
Jul 27 18:35:32.671127 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:32.671013    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec78b987-9ea1-46da-bb08-c95fdceadc6d" containerName="kserve-container"
Jul 27 18:35:32.671127 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:32.671022    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec78b987-9ea1-46da-bb08-c95fdceadc6d" containerName="kube-rbac-proxy"
Jul 27 18:35:32.675709 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:32.675687    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-tshtx"
Jul 27 18:35:32.678650 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:32.678621    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-predictor-serving-cert\""
Jul 27 18:35:32.678779 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:32.678667    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 18:35:32.679009 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:32.678989    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z6ld7\""
Jul 27 18:35:32.679101 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:32.679034    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 18:35:32.679101 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:32.679063    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 18:35:32.680088 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:32.680070    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
Jul 27 18:35:32.680135 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:32.680126    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 18:35:32.683910 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:32.683888    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-tshtx"]
Jul 27 18:35:32.713991 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:32.713952    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr4cv\" (UniqueName: \"kubernetes.io/projected/02223fd3-c89d-43ac-83c6-43e190bcd6a6-kube-api-access-fr4cv\") pod \"isvc-sklearn-predictor-5f4d89cc44-tshtx\" (UID: \"02223fd3-c89d-43ac-83c6-43e190bcd6a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-tshtx"
Jul 27 18:35:32.714163 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:32.713998    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/02223fd3-c89d-43ac-83c6-43e190bcd6a6-kserve-provision-location\") pod \"isvc-sklearn-predictor-5f4d89cc44-tshtx\" (UID: \"02223fd3-c89d-43ac-83c6-43e190bcd6a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-tshtx"
Jul 27 18:35:32.714163 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:32.714031    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02223fd3-c89d-43ac-83c6-43e190bcd6a6-proxy-tls\") pod \"isvc-sklearn-predictor-5f4d89cc44-tshtx\" (UID: \"02223fd3-c89d-43ac-83c6-43e190bcd6a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-tshtx"
Jul 27 18:35:32.714163 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:32.714067    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02223fd3-c89d-43ac-83c6-43e190bcd6a6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-5f4d89cc44-tshtx\" (UID: \"02223fd3-c89d-43ac-83c6-43e190bcd6a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-tshtx"
Jul 27 18:35:32.714163 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:32.714084    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/02223fd3-c89d-43ac-83c6-43e190bcd6a6-cabundle-cert\") pod \"isvc-sklearn-predictor-5f4d89cc44-tshtx\" (UID: \"02223fd3-c89d-43ac-83c6-43e190bcd6a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-tshtx"
Jul 27 18:35:32.815105 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:32.815068    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr4cv\" (UniqueName: \"kubernetes.io/projected/02223fd3-c89d-43ac-83c6-43e190bcd6a6-kube-api-access-fr4cv\") pod \"isvc-sklearn-predictor-5f4d89cc44-tshtx\" (UID: \"02223fd3-c89d-43ac-83c6-43e190bcd6a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-tshtx"
Jul 27 18:35:32.815105 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:32.815109    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/02223fd3-c89d-43ac-83c6-43e190bcd6a6-kserve-provision-location\") pod \"isvc-sklearn-predictor-5f4d89cc44-tshtx\" (UID: \"02223fd3-c89d-43ac-83c6-43e190bcd6a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-tshtx"
Jul 27 18:35:32.815420 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:32.815135    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02223fd3-c89d-43ac-83c6-43e190bcd6a6-proxy-tls\") pod \"isvc-sklearn-predictor-5f4d89cc44-tshtx\" (UID: \"02223fd3-c89d-43ac-83c6-43e190bcd6a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-tshtx"
Jul 27 18:35:32.815420 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:32.815170    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02223fd3-c89d-43ac-83c6-43e190bcd6a6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-5f4d89cc44-tshtx\" (UID: \"02223fd3-c89d-43ac-83c6-43e190bcd6a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-tshtx"
Jul 27 18:35:32.815420 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:32.815191    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/02223fd3-c89d-43ac-83c6-43e190bcd6a6-cabundle-cert\") pod \"isvc-sklearn-predictor-5f4d89cc44-tshtx\" (UID: \"02223fd3-c89d-43ac-83c6-43e190bcd6a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-tshtx"
Jul 27 18:35:32.815586 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:32.815564    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/02223fd3-c89d-43ac-83c6-43e190bcd6a6-kserve-provision-location\") pod \"isvc-sklearn-predictor-5f4d89cc44-tshtx\" (UID: \"02223fd3-c89d-43ac-83c6-43e190bcd6a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-tshtx"
Jul 27 18:35:32.815865 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:32.815848    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/02223fd3-c89d-43ac-83c6-43e190bcd6a6-cabundle-cert\") pod \"isvc-sklearn-predictor-5f4d89cc44-tshtx\" (UID: \"02223fd3-c89d-43ac-83c6-43e190bcd6a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-tshtx"
Jul 27 18:35:32.815929 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:32.815913    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02223fd3-c89d-43ac-83c6-43e190bcd6a6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-5f4d89cc44-tshtx\" (UID: \"02223fd3-c89d-43ac-83c6-43e190bcd6a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-tshtx"
Jul 27 18:35:32.817784 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:32.817768    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02223fd3-c89d-43ac-83c6-43e190bcd6a6-proxy-tls\") pod \"isvc-sklearn-predictor-5f4d89cc44-tshtx\" (UID: \"02223fd3-c89d-43ac-83c6-43e190bcd6a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-tshtx"
Jul 27 18:35:32.823934 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:32.823912    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr4cv\" (UniqueName: \"kubernetes.io/projected/02223fd3-c89d-43ac-83c6-43e190bcd6a6-kube-api-access-fr4cv\") pod \"isvc-sklearn-predictor-5f4d89cc44-tshtx\" (UID: \"02223fd3-c89d-43ac-83c6-43e190bcd6a6\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-tshtx"
Jul 27 18:35:32.856900 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:32.856864    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:35:32.986932 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:32.986894    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-tshtx"
Jul 27 18:35:33.129897 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:33.129871    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-tshtx"]
Jul 27 18:35:33.131810 ip-10-0-140-0 kubenswrapper[2573]: W0727 18:35:33.131780    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02223fd3_c89d_43ac_83c6_43e190bcd6a6.slice/crio-dfebf54725f6a6bae385ff4b29cf8f82606e41341fee8681befdfd5834fbbedc WatchSource:0}: Error finding container dfebf54725f6a6bae385ff4b29cf8f82606e41341fee8681befdfd5834fbbedc: Status 404 returned error can't find the container with id dfebf54725f6a6bae385ff4b29cf8f82606e41341fee8681befdfd5834fbbedc
Jul 27 18:35:34.019079 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:34.019045    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-tshtx" event={"ID":"02223fd3-c89d-43ac-83c6-43e190bcd6a6","Type":"ContainerStarted","Data":"daff2e7ef3616b9d24aef50b786078f68d4a092fb4687df055a370bd2fd12162"}
Jul 27 18:35:34.019585 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:34.019101    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-tshtx" event={"ID":"02223fd3-c89d-43ac-83c6-43e190bcd6a6","Type":"ContainerStarted","Data":"dfebf54725f6a6bae385ff4b29cf8f82606e41341fee8681befdfd5834fbbedc"}
Jul 27 18:35:38.039623 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:38.039584    2573 generic.go:356] "Generic (PLEG): container finished" podID="02223fd3-c89d-43ac-83c6-43e190bcd6a6" containerID="daff2e7ef3616b9d24aef50b786078f68d4a092fb4687df055a370bd2fd12162" exitCode=0
Jul 27 18:35:38.040028 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:38.039655    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-tshtx" event={"ID":"02223fd3-c89d-43ac-83c6-43e190bcd6a6","Type":"ContainerDied","Data":"daff2e7ef3616b9d24aef50b786078f68d4a092fb4687df055a370bd2fd12162"}
Jul 27 18:35:39.045725 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:39.045687    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-tshtx" event={"ID":"02223fd3-c89d-43ac-83c6-43e190bcd6a6","Type":"ContainerStarted","Data":"533036f79c9a4ae7846d5e040c98fbeeeb81394482e4f98b44d6ba959d63bf7f"}
Jul 27 18:35:39.046135 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:39.045755    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-tshtx" event={"ID":"02223fd3-c89d-43ac-83c6-43e190bcd6a6","Type":"ContainerStarted","Data":"76d464eaf524add89a346751710500cb78f4a9a8ae11948cc1364ed9ea2f4028"}
Jul 27 18:35:39.046135 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:39.046095    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-tshtx"
Jul 27 18:35:39.046286 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:39.046244    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-tshtx"
Jul 27 18:35:39.047592 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:39.047566    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-tshtx" podUID="02223fd3-c89d-43ac-83c6-43e190bcd6a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Jul 27 18:35:39.077813 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:39.077758    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-tshtx" podStartSLOduration=3.202317204 podStartE2EDuration="7.077742983s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.875425779s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:35:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:35:39.076430652 +0000 UTC m=+4081.219832705" watchObservedRunningTime="2026-07-27 18:35:39.077742983 +0000 UTC m=+4081.221145034"
Jul 27 18:35:40.050972 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:40.050933    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-tshtx" podUID="02223fd3-c89d-43ac-83c6-43e190bcd6a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Jul 27 18:35:45.055876 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:45.055843    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-tshtx"
Jul 27 18:35:45.056552 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:45.056523    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-tshtx" podUID="02223fd3-c89d-43ac-83c6-43e190bcd6a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Jul 27 18:35:55.057071 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:35:55.057029    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-tshtx" podUID="02223fd3-c89d-43ac-83c6-43e190bcd6a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Jul 27 18:36:05.056969 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:36:05.056928    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-tshtx" podUID="02223fd3-c89d-43ac-83c6-43e190bcd6a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Jul 27 18:36:15.057364 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:36:15.057317    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-tshtx" podUID="02223fd3-c89d-43ac-83c6-43e190bcd6a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Jul 27 18:36:25.056700 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:36:25.056663    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-tshtx" podUID="02223fd3-c89d-43ac-83c6-43e190bcd6a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Jul 27 18:36:35.057176 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:36:35.057089    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-tshtx" podUID="02223fd3-c89d-43ac-83c6-43e190bcd6a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Jul 27 18:36:45.057358 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:36:45.057316    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-tshtx" podUID="02223fd3-c89d-43ac-83c6-43e190bcd6a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Jul 27 18:36:48.467600 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:36:48.467560    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-tshtx" podUID="02223fd3-c89d-43ac-83c6-43e190bcd6a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.66:8080: connect: connection refused"
Jul 27 18:36:58.468973 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:36:58.468937    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-tshtx"
Jul 27 18:37:02.787729 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:02.787691    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-tshtx"]
Jul 27 18:37:02.788234 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:02.787962    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-tshtx" podUID="02223fd3-c89d-43ac-83c6-43e190bcd6a6" containerName="kserve-container" containerID="cri-o://76d464eaf524add89a346751710500cb78f4a9a8ae11948cc1364ed9ea2f4028" gracePeriod=30
Jul 27 18:37:02.788234 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:02.787990    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-tshtx" podUID="02223fd3-c89d-43ac-83c6-43e190bcd6a6" containerName="kube-rbac-proxy" containerID="cri-o://533036f79c9a4ae7846d5e040c98fbeeeb81394482e4f98b44d6ba959d63bf7f" gracePeriod=30
Jul 27 18:37:02.813738 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:02.813712    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:37:03.414171 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:03.414136    2573 generic.go:356] "Generic (PLEG): container finished" podID="02223fd3-c89d-43ac-83c6-43e190bcd6a6" containerID="533036f79c9a4ae7846d5e040c98fbeeeb81394482e4f98b44d6ba959d63bf7f" exitCode=2
Jul 27 18:37:03.414375 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:03.414209    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-tshtx" event={"ID":"02223fd3-c89d-43ac-83c6-43e190bcd6a6","Type":"ContainerDied","Data":"533036f79c9a4ae7846d5e040c98fbeeeb81394482e4f98b44d6ba959d63bf7f"}
Jul 27 18:37:05.052113 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:05.052076    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-tshtx" podUID="02223fd3-c89d-43ac-83c6-43e190bcd6a6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.66:8643/healthz\": dial tcp 10.133.0.66:8643: connect: connection refused"
Jul 27 18:37:07.644516 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:07.644487    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-tshtx"
Jul 27 18:37:07.720151 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:07.720116    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/02223fd3-c89d-43ac-83c6-43e190bcd6a6-kserve-provision-location\") pod \"02223fd3-c89d-43ac-83c6-43e190bcd6a6\" (UID: \"02223fd3-c89d-43ac-83c6-43e190bcd6a6\") "
Jul 27 18:37:07.720344 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:07.720168    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/02223fd3-c89d-43ac-83c6-43e190bcd6a6-cabundle-cert\") pod \"02223fd3-c89d-43ac-83c6-43e190bcd6a6\" (UID: \"02223fd3-c89d-43ac-83c6-43e190bcd6a6\") "
Jul 27 18:37:07.720344 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:07.720198    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fr4cv\" (UniqueName: \"kubernetes.io/projected/02223fd3-c89d-43ac-83c6-43e190bcd6a6-kube-api-access-fr4cv\") pod \"02223fd3-c89d-43ac-83c6-43e190bcd6a6\" (UID: \"02223fd3-c89d-43ac-83c6-43e190bcd6a6\") "
Jul 27 18:37:07.720344 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:07.720221    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02223fd3-c89d-43ac-83c6-43e190bcd6a6-kube-rbac-proxy-sar-config\") pod \"02223fd3-c89d-43ac-83c6-43e190bcd6a6\" (UID: \"02223fd3-c89d-43ac-83c6-43e190bcd6a6\") "
Jul 27 18:37:07.720344 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:07.720284    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02223fd3-c89d-43ac-83c6-43e190bcd6a6-proxy-tls\") pod \"02223fd3-c89d-43ac-83c6-43e190bcd6a6\" (UID: \"02223fd3-c89d-43ac-83c6-43e190bcd6a6\") "
Jul 27 18:37:07.720580 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:07.720551    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02223fd3-c89d-43ac-83c6-43e190bcd6a6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "02223fd3-c89d-43ac-83c6-43e190bcd6a6" (UID: "02223fd3-c89d-43ac-83c6-43e190bcd6a6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:37:07.720622 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:07.720573    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02223fd3-c89d-43ac-83c6-43e190bcd6a6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "02223fd3-c89d-43ac-83c6-43e190bcd6a6" (UID: "02223fd3-c89d-43ac-83c6-43e190bcd6a6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:37:07.720718 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:07.720683    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02223fd3-c89d-43ac-83c6-43e190bcd6a6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "02223fd3-c89d-43ac-83c6-43e190bcd6a6" (UID: "02223fd3-c89d-43ac-83c6-43e190bcd6a6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:37:07.722356 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:07.722332    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02223fd3-c89d-43ac-83c6-43e190bcd6a6-kube-api-access-fr4cv" (OuterVolumeSpecName: "kube-api-access-fr4cv") pod "02223fd3-c89d-43ac-83c6-43e190bcd6a6" (UID: "02223fd3-c89d-43ac-83c6-43e190bcd6a6"). InnerVolumeSpecName "kube-api-access-fr4cv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:37:07.722454 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:07.722375    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02223fd3-c89d-43ac-83c6-43e190bcd6a6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "02223fd3-c89d-43ac-83c6-43e190bcd6a6" (UID: "02223fd3-c89d-43ac-83c6-43e190bcd6a6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:37:07.821688 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:07.821597    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr4cv\" (UniqueName: \"kubernetes.io/projected/02223fd3-c89d-43ac-83c6-43e190bcd6a6-kube-api-access-fr4cv\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:37:07.821688 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:07.821629    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02223fd3-c89d-43ac-83c6-43e190bcd6a6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:37:07.821688 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:07.821640    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02223fd3-c89d-43ac-83c6-43e190bcd6a6-proxy-tls\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:37:07.821688 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:07.821650    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/02223fd3-c89d-43ac-83c6-43e190bcd6a6-kserve-provision-location\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:37:07.821688 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:07.821660    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/02223fd3-c89d-43ac-83c6-43e190bcd6a6-cabundle-cert\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:37:07.897448 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:07.897413    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:37:07.950512 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:07.950477    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dskjm"]
Jul 27 18:37:07.950868 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:07.950854    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02223fd3-c89d-43ac-83c6-43e190bcd6a6" containerName="storage-initializer"
Jul 27 18:37:07.950931 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:07.950870    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="02223fd3-c89d-43ac-83c6-43e190bcd6a6" containerName="storage-initializer"
Jul 27 18:37:07.950931 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:07.950884    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02223fd3-c89d-43ac-83c6-43e190bcd6a6" containerName="kserve-container"
Jul 27 18:37:07.950931 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:07.950890    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="02223fd3-c89d-43ac-83c6-43e190bcd6a6" containerName="kserve-container"
Jul 27 18:37:07.950931 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:07.950918    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02223fd3-c89d-43ac-83c6-43e190bcd6a6" containerName="kube-rbac-proxy"
Jul 27 18:37:07.950931 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:07.950925    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="02223fd3-c89d-43ac-83c6-43e190bcd6a6" containerName="kube-rbac-proxy"
Jul 27 18:37:07.951084 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:07.950978    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="02223fd3-c89d-43ac-83c6-43e190bcd6a6" containerName="kserve-container"
Jul 27 18:37:07.951084 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:07.950993    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="02223fd3-c89d-43ac-83c6-43e190bcd6a6" containerName="kube-rbac-proxy"
Jul 27 18:37:07.954460 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:07.954442    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dskjm"
Jul 27 18:37:07.957309 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:07.957285    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
Jul 27 18:37:07.957444 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:07.957371    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
Jul 27 18:37:07.964309 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:07.964285    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dskjm"]
Jul 27 18:37:08.023365 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:08.023331    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2eedf056-c257-4f09-847b-1db5cc7a3846-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-dskjm\" (UID: \"2eedf056-c257-4f09-847b-1db5cc7a3846\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dskjm"
Jul 27 18:37:08.023557 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:08.023380    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2eedf056-c257-4f09-847b-1db5cc7a3846-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-dskjm\" (UID: \"2eedf056-c257-4f09-847b-1db5cc7a3846\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dskjm"
Jul 27 18:37:08.023557 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:08.023438    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85pvn\" (UniqueName: \"kubernetes.io/projected/2eedf056-c257-4f09-847b-1db5cc7a3846-kube-api-access-85pvn\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-dskjm\" (UID: \"2eedf056-c257-4f09-847b-1db5cc7a3846\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dskjm"
Jul 27 18:37:08.023557 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:08.023467    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2eedf056-c257-4f09-847b-1db5cc7a3846-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-dskjm\" (UID: \"2eedf056-c257-4f09-847b-1db5cc7a3846\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dskjm"
Jul 27 18:37:08.023557 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:08.023487    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2eedf056-c257-4f09-847b-1db5cc7a3846-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-dskjm\" (UID: \"2eedf056-c257-4f09-847b-1db5cc7a3846\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dskjm"
Jul 27 18:37:08.085315 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:08.085217    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:37:08.124317 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:08.124267    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2eedf056-c257-4f09-847b-1db5cc7a3846-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-dskjm\" (UID: \"2eedf056-c257-4f09-847b-1db5cc7a3846\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dskjm"
Jul 27 18:37:08.124520 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:08.124334    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2eedf056-c257-4f09-847b-1db5cc7a3846-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-dskjm\" (UID: \"2eedf056-c257-4f09-847b-1db5cc7a3846\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dskjm"
Jul 27 18:37:08.124520 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:08.124360    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85pvn\" (UniqueName: \"kubernetes.io/projected/2eedf056-c257-4f09-847b-1db5cc7a3846-kube-api-access-85pvn\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-dskjm\" (UID: \"2eedf056-c257-4f09-847b-1db5cc7a3846\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dskjm"
Jul 27 18:37:08.124520 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:08.124383    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2eedf056-c257-4f09-847b-1db5cc7a3846-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-dskjm\" (UID: \"2eedf056-c257-4f09-847b-1db5cc7a3846\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dskjm"
Jul 27 18:37:08.124520 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:08.124408    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2eedf056-c257-4f09-847b-1db5cc7a3846-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-dskjm\" (UID: \"2eedf056-c257-4f09-847b-1db5cc7a3846\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dskjm"
Jul 27 18:37:08.124840 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:08.124817    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2eedf056-c257-4f09-847b-1db5cc7a3846-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-dskjm\" (UID: \"2eedf056-c257-4f09-847b-1db5cc7a3846\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dskjm"
Jul 27 18:37:08.125022 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:08.125002    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2eedf056-c257-4f09-847b-1db5cc7a3846-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-dskjm\" (UID: \"2eedf056-c257-4f09-847b-1db5cc7a3846\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dskjm"
Jul 27 18:37:08.125068 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:08.125031    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2eedf056-c257-4f09-847b-1db5cc7a3846-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-dskjm\" (UID: \"2eedf056-c257-4f09-847b-1db5cc7a3846\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dskjm"
Jul 27 18:37:08.126818 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:08.126801    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2eedf056-c257-4f09-847b-1db5cc7a3846-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-dskjm\" (UID: \"2eedf056-c257-4f09-847b-1db5cc7a3846\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dskjm"
Jul 27 18:37:08.132665 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:08.132646    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85pvn\" (UniqueName: \"kubernetes.io/projected/2eedf056-c257-4f09-847b-1db5cc7a3846-kube-api-access-85pvn\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-dskjm\" (UID: \"2eedf056-c257-4f09-847b-1db5cc7a3846\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dskjm"
Jul 27 18:37:08.265330 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:08.265278    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dskjm"
Jul 27 18:37:08.404162 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:08.404137    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dskjm"]
Jul 27 18:37:08.406415 ip-10-0-140-0 kubenswrapper[2573]: W0727 18:37:08.406385    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eedf056_c257_4f09_847b_1db5cc7a3846.slice/crio-fa420e2e1c28080aa6fb2576b63bb95d199511cb7adad25b6ba530649a217ca4 WatchSource:0}: Error finding container fa420e2e1c28080aa6fb2576b63bb95d199511cb7adad25b6ba530649a217ca4: Status 404 returned error can't find the container with id fa420e2e1c28080aa6fb2576b63bb95d199511cb7adad25b6ba530649a217ca4
Jul 27 18:37:08.437231 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:08.437200    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dskjm" event={"ID":"2eedf056-c257-4f09-847b-1db5cc7a3846","Type":"ContainerStarted","Data":"fa420e2e1c28080aa6fb2576b63bb95d199511cb7adad25b6ba530649a217ca4"}
Jul 27 18:37:08.439091 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:08.439056    2573 generic.go:356] "Generic (PLEG): container finished" podID="02223fd3-c89d-43ac-83c6-43e190bcd6a6" containerID="76d464eaf524add89a346751710500cb78f4a9a8ae11948cc1364ed9ea2f4028" exitCode=0
Jul 27 18:37:08.439201 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:08.439119    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-tshtx" event={"ID":"02223fd3-c89d-43ac-83c6-43e190bcd6a6","Type":"ContainerDied","Data":"76d464eaf524add89a346751710500cb78f4a9a8ae11948cc1364ed9ea2f4028"}
Jul 27 18:37:08.439201 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:08.439123    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-tshtx"
Jul 27 18:37:08.439201 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:08.439163    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-tshtx" event={"ID":"02223fd3-c89d-43ac-83c6-43e190bcd6a6","Type":"ContainerDied","Data":"dfebf54725f6a6bae385ff4b29cf8f82606e41341fee8681befdfd5834fbbedc"}
Jul 27 18:37:08.439201 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:08.439191    2573 scope.go:117] "RemoveContainer" containerID="533036f79c9a4ae7846d5e040c98fbeeeb81394482e4f98b44d6ba959d63bf7f"
Jul 27 18:37:08.448090 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:08.448068    2573 scope.go:117] "RemoveContainer" containerID="76d464eaf524add89a346751710500cb78f4a9a8ae11948cc1364ed9ea2f4028"
Jul 27 18:37:08.455417 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:08.455394    2573 scope.go:117] "RemoveContainer" containerID="daff2e7ef3616b9d24aef50b786078f68d4a092fb4687df055a370bd2fd12162"
Jul 27 18:37:08.463012 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:08.462988    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-tshtx"]
Jul 27 18:37:08.464239 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:08.464216    2573 scope.go:117] "RemoveContainer" containerID="533036f79c9a4ae7846d5e040c98fbeeeb81394482e4f98b44d6ba959d63bf7f"
Jul 27 18:37:08.464553 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:37:08.464528    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"533036f79c9a4ae7846d5e040c98fbeeeb81394482e4f98b44d6ba959d63bf7f\": container with ID starting with 533036f79c9a4ae7846d5e040c98fbeeeb81394482e4f98b44d6ba959d63bf7f not found: ID does not exist" containerID="533036f79c9a4ae7846d5e040c98fbeeeb81394482e4f98b44d6ba959d63bf7f"
Jul 27 18:37:08.464629 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:08.464566    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"533036f79c9a4ae7846d5e040c98fbeeeb81394482e4f98b44d6ba959d63bf7f"} err="failed to get container status \"533036f79c9a4ae7846d5e040c98fbeeeb81394482e4f98b44d6ba959d63bf7f\": rpc error: code = NotFound desc = could not find container \"533036f79c9a4ae7846d5e040c98fbeeeb81394482e4f98b44d6ba959d63bf7f\": container with ID starting with 533036f79c9a4ae7846d5e040c98fbeeeb81394482e4f98b44d6ba959d63bf7f not found: ID does not exist"
Jul 27 18:37:08.464629 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:08.464590    2573 scope.go:117] "RemoveContainer" containerID="76d464eaf524add89a346751710500cb78f4a9a8ae11948cc1364ed9ea2f4028"
Jul 27 18:37:08.464874 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:37:08.464856    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76d464eaf524add89a346751710500cb78f4a9a8ae11948cc1364ed9ea2f4028\": container with ID starting with 76d464eaf524add89a346751710500cb78f4a9a8ae11948cc1364ed9ea2f4028 not found: ID does not exist" containerID="76d464eaf524add89a346751710500cb78f4a9a8ae11948cc1364ed9ea2f4028"
Jul 27 18:37:08.464921 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:08.464879    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76d464eaf524add89a346751710500cb78f4a9a8ae11948cc1364ed9ea2f4028"} err="failed to get container status \"76d464eaf524add89a346751710500cb78f4a9a8ae11948cc1364ed9ea2f4028\": rpc error: code = NotFound desc = could not find container \"76d464eaf524add89a346751710500cb78f4a9a8ae11948cc1364ed9ea2f4028\": container with ID starting with 76d464eaf524add89a346751710500cb78f4a9a8ae11948cc1364ed9ea2f4028 not found: ID does not exist"
Jul 27 18:37:08.464921 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:08.464895    2573 scope.go:117] "RemoveContainer" containerID="daff2e7ef3616b9d24aef50b786078f68d4a092fb4687df055a370bd2fd12162"
Jul 27 18:37:08.465175 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:37:08.465158    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daff2e7ef3616b9d24aef50b786078f68d4a092fb4687df055a370bd2fd12162\": container with ID starting with daff2e7ef3616b9d24aef50b786078f68d4a092fb4687df055a370bd2fd12162 not found: ID does not exist" containerID="daff2e7ef3616b9d24aef50b786078f68d4a092fb4687df055a370bd2fd12162"
Jul 27 18:37:08.465239 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:08.465182    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daff2e7ef3616b9d24aef50b786078f68d4a092fb4687df055a370bd2fd12162"} err="failed to get container status \"daff2e7ef3616b9d24aef50b786078f68d4a092fb4687df055a370bd2fd12162\": rpc error: code = NotFound desc = could not find container \"daff2e7ef3616b9d24aef50b786078f68d4a092fb4687df055a370bd2fd12162\": container with ID starting with daff2e7ef3616b9d24aef50b786078f68d4a092fb4687df055a370bd2fd12162 not found: ID does not exist"
Jul 27 18:37:08.469393 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:08.469376    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5f4d89cc44-tshtx"]
Jul 27 18:37:09.445590 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:09.445555    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dskjm" event={"ID":"2eedf056-c257-4f09-847b-1db5cc7a3846","Type":"ContainerStarted","Data":"41c57e9df4952eb3113eb896eed9b6739db8b7690d3db8d74302cc63e2835843"}
Jul 27 18:37:10.469725 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:10.469687    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02223fd3-c89d-43ac-83c6-43e190bcd6a6" path="/var/lib/kubelet/pods/02223fd3-c89d-43ac-83c6-43e190bcd6a6/volumes"
Jul 27 18:37:12.460177 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:12.460145    2573 generic.go:356] "Generic (PLEG): container finished" podID="2eedf056-c257-4f09-847b-1db5cc7a3846" containerID="41c57e9df4952eb3113eb896eed9b6739db8b7690d3db8d74302cc63e2835843" exitCode=0
Jul 27 18:37:12.460671 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:12.460224    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dskjm" event={"ID":"2eedf056-c257-4f09-847b-1db5cc7a3846","Type":"ContainerDied","Data":"41c57e9df4952eb3113eb896eed9b6739db8b7690d3db8d74302cc63e2835843"}
Jul 27 18:37:13.466771 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:13.466740    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dskjm" event={"ID":"2eedf056-c257-4f09-847b-1db5cc7a3846","Type":"ContainerStarted","Data":"a292f7f43a21448ee5684066f64d3d366c02edde7606ec7b721ad13e7a1ede79"}
Jul 27 18:37:13.467175 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:13.466813    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dskjm" event={"ID":"2eedf056-c257-4f09-847b-1db5cc7a3846","Type":"ContainerStarted","Data":"39c774a2b3b0c1e64e21106e283bbed4d43dfd2dcaa017bdd9e839fd3ed9a47a"}
Jul 27 18:37:13.467175 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:13.467006    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dskjm"
Jul 27 18:37:13.489510 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:13.489450    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dskjm" podStartSLOduration=2.668088384 podStartE2EDuration="6.489426888s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.821338504s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:37:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:37:13.487882841 +0000 UTC m=+4175.631284892" watchObservedRunningTime="2026-07-27 18:37:13.489426888 +0000 UTC m=+4175.632828936"
Jul 27 18:37:14.471327 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:14.471291    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dskjm"
Jul 27 18:37:20.480979 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:20.480950    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dskjm"
Jul 27 18:37:38.597171 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:38.597142    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vc2tz_87dac797-f8d2-4d36-999f-f8ce855dbeff/ovn-acl-logging/0.log"
Jul 27 18:37:38.606560 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:38.606537    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vc2tz_87dac797-f8d2-4d36-999f-f8ce855dbeff/ovn-acl-logging/0.log"
Jul 27 18:37:50.488325 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:37:50.488286    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dskjm" podUID="2eedf056-c257-4f09-847b-1db5cc7a3846" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Jul 27 18:38:00.484227 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:00.484142    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dskjm"
Jul 27 18:38:07.411440 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:07.411403    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:38:08.000407 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:08.000370    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dskjm"]
Jul 27 18:38:08.000677 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:08.000651    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dskjm" podUID="2eedf056-c257-4f09-847b-1db5cc7a3846" containerName="kserve-container" containerID="cri-o://39c774a2b3b0c1e64e21106e283bbed4d43dfd2dcaa017bdd9e839fd3ed9a47a" gracePeriod=30
Jul 27 18:38:08.000845 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:08.000668    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dskjm" podUID="2eedf056-c257-4f09-847b-1db5cc7a3846" containerName="kube-rbac-proxy" containerID="cri-o://a292f7f43a21448ee5684066f64d3d366c02edde7606ec7b721ad13e7a1ede79" gracePeriod=30
Jul 27 18:38:08.300166 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:08.300078    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:38:08.708526 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:08.708495    2573 generic.go:356] "Generic (PLEG): container finished" podID="2eedf056-c257-4f09-847b-1db5cc7a3846" containerID="a292f7f43a21448ee5684066f64d3d366c02edde7606ec7b721ad13e7a1ede79" exitCode=2
Jul 27 18:38:08.708910 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:08.708563    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dskjm" event={"ID":"2eedf056-c257-4f09-847b-1db5cc7a3846","Type":"ContainerDied","Data":"a292f7f43a21448ee5684066f64d3d366c02edde7606ec7b721ad13e7a1ede79"}
Jul 27 18:38:10.477439 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:10.477397    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dskjm" podUID="2eedf056-c257-4f09-847b-1db5cc7a3846" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.67:8643/healthz\": dial tcp 10.133.0.67:8643: connect: connection refused"
Jul 27 18:38:11.523445 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:11.523402    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dskjm" podUID="2eedf056-c257-4f09-847b-1db5cc7a3846" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.67:8080/v2/models/sklearn-v2-mlserver/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 27 18:38:13.145389 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:13.145353    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:38:13.507342 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:13.507230    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:38:15.476534 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:15.476486    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dskjm" podUID="2eedf056-c257-4f09-847b-1db5cc7a3846" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.67:8643/healthz\": dial tcp 10.133.0.67:8643: connect: connection refused"
Jul 27 18:38:15.655862 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:15.655835    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dskjm"
Jul 27 18:38:15.740606 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:15.740520    2573 generic.go:356] "Generic (PLEG): container finished" podID="2eedf056-c257-4f09-847b-1db5cc7a3846" containerID="39c774a2b3b0c1e64e21106e283bbed4d43dfd2dcaa017bdd9e839fd3ed9a47a" exitCode=0
Jul 27 18:38:15.740606 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:15.740580    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dskjm"
Jul 27 18:38:15.740785 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:15.740597    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dskjm" event={"ID":"2eedf056-c257-4f09-847b-1db5cc7a3846","Type":"ContainerDied","Data":"39c774a2b3b0c1e64e21106e283bbed4d43dfd2dcaa017bdd9e839fd3ed9a47a"}
Jul 27 18:38:15.740785 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:15.740671    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dskjm" event={"ID":"2eedf056-c257-4f09-847b-1db5cc7a3846","Type":"ContainerDied","Data":"fa420e2e1c28080aa6fb2576b63bb95d199511cb7adad25b6ba530649a217ca4"}
Jul 27 18:38:15.740785 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:15.740711    2573 scope.go:117] "RemoveContainer" containerID="a292f7f43a21448ee5684066f64d3d366c02edde7606ec7b721ad13e7a1ede79"
Jul 27 18:38:15.748735 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:15.748717    2573 scope.go:117] "RemoveContainer" containerID="39c774a2b3b0c1e64e21106e283bbed4d43dfd2dcaa017bdd9e839fd3ed9a47a"
Jul 27 18:38:15.756090 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:15.756070    2573 scope.go:117] "RemoveContainer" containerID="41c57e9df4952eb3113eb896eed9b6739db8b7690d3db8d74302cc63e2835843"
Jul 27 18:38:15.763368 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:15.763350    2573 scope.go:117] "RemoveContainer" containerID="a292f7f43a21448ee5684066f64d3d366c02edde7606ec7b721ad13e7a1ede79"
Jul 27 18:38:15.763693 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:38:15.763673    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a292f7f43a21448ee5684066f64d3d366c02edde7606ec7b721ad13e7a1ede79\": container with ID starting with a292f7f43a21448ee5684066f64d3d366c02edde7606ec7b721ad13e7a1ede79 not found: ID does not exist" containerID="a292f7f43a21448ee5684066f64d3d366c02edde7606ec7b721ad13e7a1ede79"
Jul 27 18:38:15.763782 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:15.763709    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a292f7f43a21448ee5684066f64d3d366c02edde7606ec7b721ad13e7a1ede79"} err="failed to get container status \"a292f7f43a21448ee5684066f64d3d366c02edde7606ec7b721ad13e7a1ede79\": rpc error: code = NotFound desc = could not find container \"a292f7f43a21448ee5684066f64d3d366c02edde7606ec7b721ad13e7a1ede79\": container with ID starting with a292f7f43a21448ee5684066f64d3d366c02edde7606ec7b721ad13e7a1ede79 not found: ID does not exist"
Jul 27 18:38:15.763782 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:15.763736    2573 scope.go:117] "RemoveContainer" containerID="39c774a2b3b0c1e64e21106e283bbed4d43dfd2dcaa017bdd9e839fd3ed9a47a"
Jul 27 18:38:15.763996 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:38:15.763981    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39c774a2b3b0c1e64e21106e283bbed4d43dfd2dcaa017bdd9e839fd3ed9a47a\": container with ID starting with 39c774a2b3b0c1e64e21106e283bbed4d43dfd2dcaa017bdd9e839fd3ed9a47a not found: ID does not exist" containerID="39c774a2b3b0c1e64e21106e283bbed4d43dfd2dcaa017bdd9e839fd3ed9a47a"
Jul 27 18:38:15.764036 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:15.764002    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39c774a2b3b0c1e64e21106e283bbed4d43dfd2dcaa017bdd9e839fd3ed9a47a"} err="failed to get container status \"39c774a2b3b0c1e64e21106e283bbed4d43dfd2dcaa017bdd9e839fd3ed9a47a\": rpc error: code = NotFound desc = could not find container \"39c774a2b3b0c1e64e21106e283bbed4d43dfd2dcaa017bdd9e839fd3ed9a47a\": container with ID starting with 39c774a2b3b0c1e64e21106e283bbed4d43dfd2dcaa017bdd9e839fd3ed9a47a not found: ID does not exist"
Jul 27 18:38:15.764036 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:15.764020    2573 scope.go:117] "RemoveContainer" containerID="41c57e9df4952eb3113eb896eed9b6739db8b7690d3db8d74302cc63e2835843"
Jul 27 18:38:15.764207 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:38:15.764193    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41c57e9df4952eb3113eb896eed9b6739db8b7690d3db8d74302cc63e2835843\": container with ID starting with 41c57e9df4952eb3113eb896eed9b6739db8b7690d3db8d74302cc63e2835843 not found: ID does not exist" containerID="41c57e9df4952eb3113eb896eed9b6739db8b7690d3db8d74302cc63e2835843"
Jul 27 18:38:15.764267 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:15.764209    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41c57e9df4952eb3113eb896eed9b6739db8b7690d3db8d74302cc63e2835843"} err="failed to get container status \"41c57e9df4952eb3113eb896eed9b6739db8b7690d3db8d74302cc63e2835843\": rpc error: code = NotFound desc = could not find container \"41c57e9df4952eb3113eb896eed9b6739db8b7690d3db8d74302cc63e2835843\": container with ID starting with 41c57e9df4952eb3113eb896eed9b6739db8b7690d3db8d74302cc63e2835843 not found: ID does not exist"
Jul 27 18:38:15.778694 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:15.778664    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-85pvn\" (UniqueName: \"kubernetes.io/projected/2eedf056-c257-4f09-847b-1db5cc7a3846-kube-api-access-85pvn\") pod \"2eedf056-c257-4f09-847b-1db5cc7a3846\" (UID: \"2eedf056-c257-4f09-847b-1db5cc7a3846\") "
Jul 27 18:38:15.778824 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:15.778733    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2eedf056-c257-4f09-847b-1db5cc7a3846-kube-rbac-proxy-sar-config\") pod \"2eedf056-c257-4f09-847b-1db5cc7a3846\" (UID: \"2eedf056-c257-4f09-847b-1db5cc7a3846\") "
Jul 27 18:38:15.778824 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:15.778753    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2eedf056-c257-4f09-847b-1db5cc7a3846-cabundle-cert\") pod \"2eedf056-c257-4f09-847b-1db5cc7a3846\" (UID: \"2eedf056-c257-4f09-847b-1db5cc7a3846\") "
Jul 27 18:38:15.778824 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:15.778795    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2eedf056-c257-4f09-847b-1db5cc7a3846-kserve-provision-location\") pod \"2eedf056-c257-4f09-847b-1db5cc7a3846\" (UID: \"2eedf056-c257-4f09-847b-1db5cc7a3846\") "
Jul 27 18:38:15.778984 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:15.778823    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2eedf056-c257-4f09-847b-1db5cc7a3846-proxy-tls\") pod \"2eedf056-c257-4f09-847b-1db5cc7a3846\" (UID: \"2eedf056-c257-4f09-847b-1db5cc7a3846\") "
Jul 27 18:38:15.779140 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:15.779113    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eedf056-c257-4f09-847b-1db5cc7a3846-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2eedf056-c257-4f09-847b-1db5cc7a3846" (UID: "2eedf056-c257-4f09-847b-1db5cc7a3846"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:38:15.779140 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:15.779125    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eedf056-c257-4f09-847b-1db5cc7a3846-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2eedf056-c257-4f09-847b-1db5cc7a3846" (UID: "2eedf056-c257-4f09-847b-1db5cc7a3846"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:38:15.779314 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:15.779151    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eedf056-c257-4f09-847b-1db5cc7a3846-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2eedf056-c257-4f09-847b-1db5cc7a3846" (UID: "2eedf056-c257-4f09-847b-1db5cc7a3846"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:38:15.780882 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:15.780852    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eedf056-c257-4f09-847b-1db5cc7a3846-kube-api-access-85pvn" (OuterVolumeSpecName: "kube-api-access-85pvn") pod "2eedf056-c257-4f09-847b-1db5cc7a3846" (UID: "2eedf056-c257-4f09-847b-1db5cc7a3846"). InnerVolumeSpecName "kube-api-access-85pvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:38:15.780882 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:15.780852    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2eedf056-c257-4f09-847b-1db5cc7a3846-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2eedf056-c257-4f09-847b-1db5cc7a3846" (UID: "2eedf056-c257-4f09-847b-1db5cc7a3846"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:38:15.879847 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:15.879807    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2eedf056-c257-4f09-847b-1db5cc7a3846-kserve-provision-location\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:38:15.879847 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:15.879840    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2eedf056-c257-4f09-847b-1db5cc7a3846-proxy-tls\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:38:15.879847 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:15.879849    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85pvn\" (UniqueName: \"kubernetes.io/projected/2eedf056-c257-4f09-847b-1db5cc7a3846-kube-api-access-85pvn\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:38:15.879847 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:15.879858    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2eedf056-c257-4f09-847b-1db5cc7a3846-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:38:15.880120 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:15.879866    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2eedf056-c257-4f09-847b-1db5cc7a3846-cabundle-cert\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:38:16.063892 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:16.063861    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dskjm"]
Jul 27 18:38:16.067606 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:16.067574    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-dskjm"]
Jul 27 18:38:16.469183 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:16.469147    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2eedf056-c257-4f09-847b-1db5cc7a3846" path="/var/lib/kubelet/pods/2eedf056-c257-4f09-847b-1db5cc7a3846/volumes"
Jul 27 18:38:45.078904 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:45.078871    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:38:50.182331 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:50.182289    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:38:50.204990 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:50.204948    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z7cfn"]
Jul 27 18:38:50.205389 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:50.205372    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2eedf056-c257-4f09-847b-1db5cc7a3846" containerName="storage-initializer"
Jul 27 18:38:50.205489 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:50.205391    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2eedf056-c257-4f09-847b-1db5cc7a3846" containerName="storage-initializer"
Jul 27 18:38:50.205489 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:50.205405    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2eedf056-c257-4f09-847b-1db5cc7a3846" containerName="kube-rbac-proxy"
Jul 27 18:38:50.205489 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:50.205413    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2eedf056-c257-4f09-847b-1db5cc7a3846" containerName="kube-rbac-proxy"
Jul 27 18:38:50.205489 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:50.205437    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2eedf056-c257-4f09-847b-1db5cc7a3846" containerName="kserve-container"
Jul 27 18:38:50.205489 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:50.205446    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2eedf056-c257-4f09-847b-1db5cc7a3846" containerName="kserve-container"
Jul 27 18:38:50.205749 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:50.205537    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2eedf056-c257-4f09-847b-1db5cc7a3846" containerName="kube-rbac-proxy"
Jul 27 18:38:50.205749 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:50.205552    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2eedf056-c257-4f09-847b-1db5cc7a3846" containerName="kserve-container"
Jul 27 18:38:50.210062 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:50.210040    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z7cfn"
Jul 27 18:38:50.213111 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:50.213083    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
Jul 27 18:38:50.213111 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:50.213092    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 18:38:50.213368 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:50.213146    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 18:38:50.213368 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:50.213161    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 18:38:50.213368 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:50.213176    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 18:38:50.213368 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:50.213146    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 27 18:38:50.213368 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:50.213225    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z6ld7\""
Jul 27 18:38:50.218113 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:50.218088    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z7cfn"]
Jul 27 18:38:50.274404 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:50.274370    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e78d681-d9b5-4ab9-90a5-e721ac32670d-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-z7cfn\" (UID: \"2e78d681-d9b5-4ab9-90a5-e721ac32670d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z7cfn"
Jul 27 18:38:50.274578 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:50.274411    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt6c9\" (UniqueName: \"kubernetes.io/projected/2e78d681-d9b5-4ab9-90a5-e721ac32670d-kube-api-access-nt6c9\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-z7cfn\" (UID: \"2e78d681-d9b5-4ab9-90a5-e721ac32670d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z7cfn"
Jul 27 18:38:50.274578 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:50.274442    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e78d681-d9b5-4ab9-90a5-e721ac32670d-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-z7cfn\" (UID: \"2e78d681-d9b5-4ab9-90a5-e721ac32670d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z7cfn"
Jul 27 18:38:50.274578 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:50.274491    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e78d681-d9b5-4ab9-90a5-e721ac32670d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-z7cfn\" (UID: \"2e78d681-d9b5-4ab9-90a5-e721ac32670d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z7cfn"
Jul 27 18:38:50.274578 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:50.274511    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e78d681-d9b5-4ab9-90a5-e721ac32670d-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-z7cfn\" (UID: \"2e78d681-d9b5-4ab9-90a5-e721ac32670d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z7cfn"
Jul 27 18:38:50.375093 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:50.375058    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e78d681-d9b5-4ab9-90a5-e721ac32670d-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-z7cfn\" (UID: \"2e78d681-d9b5-4ab9-90a5-e721ac32670d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z7cfn"
Jul 27 18:38:50.375309 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:50.375116    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e78d681-d9b5-4ab9-90a5-e721ac32670d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-z7cfn\" (UID: \"2e78d681-d9b5-4ab9-90a5-e721ac32670d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z7cfn"
Jul 27 18:38:50.375309 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:50.375151    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e78d681-d9b5-4ab9-90a5-e721ac32670d-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-z7cfn\" (UID: \"2e78d681-d9b5-4ab9-90a5-e721ac32670d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z7cfn"
Jul 27 18:38:50.375309 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:50.375192    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e78d681-d9b5-4ab9-90a5-e721ac32670d-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-z7cfn\" (UID: \"2e78d681-d9b5-4ab9-90a5-e721ac32670d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z7cfn"
Jul 27 18:38:50.375309 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:50.375222    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt6c9\" (UniqueName: \"kubernetes.io/projected/2e78d681-d9b5-4ab9-90a5-e721ac32670d-kube-api-access-nt6c9\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-z7cfn\" (UID: \"2e78d681-d9b5-4ab9-90a5-e721ac32670d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z7cfn"
Jul 27 18:38:50.375553 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:38:50.375363    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-serving-cert: secret "isvc-sklearn-v2-runtime-predictor-serving-cert" not found
Jul 27 18:38:50.375553 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:38:50.375439    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e78d681-d9b5-4ab9-90a5-e721ac32670d-proxy-tls podName:2e78d681-d9b5-4ab9-90a5-e721ac32670d nodeName:}" failed. No retries permitted until 2026-07-27 18:38:50.875423403 +0000 UTC m=+4273.018825435 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2e78d681-d9b5-4ab9-90a5-e721ac32670d-proxy-tls") pod "isvc-sklearn-v2-runtime-predictor-5f69c7d498-z7cfn" (UID: "2e78d681-d9b5-4ab9-90a5-e721ac32670d") : secret "isvc-sklearn-v2-runtime-predictor-serving-cert" not found
Jul 27 18:38:50.375653 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:50.375571    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e78d681-d9b5-4ab9-90a5-e721ac32670d-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-z7cfn\" (UID: \"2e78d681-d9b5-4ab9-90a5-e721ac32670d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z7cfn"
Jul 27 18:38:50.375937 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:50.375915    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e78d681-d9b5-4ab9-90a5-e721ac32670d-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-z7cfn\" (UID: \"2e78d681-d9b5-4ab9-90a5-e721ac32670d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z7cfn"
Jul 27 18:38:50.375979 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:50.375934    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e78d681-d9b5-4ab9-90a5-e721ac32670d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-z7cfn\" (UID: \"2e78d681-d9b5-4ab9-90a5-e721ac32670d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z7cfn"
Jul 27 18:38:50.384212 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:50.384194    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt6c9\" (UniqueName: \"kubernetes.io/projected/2e78d681-d9b5-4ab9-90a5-e721ac32670d-kube-api-access-nt6c9\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-z7cfn\" (UID: \"2e78d681-d9b5-4ab9-90a5-e721ac32670d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z7cfn"
Jul 27 18:38:50.690847 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:50.690812    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:38:50.878339 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:50.878299    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e78d681-d9b5-4ab9-90a5-e721ac32670d-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-z7cfn\" (UID: \"2e78d681-d9b5-4ab9-90a5-e721ac32670d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z7cfn"
Jul 27 18:38:50.880732 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:50.880709    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e78d681-d9b5-4ab9-90a5-e721ac32670d-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-z7cfn\" (UID: \"2e78d681-d9b5-4ab9-90a5-e721ac32670d\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z7cfn"
Jul 27 18:38:51.121787 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:51.121692    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z7cfn"
Jul 27 18:38:51.259588 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:51.259564    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z7cfn"]
Jul 27 18:38:51.262400 ip-10-0-140-0 kubenswrapper[2573]: W0727 18:38:51.262374    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e78d681_d9b5_4ab9_90a5_e721ac32670d.slice/crio-465104aa595bed6c51a2b86dd1faef14f2677abff77b086d4bf986731f9ba145 WatchSource:0}: Error finding container 465104aa595bed6c51a2b86dd1faef14f2677abff77b086d4bf986731f9ba145: Status 404 returned error can't find the container with id 465104aa595bed6c51a2b86dd1faef14f2677abff77b086d4bf986731f9ba145
Jul 27 18:38:51.893174 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:51.893134    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z7cfn" event={"ID":"2e78d681-d9b5-4ab9-90a5-e721ac32670d","Type":"ContainerStarted","Data":"407ef7874b314dfcdc91efef4ac7b309996879c66a7e50bb8a9db841e7c81510"}
Jul 27 18:38:51.893374 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:51.893195    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z7cfn" event={"ID":"2e78d681-d9b5-4ab9-90a5-e721ac32670d","Type":"ContainerStarted","Data":"465104aa595bed6c51a2b86dd1faef14f2677abff77b086d4bf986731f9ba145"}
Jul 27 18:38:54.907432 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:54.907401    2573 generic.go:356] "Generic (PLEG): container finished" podID="2e78d681-d9b5-4ab9-90a5-e721ac32670d" containerID="407ef7874b314dfcdc91efef4ac7b309996879c66a7e50bb8a9db841e7c81510" exitCode=0
Jul 27 18:38:54.907802 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:54.907454    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z7cfn" event={"ID":"2e78d681-d9b5-4ab9-90a5-e721ac32670d","Type":"ContainerDied","Data":"407ef7874b314dfcdc91efef4ac7b309996879c66a7e50bb8a9db841e7c81510"}
Jul 27 18:38:55.913738 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:55.913705    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z7cfn" event={"ID":"2e78d681-d9b5-4ab9-90a5-e721ac32670d","Type":"ContainerStarted","Data":"0236dd0f59f0713cd5eced9555a31dc529c495a8213289722369f95e855e6105"}
Jul 27 18:38:55.914211 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:55.913769    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z7cfn" event={"ID":"2e78d681-d9b5-4ab9-90a5-e721ac32670d","Type":"ContainerStarted","Data":"d1ad09af24eeef236d90ca60b9efa4e22cb674cd5673ade4a55f8852faad5593"}
Jul 27 18:38:55.914211 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:55.913961    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z7cfn"
Jul 27 18:38:55.937552 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:55.937505    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z7cfn" podStartSLOduration=2.43092164 podStartE2EDuration="5.937488805s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.506567165s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:38:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:38:55.935822907 +0000 UTC m=+4278.079224959" watchObservedRunningTime="2026-07-27 18:38:55.937488805 +0000 UTC m=+4278.080890856"
Jul 27 18:38:56.917915 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:38:56.917878    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z7cfn"
Jul 27 18:39:02.927455 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:39:02.927424    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z7cfn"
Jul 27 18:39:32.987922 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:39:32.987818    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z7cfn" podUID="2e78d681-d9b5-4ab9-90a5-e721ac32670d" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Jul 27 18:39:42.929859 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:39:42.929831    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z7cfn"
Jul 27 18:39:48.402583 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:39:48.402547    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:39:50.310599 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:39:50.310568    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z7cfn"]
Jul 27 18:39:50.311088 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:39:50.310858    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z7cfn" podUID="2e78d681-d9b5-4ab9-90a5-e721ac32670d" containerName="kserve-container" containerID="cri-o://d1ad09af24eeef236d90ca60b9efa4e22cb674cd5673ade4a55f8852faad5593" gracePeriod=30
Jul 27 18:39:50.311088 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:39:50.310974    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z7cfn" podUID="2e78d681-d9b5-4ab9-90a5-e721ac32670d" containerName="kube-rbac-proxy" containerID="cri-o://0236dd0f59f0713cd5eced9555a31dc529c495a8213289722369f95e855e6105" gracePeriod=30
Jul 27 18:39:50.334412 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:39:50.334379    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:39:51.153060 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:39:51.153026    2573 generic.go:356] "Generic (PLEG): container finished" podID="2e78d681-d9b5-4ab9-90a5-e721ac32670d" containerID="0236dd0f59f0713cd5eced9555a31dc529c495a8213289722369f95e855e6105" exitCode=2
Jul 27 18:39:51.153236 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:39:51.153087    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z7cfn" event={"ID":"2e78d681-d9b5-4ab9-90a5-e721ac32670d","Type":"ContainerDied","Data":"0236dd0f59f0713cd5eced9555a31dc529c495a8213289722369f95e855e6105"}
Jul 27 18:39:52.922346 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:39:52.922296    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z7cfn" podUID="2e78d681-d9b5-4ab9-90a5-e721ac32670d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.68:8643/healthz\": dial tcp 10.133.0.68:8643: connect: connection refused"
Jul 27 18:39:55.441222 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:39:55.441186    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh"]
Jul 27 18:39:55.444722 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:39:55.444705    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh"
Jul 27 18:39:55.447711 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:39:55.447687    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
Jul 27 18:39:55.447855 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:39:55.447691    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-predictor-serving-cert\""
Jul 27 18:39:55.449163 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:39:55.449138    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:39:55.454530 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:39:55.454506    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh"]
Jul 27 18:39:55.499348 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:39:55.499313    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcbj8\" (UniqueName: \"kubernetes.io/projected/8c40723d-6181-468b-9ea7-f43f464b3667-kube-api-access-pcbj8\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh\" (UID: \"8c40723d-6181-468b-9ea7-f43f464b3667\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh"
Jul 27 18:39:55.499542 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:39:55.499360    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c40723d-6181-468b-9ea7-f43f464b3667-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh\" (UID: \"8c40723d-6181-468b-9ea7-f43f464b3667\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh"
Jul 27 18:39:55.499542 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:39:55.499452    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c40723d-6181-468b-9ea7-f43f464b3667-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh\" (UID: \"8c40723d-6181-468b-9ea7-f43f464b3667\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh"
Jul 27 18:39:55.499542 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:39:55.499493    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c40723d-6181-468b-9ea7-f43f464b3667-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh\" (UID: \"8c40723d-6181-468b-9ea7-f43f464b3667\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh"
Jul 27 18:39:55.499542 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:39:55.499509    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c40723d-6181-468b-9ea7-f43f464b3667-proxy-tls\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh\" (UID: \"8c40723d-6181-468b-9ea7-f43f464b3667\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh"
Jul 27 18:39:55.548403 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:39:55.548368    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:39:55.600498 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:39:55.600466    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c40723d-6181-468b-9ea7-f43f464b3667-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh\" (UID: \"8c40723d-6181-468b-9ea7-f43f464b3667\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh"
Jul 27 18:39:55.600690 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:39:55.600517    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c40723d-6181-468b-9ea7-f43f464b3667-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh\" (UID: \"8c40723d-6181-468b-9ea7-f43f464b3667\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh"
Jul 27 18:39:55.600690 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:39:55.600549    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c40723d-6181-468b-9ea7-f43f464b3667-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh\" (UID: \"8c40723d-6181-468b-9ea7-f43f464b3667\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh"
Jul 27 18:39:55.600690 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:39:55.600569    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c40723d-6181-468b-9ea7-f43f464b3667-proxy-tls\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh\" (UID: \"8c40723d-6181-468b-9ea7-f43f464b3667\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh"
Jul 27 18:39:55.600690 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:39:55.600594    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcbj8\" (UniqueName: \"kubernetes.io/projected/8c40723d-6181-468b-9ea7-f43f464b3667-kube-api-access-pcbj8\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh\" (UID: \"8c40723d-6181-468b-9ea7-f43f464b3667\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh"
Jul 27 18:39:55.600929 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:39:55.600894    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c40723d-6181-468b-9ea7-f43f464b3667-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh\" (UID: \"8c40723d-6181-468b-9ea7-f43f464b3667\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh"
Jul 27 18:39:55.601269 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:39:55.601226    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c40723d-6181-468b-9ea7-f43f464b3667-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh\" (UID: \"8c40723d-6181-468b-9ea7-f43f464b3667\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh"
Jul 27 18:39:55.601367 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:39:55.601292    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c40723d-6181-468b-9ea7-f43f464b3667-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh\" (UID: \"8c40723d-6181-468b-9ea7-f43f464b3667\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh"
Jul 27 18:39:55.603164 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:39:55.603146    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c40723d-6181-468b-9ea7-f43f464b3667-proxy-tls\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh\" (UID: \"8c40723d-6181-468b-9ea7-f43f464b3667\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh"
Jul 27 18:39:55.609732 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:39:55.609705    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcbj8\" (UniqueName: \"kubernetes.io/projected/8c40723d-6181-468b-9ea7-f43f464b3667-kube-api-access-pcbj8\") pod \"isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh\" (UID: \"8c40723d-6181-468b-9ea7-f43f464b3667\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh"
Jul 27 18:39:55.756613 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:39:55.756524    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh"
Jul 27 18:39:55.894379 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:39:55.894352    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh"]
Jul 27 18:39:55.897129 ip-10-0-140-0 kubenswrapper[2573]: W0727 18:39:55.897098    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c40723d_6181_468b_9ea7_f43f464b3667.slice/crio-6d575d124df92e555fec2d5b04dd3fe0e782d16e489a934eb1efb9f3c04dc5c3 WatchSource:0}: Error finding container 6d575d124df92e555fec2d5b04dd3fe0e782d16e489a934eb1efb9f3c04dc5c3: Status 404 returned error can't find the container with id 6d575d124df92e555fec2d5b04dd3fe0e782d16e489a934eb1efb9f3c04dc5c3
Jul 27 18:39:56.177147 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:39:56.177110    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh" event={"ID":"8c40723d-6181-468b-9ea7-f43f464b3667","Type":"ContainerStarted","Data":"f7fa5c38000202dea59359d347772818da6178114bc98e37e2be53e3dcc76b31"}
Jul 27 18:39:56.177362 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:39:56.177182    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh" event={"ID":"8c40723d-6181-468b-9ea7-f43f464b3667","Type":"ContainerStarted","Data":"6d575d124df92e555fec2d5b04dd3fe0e782d16e489a934eb1efb9f3c04dc5c3"}
Jul 27 18:39:57.923320 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:39:57.923270    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z7cfn" podUID="2e78d681-d9b5-4ab9-90a5-e721ac32670d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.68:8643/healthz\": dial tcp 10.133.0.68:8643: connect: connection refused"
Jul 27 18:40:00.197021 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:40:00.196989    2573 generic.go:356] "Generic (PLEG): container finished" podID="8c40723d-6181-468b-9ea7-f43f464b3667" containerID="f7fa5c38000202dea59359d347772818da6178114bc98e37e2be53e3dcc76b31" exitCode=0
Jul 27 18:40:00.197478 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:40:00.197068    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh" event={"ID":"8c40723d-6181-468b-9ea7-f43f464b3667","Type":"ContainerDied","Data":"f7fa5c38000202dea59359d347772818da6178114bc98e37e2be53e3dcc76b31"}
Jul 27 18:40:01.204150 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:40:01.204116    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh" event={"ID":"8c40723d-6181-468b-9ea7-f43f464b3667","Type":"ContainerStarted","Data":"0f00f6ae50357c67822e7622212d22e94c9abd2f57abd3455a11169126709b90"}
Jul 27 18:40:01.204557 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:40:01.204174    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh" event={"ID":"8c40723d-6181-468b-9ea7-f43f464b3667","Type":"ContainerStarted","Data":"6660fb26aef30df35c9aa6e3a619d5ce66491aa03271c58c8f664c901b1fc2d8"}
Jul 27 18:40:01.204557 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:40:01.204522    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh"
Jul 27 18:40:01.204680 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:40:01.204666    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh"
Jul 27 18:40:01.206069 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:40:01.206043    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh" podUID="8c40723d-6181-468b-9ea7-f43f464b3667" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.69:8080: connect: connection refused"
Jul 27 18:40:01.227875 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:40:01.227824    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh" podStartSLOduration=2.733982429 podStartE2EDuration="6.227806626s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.493824197s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:39:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:40:01.226237717 +0000 UTC m=+4343.369639779" watchObservedRunningTime="2026-07-27 18:40:01.227806626 +0000 UTC m=+4343.371208677"
Jul 27 18:40:01.869132 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:40:01.869103    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z7cfn"
Jul 27 18:40:01.945917 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:40:01.945826    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e78d681-d9b5-4ab9-90a5-e721ac32670d-kube-rbac-proxy-sar-config\") pod \"2e78d681-d9b5-4ab9-90a5-e721ac32670d\" (UID: \"2e78d681-d9b5-4ab9-90a5-e721ac32670d\") "
Jul 27 18:40:01.945917 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:40:01.945900    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e78d681-d9b5-4ab9-90a5-e721ac32670d-proxy-tls\") pod \"2e78d681-d9b5-4ab9-90a5-e721ac32670d\" (UID: \"2e78d681-d9b5-4ab9-90a5-e721ac32670d\") "
Jul 27 18:40:01.946129 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:40:01.945932    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nt6c9\" (UniqueName: \"kubernetes.io/projected/2e78d681-d9b5-4ab9-90a5-e721ac32670d-kube-api-access-nt6c9\") pod \"2e78d681-d9b5-4ab9-90a5-e721ac32670d\" (UID: \"2e78d681-d9b5-4ab9-90a5-e721ac32670d\") "
Jul 27 18:40:01.946129 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:40:01.945972    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e78d681-d9b5-4ab9-90a5-e721ac32670d-kserve-provision-location\") pod \"2e78d681-d9b5-4ab9-90a5-e721ac32670d\" (UID: \"2e78d681-d9b5-4ab9-90a5-e721ac32670d\") "
Jul 27 18:40:01.946129 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:40:01.946010    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e78d681-d9b5-4ab9-90a5-e721ac32670d-cabundle-cert\") pod \"2e78d681-d9b5-4ab9-90a5-e721ac32670d\" (UID: \"2e78d681-d9b5-4ab9-90a5-e721ac32670d\") "
Jul 27 18:40:01.946351 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:40:01.946328    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e78d681-d9b5-4ab9-90a5-e721ac32670d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2e78d681-d9b5-4ab9-90a5-e721ac32670d" (UID: "2e78d681-d9b5-4ab9-90a5-e721ac32670d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:40:01.946429 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:40:01.946395    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e78d681-d9b5-4ab9-90a5-e721ac32670d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2e78d681-d9b5-4ab9-90a5-e721ac32670d" (UID: "2e78d681-d9b5-4ab9-90a5-e721ac32670d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:40:01.946532 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:40:01.946508    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e78d681-d9b5-4ab9-90a5-e721ac32670d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2e78d681-d9b5-4ab9-90a5-e721ac32670d" (UID: "2e78d681-d9b5-4ab9-90a5-e721ac32670d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:40:01.948094 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:40:01.948077    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e78d681-d9b5-4ab9-90a5-e721ac32670d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2e78d681-d9b5-4ab9-90a5-e721ac32670d" (UID: "2e78d681-d9b5-4ab9-90a5-e721ac32670d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:40:01.948201 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:40:01.948188    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e78d681-d9b5-4ab9-90a5-e721ac32670d-kube-api-access-nt6c9" (OuterVolumeSpecName: "kube-api-access-nt6c9") pod "2e78d681-d9b5-4ab9-90a5-e721ac32670d" (UID: "2e78d681-d9b5-4ab9-90a5-e721ac32670d"). InnerVolumeSpecName "kube-api-access-nt6c9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:40:02.047554 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:40:02.047497    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e78d681-d9b5-4ab9-90a5-e721ac32670d-kserve-provision-location\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:40:02.047554 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:40:02.047547    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e78d681-d9b5-4ab9-90a5-e721ac32670d-cabundle-cert\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:40:02.047554 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:40:02.047560    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e78d681-d9b5-4ab9-90a5-e721ac32670d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:40:02.047810 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:40:02.047573    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e78d681-d9b5-4ab9-90a5-e721ac32670d-proxy-tls\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:40:02.047810 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:40:02.047585    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt6c9\" (UniqueName: \"kubernetes.io/projected/2e78d681-d9b5-4ab9-90a5-e721ac32670d-kube-api-access-nt6c9\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:40:02.210109 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:40:02.210017    2573 generic.go:356] "Generic (PLEG): container finished" podID="2e78d681-d9b5-4ab9-90a5-e721ac32670d" containerID="d1ad09af24eeef236d90ca60b9efa4e22cb674cd5673ade4a55f8852faad5593" exitCode=0
Jul 27 18:40:02.210109 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:40:02.210068    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z7cfn"
Jul 27 18:40:02.210109 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:40:02.210099    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z7cfn" event={"ID":"2e78d681-d9b5-4ab9-90a5-e721ac32670d","Type":"ContainerDied","Data":"d1ad09af24eeef236d90ca60b9efa4e22cb674cd5673ade4a55f8852faad5593"}
Jul 27 18:40:02.210665 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:40:02.210157    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z7cfn" event={"ID":"2e78d681-d9b5-4ab9-90a5-e721ac32670d","Type":"ContainerDied","Data":"465104aa595bed6c51a2b86dd1faef14f2677abff77b086d4bf986731f9ba145"}
Jul 27 18:40:02.210665 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:40:02.210184    2573 scope.go:117] "RemoveContainer" containerID="0236dd0f59f0713cd5eced9555a31dc529c495a8213289722369f95e855e6105"
Jul 27 18:40:02.210745 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:40:02.210679    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh" podUID="8c40723d-6181-468b-9ea7-f43f464b3667" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.69:8080: connect: connection refused"
Jul 27 18:40:02.219289 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:40:02.219267    2573 scope.go:117] "RemoveContainer" containerID="d1ad09af24eeef236d90ca60b9efa4e22cb674cd5673ade4a55f8852faad5593"
Jul 27 18:40:02.229440 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:40:02.229392    2573 scope.go:117] "RemoveContainer" containerID="407ef7874b314dfcdc91efef4ac7b309996879c66a7e50bb8a9db841e7c81510"
Jul 27 18:40:02.234888 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:40:02.234863    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z7cfn"]
Jul 27 18:40:02.238763 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:40:02.238737    2573 scope.go:117] "RemoveContainer" containerID="0236dd0f59f0713cd5eced9555a31dc529c495a8213289722369f95e855e6105"
Jul 27 18:40:02.238895 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:40:02.238816    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-z7cfn"]
Jul 27 18:40:02.239076 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:40:02.239059    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0236dd0f59f0713cd5eced9555a31dc529c495a8213289722369f95e855e6105\": container with ID starting with 0236dd0f59f0713cd5eced9555a31dc529c495a8213289722369f95e855e6105 not found: ID does not exist" containerID="0236dd0f59f0713cd5eced9555a31dc529c495a8213289722369f95e855e6105"
Jul 27 18:40:02.239130 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:40:02.239084    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0236dd0f59f0713cd5eced9555a31dc529c495a8213289722369f95e855e6105"} err="failed to get container status \"0236dd0f59f0713cd5eced9555a31dc529c495a8213289722369f95e855e6105\": rpc error: code = NotFound desc = could not find container \"0236dd0f59f0713cd5eced9555a31dc529c495a8213289722369f95e855e6105\": container with ID starting with 0236dd0f59f0713cd5eced9555a31dc529c495a8213289722369f95e855e6105 not found: ID does not exist"
Jul 27 18:40:02.239130 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:40:02.239101    2573 scope.go:117] "RemoveContainer" containerID="d1ad09af24eeef236d90ca60b9efa4e22cb674cd5673ade4a55f8852faad5593"
Jul 27 18:40:02.239359 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:40:02.239342    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1ad09af24eeef236d90ca60b9efa4e22cb674cd5673ade4a55f8852faad5593\": container with ID starting with d1ad09af24eeef236d90ca60b9efa4e22cb674cd5673ade4a55f8852faad5593 not found: ID does not exist" containerID="d1ad09af24eeef236d90ca60b9efa4e22cb674cd5673ade4a55f8852faad5593"
Jul 27 18:40:02.239406 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:40:02.239367    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1ad09af24eeef236d90ca60b9efa4e22cb674cd5673ade4a55f8852faad5593"} err="failed to get container status \"d1ad09af24eeef236d90ca60b9efa4e22cb674cd5673ade4a55f8852faad5593\": rpc error: code = NotFound desc = could not find container \"d1ad09af24eeef236d90ca60b9efa4e22cb674cd5673ade4a55f8852faad5593\": container with ID starting with d1ad09af24eeef236d90ca60b9efa4e22cb674cd5673ade4a55f8852faad5593 not found: ID does not exist"
Jul 27 18:40:02.239406 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:40:02.239384    2573 scope.go:117] "RemoveContainer" containerID="407ef7874b314dfcdc91efef4ac7b309996879c66a7e50bb8a9db841e7c81510"
Jul 27 18:40:02.239641 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:40:02.239621    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"407ef7874b314dfcdc91efef4ac7b309996879c66a7e50bb8a9db841e7c81510\": container with ID starting with 407ef7874b314dfcdc91efef4ac7b309996879c66a7e50bb8a9db841e7c81510 not found: ID does not exist" containerID="407ef7874b314dfcdc91efef4ac7b309996879c66a7e50bb8a9db841e7c81510"
Jul 27 18:40:02.239693 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:40:02.239646    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"407ef7874b314dfcdc91efef4ac7b309996879c66a7e50bb8a9db841e7c81510"} err="failed to get container status \"407ef7874b314dfcdc91efef4ac7b309996879c66a7e50bb8a9db841e7c81510\": rpc error: code = NotFound desc = could not find container \"407ef7874b314dfcdc91efef4ac7b309996879c66a7e50bb8a9db841e7c81510\": container with ID starting with 407ef7874b314dfcdc91efef4ac7b309996879c66a7e50bb8a9db841e7c81510 not found: ID does not exist"
Jul 27 18:40:02.473860 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:40:02.470626    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e78d681-d9b5-4ab9-90a5-e721ac32670d" path="/var/lib/kubelet/pods/2e78d681-d9b5-4ab9-90a5-e721ac32670d/volumes"
Jul 27 18:40:07.215507 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:40:07.215478    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh"
Jul 27 18:40:07.215996 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:40:07.215966    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh" podUID="8c40723d-6181-468b-9ea7-f43f464b3667" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.69:8080: connect: connection refused"
Jul 27 18:40:17.216478 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:40:17.216433    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh" podUID="8c40723d-6181-468b-9ea7-f43f464b3667" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.69:8080: connect: connection refused"
Jul 27 18:40:27.216882 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:40:27.216842    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh" podUID="8c40723d-6181-468b-9ea7-f43f464b3667" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.69:8080: connect: connection refused"
Jul 27 18:40:37.216679 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:40:37.216636    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh" podUID="8c40723d-6181-468b-9ea7-f43f464b3667" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.69:8080: connect: connection refused"
Jul 27 18:40:47.216371 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:40:47.216328    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh" podUID="8c40723d-6181-468b-9ea7-f43f464b3667" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.69:8080: connect: connection refused"
Jul 27 18:40:57.215953 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:40:57.215908    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh" podUID="8c40723d-6181-468b-9ea7-f43f464b3667" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.69:8080: connect: connection refused"
Jul 27 18:41:07.216040 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:07.215997    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh" podUID="8c40723d-6181-468b-9ea7-f43f464b3667" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.69:8080: connect: connection refused"
Jul 27 18:41:17.216860 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:17.216825    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh"
Jul 27 18:41:25.666214 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:25.666176    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh"]
Jul 27 18:41:25.666724 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:25.666518    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh" podUID="8c40723d-6181-468b-9ea7-f43f464b3667" containerName="kserve-container" containerID="cri-o://6660fb26aef30df35c9aa6e3a619d5ce66491aa03271c58c8f664c901b1fc2d8" gracePeriod=30
Jul 27 18:41:25.666724 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:25.666535    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh" podUID="8c40723d-6181-468b-9ea7-f43f464b3667" containerName="kube-rbac-proxy" containerID="cri-o://0f00f6ae50357c67822e7622212d22e94c9abd2f57abd3455a11169126709b90" gracePeriod=30
Jul 27 18:41:26.580839 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:26.580805    2573 generic.go:356] "Generic (PLEG): container finished" podID="8c40723d-6181-468b-9ea7-f43f464b3667" containerID="0f00f6ae50357c67822e7622212d22e94c9abd2f57abd3455a11169126709b90" exitCode=2
Jul 27 18:41:26.581010 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:26.580878    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh" event={"ID":"8c40723d-6181-468b-9ea7-f43f464b3667","Type":"ContainerDied","Data":"0f00f6ae50357c67822e7622212d22e94c9abd2f57abd3455a11169126709b90"}
Jul 27 18:41:26.918264 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:26.918229    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:41:27.211822 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:27.211732    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh" podUID="8c40723d-6181-468b-9ea7-f43f464b3667" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.69:8643/healthz\": dial tcp 10.133.0.69:8643: connect: connection refused"
Jul 27 18:41:27.216012 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:27.215991    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh" podUID="8c40723d-6181-468b-9ea7-f43f464b3667" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.69:8080: connect: connection refused"
Jul 27 18:41:30.602701 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:30.602668    2573 generic.go:356] "Generic (PLEG): container finished" podID="8c40723d-6181-468b-9ea7-f43f464b3667" containerID="6660fb26aef30df35c9aa6e3a619d5ce66491aa03271c58c8f664c901b1fc2d8" exitCode=0
Jul 27 18:41:30.603053 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:30.602737    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh" event={"ID":"8c40723d-6181-468b-9ea7-f43f464b3667","Type":"ContainerDied","Data":"6660fb26aef30df35c9aa6e3a619d5ce66491aa03271c58c8f664c901b1fc2d8"}
Jul 27 18:41:30.716171 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:30.716145    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh"
Jul 27 18:41:30.758815 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:30.758737    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:41:30.776525 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:30.776495    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c40723d-6181-468b-9ea7-f43f464b3667-proxy-tls\") pod \"8c40723d-6181-468b-9ea7-f43f464b3667\" (UID: \"8c40723d-6181-468b-9ea7-f43f464b3667\") "
Jul 27 18:41:30.776708 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:30.776543    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c40723d-6181-468b-9ea7-f43f464b3667-cabundle-cert\") pod \"8c40723d-6181-468b-9ea7-f43f464b3667\" (UID: \"8c40723d-6181-468b-9ea7-f43f464b3667\") "
Jul 27 18:41:30.776708 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:30.776584    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c40723d-6181-468b-9ea7-f43f464b3667-kube-rbac-proxy-sar-config\") pod \"8c40723d-6181-468b-9ea7-f43f464b3667\" (UID: \"8c40723d-6181-468b-9ea7-f43f464b3667\") "
Jul 27 18:41:30.776708 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:30.776603    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pcbj8\" (UniqueName: \"kubernetes.io/projected/8c40723d-6181-468b-9ea7-f43f464b3667-kube-api-access-pcbj8\") pod \"8c40723d-6181-468b-9ea7-f43f464b3667\" (UID: \"8c40723d-6181-468b-9ea7-f43f464b3667\") "
Jul 27 18:41:30.776708 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:30.776628    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c40723d-6181-468b-9ea7-f43f464b3667-kserve-provision-location\") pod \"8c40723d-6181-468b-9ea7-f43f464b3667\" (UID: \"8c40723d-6181-468b-9ea7-f43f464b3667\") "
Jul 27 18:41:30.776995 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:30.776970    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c40723d-6181-468b-9ea7-f43f464b3667-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8c40723d-6181-468b-9ea7-f43f464b3667" (UID: "8c40723d-6181-468b-9ea7-f43f464b3667"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:41:30.777054 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:30.776982    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c40723d-6181-468b-9ea7-f43f464b3667-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8c40723d-6181-468b-9ea7-f43f464b3667" (UID: "8c40723d-6181-468b-9ea7-f43f464b3667"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:41:30.777054 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:30.776993    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c40723d-6181-468b-9ea7-f43f464b3667-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8c40723d-6181-468b-9ea7-f43f464b3667" (UID: "8c40723d-6181-468b-9ea7-f43f464b3667"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:41:30.778833 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:30.778807    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c40723d-6181-468b-9ea7-f43f464b3667-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8c40723d-6181-468b-9ea7-f43f464b3667" (UID: "8c40723d-6181-468b-9ea7-f43f464b3667"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:41:30.778943 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:30.778850    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c40723d-6181-468b-9ea7-f43f464b3667-kube-api-access-pcbj8" (OuterVolumeSpecName: "kube-api-access-pcbj8") pod "8c40723d-6181-468b-9ea7-f43f464b3667" (UID: "8c40723d-6181-468b-9ea7-f43f464b3667"). InnerVolumeSpecName "kube-api-access-pcbj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:41:30.803273 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:30.803213    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-474z8"]
Jul 27 18:41:30.803623 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:30.803605    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c40723d-6181-468b-9ea7-f43f464b3667" containerName="kube-rbac-proxy"
Jul 27 18:41:30.803714 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:30.803627    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c40723d-6181-468b-9ea7-f43f464b3667" containerName="kube-rbac-proxy"
Jul 27 18:41:30.803714 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:30.803642    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e78d681-d9b5-4ab9-90a5-e721ac32670d" containerName="kserve-container"
Jul 27 18:41:30.803714 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:30.803650    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e78d681-d9b5-4ab9-90a5-e721ac32670d" containerName="kserve-container"
Jul 27 18:41:30.803714 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:30.803661    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e78d681-d9b5-4ab9-90a5-e721ac32670d" containerName="storage-initializer"
Jul 27 18:41:30.803714 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:30.803669    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e78d681-d9b5-4ab9-90a5-e721ac32670d" containerName="storage-initializer"
Jul 27 18:41:30.803714 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:30.803690    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e78d681-d9b5-4ab9-90a5-e721ac32670d" containerName="kube-rbac-proxy"
Jul 27 18:41:30.803714 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:30.803699    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e78d681-d9b5-4ab9-90a5-e721ac32670d" containerName="kube-rbac-proxy"
Jul 27 18:41:30.803714 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:30.803708    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c40723d-6181-468b-9ea7-f43f464b3667" containerName="storage-initializer"
Jul 27 18:41:30.804118 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:30.803718    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c40723d-6181-468b-9ea7-f43f464b3667" containerName="storage-initializer"
Jul 27 18:41:30.804118 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:30.803731    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c40723d-6181-468b-9ea7-f43f464b3667" containerName="kserve-container"
Jul 27 18:41:30.804118 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:30.803739    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c40723d-6181-468b-9ea7-f43f464b3667" containerName="kserve-container"
Jul 27 18:41:30.804118 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:30.803813    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e78d681-d9b5-4ab9-90a5-e721ac32670d" containerName="kserve-container"
Jul 27 18:41:30.804118 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:30.803827    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c40723d-6181-468b-9ea7-f43f464b3667" containerName="kube-rbac-proxy"
Jul 27 18:41:30.804118 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:30.803839    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e78d681-d9b5-4ab9-90a5-e721ac32670d" containerName="kube-rbac-proxy"
Jul 27 18:41:30.804118 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:30.803850    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c40723d-6181-468b-9ea7-f43f464b3667" containerName="kserve-container"
Jul 27 18:41:30.807223 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:30.807203    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-474z8"
Jul 27 18:41:30.810117 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:30.810097    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
Jul 27 18:41:30.810211 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:30.810098    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
Jul 27 18:41:30.817990 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:30.817968    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-474z8"]
Jul 27 18:41:30.877679 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:30.877648    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd182efd-a4af-4464-87d8-2dc979d1d462-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-474z8\" (UID: \"bd182efd-a4af-4464-87d8-2dc979d1d462\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-474z8"
Jul 27 18:41:30.877679 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:30.877685    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd182efd-a4af-4464-87d8-2dc979d1d462-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-474z8\" (UID: \"bd182efd-a4af-4464-87d8-2dc979d1d462\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-474z8"
Jul 27 18:41:30.877957 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:30.877711    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd182efd-a4af-4464-87d8-2dc979d1d462-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-474z8\" (UID: \"bd182efd-a4af-4464-87d8-2dc979d1d462\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-474z8"
Jul 27 18:41:30.877957 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:30.877864    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd182efd-a4af-4464-87d8-2dc979d1d462-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-474z8\" (UID: \"bd182efd-a4af-4464-87d8-2dc979d1d462\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-474z8"
Jul 27 18:41:30.877957 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:30.877926    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8gnj\" (UniqueName: \"kubernetes.io/projected/bd182efd-a4af-4464-87d8-2dc979d1d462-kube-api-access-r8gnj\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-474z8\" (UID: \"bd182efd-a4af-4464-87d8-2dc979d1d462\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-474z8"
Jul 27 18:41:30.878116 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:30.877990    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c40723d-6181-468b-9ea7-f43f464b3667-proxy-tls\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:41:30.878116 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:30.878006    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c40723d-6181-468b-9ea7-f43f464b3667-cabundle-cert\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:41:30.878116 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:30.878022    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c40723d-6181-468b-9ea7-f43f464b3667-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:41:30.878116 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:30.878035    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcbj8\" (UniqueName: \"kubernetes.io/projected/8c40723d-6181-468b-9ea7-f43f464b3667-kube-api-access-pcbj8\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:41:30.878116 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:30.878048    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c40723d-6181-468b-9ea7-f43f464b3667-kserve-provision-location\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:41:30.978850 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:30.978817    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8gnj\" (UniqueName: \"kubernetes.io/projected/bd182efd-a4af-4464-87d8-2dc979d1d462-kube-api-access-r8gnj\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-474z8\" (UID: \"bd182efd-a4af-4464-87d8-2dc979d1d462\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-474z8"
Jul 27 18:41:30.979042 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:30.978863    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd182efd-a4af-4464-87d8-2dc979d1d462-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-474z8\" (UID: \"bd182efd-a4af-4464-87d8-2dc979d1d462\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-474z8"
Jul 27 18:41:30.979042 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:30.978884    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd182efd-a4af-4464-87d8-2dc979d1d462-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-474z8\" (UID: \"bd182efd-a4af-4464-87d8-2dc979d1d462\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-474z8"
Jul 27 18:41:30.979042 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:41:30.978987    2573 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-serving-cert: secret "isvc-sklearn-v2-mixed-predictor-serving-cert" not found
Jul 27 18:41:30.979042 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:30.979001    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd182efd-a4af-4464-87d8-2dc979d1d462-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-474z8\" (UID: \"bd182efd-a4af-4464-87d8-2dc979d1d462\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-474z8"
Jul 27 18:41:30.979290 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:41:30.979071    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd182efd-a4af-4464-87d8-2dc979d1d462-proxy-tls podName:bd182efd-a4af-4464-87d8-2dc979d1d462 nodeName:}" failed. No retries permitted until 2026-07-27 18:41:31.479050151 +0000 UTC m=+4433.622452182 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bd182efd-a4af-4464-87d8-2dc979d1d462-proxy-tls") pod "isvc-sklearn-v2-mixed-predictor-5746969457-474z8" (UID: "bd182efd-a4af-4464-87d8-2dc979d1d462") : secret "isvc-sklearn-v2-mixed-predictor-serving-cert" not found
Jul 27 18:41:30.979290 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:30.979136    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd182efd-a4af-4464-87d8-2dc979d1d462-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-474z8\" (UID: \"bd182efd-a4af-4464-87d8-2dc979d1d462\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-474z8"
Jul 27 18:41:30.979519 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:30.979497    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd182efd-a4af-4464-87d8-2dc979d1d462-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-474z8\" (UID: \"bd182efd-a4af-4464-87d8-2dc979d1d462\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-474z8"
Jul 27 18:41:30.979575 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:30.979558    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd182efd-a4af-4464-87d8-2dc979d1d462-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-474z8\" (UID: \"bd182efd-a4af-4464-87d8-2dc979d1d462\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-474z8"
Jul 27 18:41:30.979665 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:30.979648    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd182efd-a4af-4464-87d8-2dc979d1d462-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-474z8\" (UID: \"bd182efd-a4af-4464-87d8-2dc979d1d462\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-474z8"
Jul 27 18:41:30.990624 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:30.990600    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8gnj\" (UniqueName: \"kubernetes.io/projected/bd182efd-a4af-4464-87d8-2dc979d1d462-kube-api-access-r8gnj\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-474z8\" (UID: \"bd182efd-a4af-4464-87d8-2dc979d1d462\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-474z8"
Jul 27 18:41:31.131189 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:31.131112    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:41:31.483649 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:31.483610    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd182efd-a4af-4464-87d8-2dc979d1d462-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-474z8\" (UID: \"bd182efd-a4af-4464-87d8-2dc979d1d462\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-474z8"
Jul 27 18:41:31.486029 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:31.486009    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd182efd-a4af-4464-87d8-2dc979d1d462-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-5746969457-474z8\" (UID: \"bd182efd-a4af-4464-87d8-2dc979d1d462\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-474z8"
Jul 27 18:41:31.610096 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:31.610067    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh" event={"ID":"8c40723d-6181-468b-9ea7-f43f464b3667","Type":"ContainerDied","Data":"6d575d124df92e555fec2d5b04dd3fe0e782d16e489a934eb1efb9f3c04dc5c3"}
Jul 27 18:41:31.610580 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:31.610080    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh"
Jul 27 18:41:31.610580 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:31.610135    2573 scope.go:117] "RemoveContainer" containerID="0f00f6ae50357c67822e7622212d22e94c9abd2f57abd3455a11169126709b90"
Jul 27 18:41:31.619261 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:31.619226    2573 scope.go:117] "RemoveContainer" containerID="6660fb26aef30df35c9aa6e3a619d5ce66491aa03271c58c8f664c901b1fc2d8"
Jul 27 18:41:31.626642 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:31.626623    2573 scope.go:117] "RemoveContainer" containerID="f7fa5c38000202dea59359d347772818da6178114bc98e37e2be53e3dcc76b31"
Jul 27 18:41:31.634063 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:31.634032    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh"]
Jul 27 18:41:31.639622 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:31.639599    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-77f4cf5bd-pdgzh"]
Jul 27 18:41:31.717988 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:31.717950    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-474z8"
Jul 27 18:41:31.856202 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:31.856178    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-474z8"]
Jul 27 18:41:31.858839 ip-10-0-140-0 kubenswrapper[2573]: W0727 18:41:31.858810    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd182efd_a4af_4464_87d8_2dc979d1d462.slice/crio-35a7da6d4466a56dae5dbdd3d30c0fe10d18c7d01c7182e71dcd285aab16db36 WatchSource:0}: Error finding container 35a7da6d4466a56dae5dbdd3d30c0fe10d18c7d01c7182e71dcd285aab16db36: Status 404 returned error can't find the container with id 35a7da6d4466a56dae5dbdd3d30c0fe10d18c7d01c7182e71dcd285aab16db36
Jul 27 18:41:32.469999 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:32.469964    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c40723d-6181-468b-9ea7-f43f464b3667" path="/var/lib/kubelet/pods/8c40723d-6181-468b-9ea7-f43f464b3667/volumes"
Jul 27 18:41:32.617992 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:32.617956    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-474z8" event={"ID":"bd182efd-a4af-4464-87d8-2dc979d1d462","Type":"ContainerStarted","Data":"7a719d9f6d6bc691d3f05d7e6e6542b4b5e22d87137670f80052e6063619a4f7"}
Jul 27 18:41:32.618497 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:32.618058    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-474z8" event={"ID":"bd182efd-a4af-4464-87d8-2dc979d1d462","Type":"ContainerStarted","Data":"35a7da6d4466a56dae5dbdd3d30c0fe10d18c7d01c7182e71dcd285aab16db36"}
Jul 27 18:41:35.632430 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:35.632398    2573 generic.go:356] "Generic (PLEG): container finished" podID="bd182efd-a4af-4464-87d8-2dc979d1d462" containerID="7a719d9f6d6bc691d3f05d7e6e6542b4b5e22d87137670f80052e6063619a4f7" exitCode=0
Jul 27 18:41:35.632806 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:35.632471    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-474z8" event={"ID":"bd182efd-a4af-4464-87d8-2dc979d1d462","Type":"ContainerDied","Data":"7a719d9f6d6bc691d3f05d7e6e6542b4b5e22d87137670f80052e6063619a4f7"}
Jul 27 18:41:36.640364 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:36.640322    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-474z8" event={"ID":"bd182efd-a4af-4464-87d8-2dc979d1d462","Type":"ContainerStarted","Data":"9be57b3bac6d6b9d5e8a1e20fde16361c3b13bd0579bffd89fab533aa13f2b4d"}
Jul 27 18:41:36.640790 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:36.640383    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-474z8" event={"ID":"bd182efd-a4af-4464-87d8-2dc979d1d462","Type":"ContainerStarted","Data":"e7b0c1152f86b7bf79c8396e76e31598ef319f249743be9f98760ba9736dfcb8"}
Jul 27 18:41:36.640790 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:36.640660    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-474z8"
Jul 27 18:41:36.640912 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:36.640801    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-474z8"
Jul 27 18:41:36.642000 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:36.641978    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-474z8" podUID="bd182efd-a4af-4464-87d8-2dc979d1d462" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.70:8080: connect: connection refused"
Jul 27 18:41:36.665908 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:36.665866    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-474z8" podStartSLOduration=3.044205519 podStartE2EDuration="6.665852286s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.621646767s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:41:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:41:36.665306182 +0000 UTC m=+4438.808708230" watchObservedRunningTime="2026-07-27 18:41:36.665852286 +0000 UTC m=+4438.809254336"
Jul 27 18:41:37.651027 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:37.650980    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-474z8" podUID="bd182efd-a4af-4464-87d8-2dc979d1d462" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.70:8080: connect: connection refused"
Jul 27 18:41:40.104235 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:40.104194    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:41:42.653589 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:42.653563    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-474z8"
Jul 27 18:41:42.654056 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:42.654032    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-474z8" podUID="bd182efd-a4af-4464-87d8-2dc979d1d462" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.70:8080: connect: connection refused"
Jul 27 18:41:52.654594 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:41:52.654556    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-474z8" podUID="bd182efd-a4af-4464-87d8-2dc979d1d462" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.70:8080: connect: connection refused"
Jul 27 18:42:02.654175 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:42:02.654139    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-474z8" podUID="bd182efd-a4af-4464-87d8-2dc979d1d462" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.70:8080: connect: connection refused"
Jul 27 18:42:06.697021 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:42:06.696978    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:42:08.392603 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:42:08.392566    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:42:12.654720 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:42:12.654681    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-474z8" podUID="bd182efd-a4af-4464-87d8-2dc979d1d462" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.70:8080: connect: connection refused"
Jul 27 18:42:22.654995 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:42:22.654955    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-474z8" podUID="bd182efd-a4af-4464-87d8-2dc979d1d462" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.70:8080: connect: connection refused"
Jul 27 18:42:32.654379 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:42:32.654337    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-474z8" podUID="bd182efd-a4af-4464-87d8-2dc979d1d462" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.70:8080: connect: connection refused"
Jul 27 18:42:38.624001 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:42:38.623971    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vc2tz_87dac797-f8d2-4d36-999f-f8ce855dbeff/ovn-acl-logging/0.log"
Jul 27 18:42:38.633235 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:42:38.633210    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vc2tz_87dac797-f8d2-4d36-999f-f8ce855dbeff/ovn-acl-logging/0.log"
Jul 27 18:42:42.654177 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:42:42.654141    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-474z8" podUID="bd182efd-a4af-4464-87d8-2dc979d1d462" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.70:8080: connect: connection refused"
Jul 27 18:42:52.654412 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:42:52.654382    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-474z8"
Jul 27 18:43:00.965789 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:00.965756    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-474z8"]
Jul 27 18:43:00.966350 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:00.966008    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-474z8" podUID="bd182efd-a4af-4464-87d8-2dc979d1d462" containerName="kserve-container" containerID="cri-o://e7b0c1152f86b7bf79c8396e76e31598ef319f249743be9f98760ba9736dfcb8" gracePeriod=30
Jul 27 18:43:00.966350 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:00.966042    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-474z8" podUID="bd182efd-a4af-4464-87d8-2dc979d1d462" containerName="kube-rbac-proxy" containerID="cri-o://9be57b3bac6d6b9d5e8a1e20fde16361c3b13bd0579bffd89fab533aa13f2b4d" gracePeriod=30
Jul 27 18:43:00.994220 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:00.994187    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:43:02.013002 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:02.012970    2573 generic.go:356] "Generic (PLEG): container finished" podID="bd182efd-a4af-4464-87d8-2dc979d1d462" containerID="9be57b3bac6d6b9d5e8a1e20fde16361c3b13bd0579bffd89fab533aa13f2b4d" exitCode=2
Jul 27 18:43:02.013401 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:02.013018    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-474z8" event={"ID":"bd182efd-a4af-4464-87d8-2dc979d1d462","Type":"ContainerDied","Data":"9be57b3bac6d6b9d5e8a1e20fde16361c3b13bd0579bffd89fab533aa13f2b4d"}
Jul 27 18:43:02.648351 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:02.648309    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-474z8" podUID="bd182efd-a4af-4464-87d8-2dc979d1d462" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.70:8643/healthz\": dial tcp 10.133.0.70:8643: connect: connection refused"
Jul 27 18:43:02.654731 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:02.654697    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-474z8" podUID="bd182efd-a4af-4464-87d8-2dc979d1d462" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.70:8080: connect: connection refused"
Jul 27 18:43:06.025240 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.025212    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-474z8"
Jul 27 18:43:06.036275 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.036214    2573 generic.go:356] "Generic (PLEG): container finished" podID="bd182efd-a4af-4464-87d8-2dc979d1d462" containerID="e7b0c1152f86b7bf79c8396e76e31598ef319f249743be9f98760ba9736dfcb8" exitCode=0
Jul 27 18:43:06.036444 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.036288    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-474z8" event={"ID":"bd182efd-a4af-4464-87d8-2dc979d1d462","Type":"ContainerDied","Data":"e7b0c1152f86b7bf79c8396e76e31598ef319f249743be9f98760ba9736dfcb8"}
Jul 27 18:43:06.036444 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.036302    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-474z8"
Jul 27 18:43:06.036444 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.036384    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-474z8" event={"ID":"bd182efd-a4af-4464-87d8-2dc979d1d462","Type":"ContainerDied","Data":"35a7da6d4466a56dae5dbdd3d30c0fe10d18c7d01c7182e71dcd285aab16db36"}
Jul 27 18:43:06.036444 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.036414    2573 scope.go:117] "RemoveContainer" containerID="9be57b3bac6d6b9d5e8a1e20fde16361c3b13bd0579bffd89fab533aa13f2b4d"
Jul 27 18:43:06.044590 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.044560    2573 scope.go:117] "RemoveContainer" containerID="e7b0c1152f86b7bf79c8396e76e31598ef319f249743be9f98760ba9736dfcb8"
Jul 27 18:43:06.052236 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.052216    2573 scope.go:117] "RemoveContainer" containerID="7a719d9f6d6bc691d3f05d7e6e6542b4b5e22d87137670f80052e6063619a4f7"
Jul 27 18:43:06.059681 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.059661    2573 scope.go:117] "RemoveContainer" containerID="9be57b3bac6d6b9d5e8a1e20fde16361c3b13bd0579bffd89fab533aa13f2b4d"
Jul 27 18:43:06.059944 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:43:06.059924    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9be57b3bac6d6b9d5e8a1e20fde16361c3b13bd0579bffd89fab533aa13f2b4d\": container with ID starting with 9be57b3bac6d6b9d5e8a1e20fde16361c3b13bd0579bffd89fab533aa13f2b4d not found: ID does not exist" containerID="9be57b3bac6d6b9d5e8a1e20fde16361c3b13bd0579bffd89fab533aa13f2b4d"
Jul 27 18:43:06.060015 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.059955    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9be57b3bac6d6b9d5e8a1e20fde16361c3b13bd0579bffd89fab533aa13f2b4d"} err="failed to get container status \"9be57b3bac6d6b9d5e8a1e20fde16361c3b13bd0579bffd89fab533aa13f2b4d\": rpc error: code = NotFound desc = could not find container \"9be57b3bac6d6b9d5e8a1e20fde16361c3b13bd0579bffd89fab533aa13f2b4d\": container with ID starting with 9be57b3bac6d6b9d5e8a1e20fde16361c3b13bd0579bffd89fab533aa13f2b4d not found: ID does not exist"
Jul 27 18:43:06.060015 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.059973    2573 scope.go:117] "RemoveContainer" containerID="e7b0c1152f86b7bf79c8396e76e31598ef319f249743be9f98760ba9736dfcb8"
Jul 27 18:43:06.060209 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:43:06.060191    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7b0c1152f86b7bf79c8396e76e31598ef319f249743be9f98760ba9736dfcb8\": container with ID starting with e7b0c1152f86b7bf79c8396e76e31598ef319f249743be9f98760ba9736dfcb8 not found: ID does not exist" containerID="e7b0c1152f86b7bf79c8396e76e31598ef319f249743be9f98760ba9736dfcb8"
Jul 27 18:43:06.060290 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.060212    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7b0c1152f86b7bf79c8396e76e31598ef319f249743be9f98760ba9736dfcb8"} err="failed to get container status \"e7b0c1152f86b7bf79c8396e76e31598ef319f249743be9f98760ba9736dfcb8\": rpc error: code = NotFound desc = could not find container \"e7b0c1152f86b7bf79c8396e76e31598ef319f249743be9f98760ba9736dfcb8\": container with ID starting with e7b0c1152f86b7bf79c8396e76e31598ef319f249743be9f98760ba9736dfcb8 not found: ID does not exist"
Jul 27 18:43:06.060290 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.060225    2573 scope.go:117] "RemoveContainer" containerID="7a719d9f6d6bc691d3f05d7e6e6542b4b5e22d87137670f80052e6063619a4f7"
Jul 27 18:43:06.060468 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:43:06.060451    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a719d9f6d6bc691d3f05d7e6e6542b4b5e22d87137670f80052e6063619a4f7\": container with ID starting with 7a719d9f6d6bc691d3f05d7e6e6542b4b5e22d87137670f80052e6063619a4f7 not found: ID does not exist" containerID="7a719d9f6d6bc691d3f05d7e6e6542b4b5e22d87137670f80052e6063619a4f7"
Jul 27 18:43:06.060511 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.060477    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a719d9f6d6bc691d3f05d7e6e6542b4b5e22d87137670f80052e6063619a4f7"} err="failed to get container status \"7a719d9f6d6bc691d3f05d7e6e6542b4b5e22d87137670f80052e6063619a4f7\": rpc error: code = NotFound desc = could not find container \"7a719d9f6d6bc691d3f05d7e6e6542b4b5e22d87137670f80052e6063619a4f7\": container with ID starting with 7a719d9f6d6bc691d3f05d7e6e6542b4b5e22d87137670f80052e6063619a4f7 not found: ID does not exist"
Jul 27 18:43:06.074566 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.074538    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd182efd-a4af-4464-87d8-2dc979d1d462-kserve-provision-location\") pod \"bd182efd-a4af-4464-87d8-2dc979d1d462\" (UID: \"bd182efd-a4af-4464-87d8-2dc979d1d462\") "
Jul 27 18:43:06.074753 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.074598    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd182efd-a4af-4464-87d8-2dc979d1d462-cabundle-cert\") pod \"bd182efd-a4af-4464-87d8-2dc979d1d462\" (UID: \"bd182efd-a4af-4464-87d8-2dc979d1d462\") "
Jul 27 18:43:06.074753 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.074707    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd182efd-a4af-4464-87d8-2dc979d1d462-proxy-tls\") pod \"bd182efd-a4af-4464-87d8-2dc979d1d462\" (UID: \"bd182efd-a4af-4464-87d8-2dc979d1d462\") "
Jul 27 18:43:06.074753 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.074744    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r8gnj\" (UniqueName: \"kubernetes.io/projected/bd182efd-a4af-4464-87d8-2dc979d1d462-kube-api-access-r8gnj\") pod \"bd182efd-a4af-4464-87d8-2dc979d1d462\" (UID: \"bd182efd-a4af-4464-87d8-2dc979d1d462\") "
Jul 27 18:43:06.074949 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.074773    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd182efd-a4af-4464-87d8-2dc979d1d462-kube-rbac-proxy-sar-config\") pod \"bd182efd-a4af-4464-87d8-2dc979d1d462\" (UID: \"bd182efd-a4af-4464-87d8-2dc979d1d462\") "
Jul 27 18:43:06.074949 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.074896    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd182efd-a4af-4464-87d8-2dc979d1d462-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bd182efd-a4af-4464-87d8-2dc979d1d462" (UID: "bd182efd-a4af-4464-87d8-2dc979d1d462"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:43:06.075061 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.074991    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd182efd-a4af-4464-87d8-2dc979d1d462-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bd182efd-a4af-4464-87d8-2dc979d1d462" (UID: "bd182efd-a4af-4464-87d8-2dc979d1d462"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:43:06.075061 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.075011    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd182efd-a4af-4464-87d8-2dc979d1d462-kserve-provision-location\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:43:06.075308 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.075237    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd182efd-a4af-4464-87d8-2dc979d1d462-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bd182efd-a4af-4464-87d8-2dc979d1d462" (UID: "bd182efd-a4af-4464-87d8-2dc979d1d462"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:43:06.076973 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.076937    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd182efd-a4af-4464-87d8-2dc979d1d462-kube-api-access-r8gnj" (OuterVolumeSpecName: "kube-api-access-r8gnj") pod "bd182efd-a4af-4464-87d8-2dc979d1d462" (UID: "bd182efd-a4af-4464-87d8-2dc979d1d462"). InnerVolumeSpecName "kube-api-access-r8gnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:43:06.077066 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.077015    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd182efd-a4af-4464-87d8-2dc979d1d462-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bd182efd-a4af-4464-87d8-2dc979d1d462" (UID: "bd182efd-a4af-4464-87d8-2dc979d1d462"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:43:06.141895 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.141860    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:43:06.175940 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.175906    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd182efd-a4af-4464-87d8-2dc979d1d462-proxy-tls\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:43:06.175940 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.175934    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8gnj\" (UniqueName: \"kubernetes.io/projected/bd182efd-a4af-4464-87d8-2dc979d1d462-kube-api-access-r8gnj\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:43:06.175940 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.175945    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd182efd-a4af-4464-87d8-2dc979d1d462-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:43:06.176178 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.175953    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd182efd-a4af-4464-87d8-2dc979d1d462-cabundle-cert\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:43:06.197449 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.197422    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-72229"]
Jul 27 18:43:06.197754 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.197742    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd182efd-a4af-4464-87d8-2dc979d1d462" containerName="storage-initializer"
Jul 27 18:43:06.197802 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.197757    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd182efd-a4af-4464-87d8-2dc979d1d462" containerName="storage-initializer"
Jul 27 18:43:06.197802 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.197779    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd182efd-a4af-4464-87d8-2dc979d1d462" containerName="kube-rbac-proxy"
Jul 27 18:43:06.197802 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.197786    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd182efd-a4af-4464-87d8-2dc979d1d462" containerName="kube-rbac-proxy"
Jul 27 18:43:06.197802 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.197800    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd182efd-a4af-4464-87d8-2dc979d1d462" containerName="kserve-container"
Jul 27 18:43:06.197930 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.197805    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd182efd-a4af-4464-87d8-2dc979d1d462" containerName="kserve-container"
Jul 27 18:43:06.197930 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.197862    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd182efd-a4af-4464-87d8-2dc979d1d462" containerName="kube-rbac-proxy"
Jul 27 18:43:06.197930 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.197871    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd182efd-a4af-4464-87d8-2dc979d1d462" containerName="kserve-container"
Jul 27 18:43:06.201161 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.201135    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-72229"
Jul 27 18:43:06.203967 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.203946    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\""
Jul 27 18:43:06.204051 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.203979    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-predictor-serving-cert\""
Jul 27 18:43:06.210646 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.210618    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-72229"]
Jul 27 18:43:06.277240 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.277201    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc139aca-0b76-4bed-9bbb-9f020ebd6b62-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-72229\" (UID: \"fc139aca-0b76-4bed-9bbb-9f020ebd6b62\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-72229"
Jul 27 18:43:06.277240 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.277236    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5hl2\" (UniqueName: \"kubernetes.io/projected/fc139aca-0b76-4bed-9bbb-9f020ebd6b62-kube-api-access-n5hl2\") pod \"isvc-tensorflow-predictor-547dfc5669-72229\" (UID: \"fc139aca-0b76-4bed-9bbb-9f020ebd6b62\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-72229"
Jul 27 18:43:06.277460 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.277352    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc139aca-0b76-4bed-9bbb-9f020ebd6b62-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-72229\" (UID: \"fc139aca-0b76-4bed-9bbb-9f020ebd6b62\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-72229"
Jul 27 18:43:06.277460 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.277377    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc139aca-0b76-4bed-9bbb-9f020ebd6b62-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-72229\" (UID: \"fc139aca-0b76-4bed-9bbb-9f020ebd6b62\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-72229"
Jul 27 18:43:06.277460 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.277405    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc139aca-0b76-4bed-9bbb-9f020ebd6b62-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-72229\" (UID: \"fc139aca-0b76-4bed-9bbb-9f020ebd6b62\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-72229"
Jul 27 18:43:06.348867 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.348836    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:43:06.359168 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.359145    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-474z8"]
Jul 27 18:43:06.362885 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.362860    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-5746969457-474z8"]
Jul 27 18:43:06.378596 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.378568    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc139aca-0b76-4bed-9bbb-9f020ebd6b62-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-72229\" (UID: \"fc139aca-0b76-4bed-9bbb-9f020ebd6b62\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-72229"
Jul 27 18:43:06.378727 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.378604    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc139aca-0b76-4bed-9bbb-9f020ebd6b62-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-72229\" (UID: \"fc139aca-0b76-4bed-9bbb-9f020ebd6b62\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-72229"
Jul 27 18:43:06.378727 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.378641    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc139aca-0b76-4bed-9bbb-9f020ebd6b62-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-72229\" (UID: \"fc139aca-0b76-4bed-9bbb-9f020ebd6b62\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-72229"
Jul 27 18:43:06.378727 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.378685    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc139aca-0b76-4bed-9bbb-9f020ebd6b62-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-72229\" (UID: \"fc139aca-0b76-4bed-9bbb-9f020ebd6b62\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-72229"
Jul 27 18:43:06.378727 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.378708    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5hl2\" (UniqueName: \"kubernetes.io/projected/fc139aca-0b76-4bed-9bbb-9f020ebd6b62-kube-api-access-n5hl2\") pod \"isvc-tensorflow-predictor-547dfc5669-72229\" (UID: \"fc139aca-0b76-4bed-9bbb-9f020ebd6b62\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-72229"
Jul 27 18:43:06.379057 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.379032    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc139aca-0b76-4bed-9bbb-9f020ebd6b62-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-72229\" (UID: \"fc139aca-0b76-4bed-9bbb-9f020ebd6b62\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-72229"
Jul 27 18:43:06.379514 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.379494    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc139aca-0b76-4bed-9bbb-9f020ebd6b62-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-72229\" (UID: \"fc139aca-0b76-4bed-9bbb-9f020ebd6b62\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-72229"
Jul 27 18:43:06.379587 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.379494    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc139aca-0b76-4bed-9bbb-9f020ebd6b62-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-72229\" (UID: \"fc139aca-0b76-4bed-9bbb-9f020ebd6b62\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-72229"
Jul 27 18:43:06.381031 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.381009    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc139aca-0b76-4bed-9bbb-9f020ebd6b62-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-72229\" (UID: \"fc139aca-0b76-4bed-9bbb-9f020ebd6b62\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-72229"
Jul 27 18:43:06.387767 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.387747    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5hl2\" (UniqueName: \"kubernetes.io/projected/fc139aca-0b76-4bed-9bbb-9f020ebd6b62-kube-api-access-n5hl2\") pod \"isvc-tensorflow-predictor-547dfc5669-72229\" (UID: \"fc139aca-0b76-4bed-9bbb-9f020ebd6b62\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-72229"
Jul 27 18:43:06.470302 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.470232    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd182efd-a4af-4464-87d8-2dc979d1d462" path="/var/lib/kubelet/pods/bd182efd-a4af-4464-87d8-2dc979d1d462/volumes"
Jul 27 18:43:06.511208 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.511166    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-72229"
Jul 27 18:43:06.658405 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:06.658266    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-72229"]
Jul 27 18:43:06.661231 ip-10-0-140-0 kubenswrapper[2573]: W0727 18:43:06.661203    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc139aca_0b76_4bed_9bbb_9f020ebd6b62.slice/crio-f924b0618e987df0e40a0959ff773c7f18763984fa70264f46bf8683147b1b72 WatchSource:0}: Error finding container f924b0618e987df0e40a0959ff773c7f18763984fa70264f46bf8683147b1b72: Status 404 returned error can't find the container with id f924b0618e987df0e40a0959ff773c7f18763984fa70264f46bf8683147b1b72
Jul 27 18:43:07.043074 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:07.043034    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-72229" event={"ID":"fc139aca-0b76-4bed-9bbb-9f020ebd6b62","Type":"ContainerStarted","Data":"0a1a7d9a89ae4a4ab05c56947742e78b20bd8c445e0f278de402fdd42f87a08e"}
Jul 27 18:43:07.043493 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:07.043095    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-72229" event={"ID":"fc139aca-0b76-4bed-9bbb-9f020ebd6b62","Type":"ContainerStarted","Data":"f924b0618e987df0e40a0959ff773c7f18763984fa70264f46bf8683147b1b72"}
Jul 27 18:43:12.066635 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:12.066593    2573 generic.go:356] "Generic (PLEG): container finished" podID="fc139aca-0b76-4bed-9bbb-9f020ebd6b62" containerID="0a1a7d9a89ae4a4ab05c56947742e78b20bd8c445e0f278de402fdd42f87a08e" exitCode=0
Jul 27 18:43:12.067137 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:12.066670    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-72229" event={"ID":"fc139aca-0b76-4bed-9bbb-9f020ebd6b62","Type":"ContainerDied","Data":"0a1a7d9a89ae4a4ab05c56947742e78b20bd8c445e0f278de402fdd42f87a08e"}
Jul 27 18:43:12.068025 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:12.068009    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 18:43:16.092721 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:16.092687    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-72229" event={"ID":"fc139aca-0b76-4bed-9bbb-9f020ebd6b62","Type":"ContainerStarted","Data":"d8e831d52e0d3ea54bedbbb122a9cb79ca1cb795dd135f8cec4c9ca585373a67"}
Jul 27 18:43:17.099117 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:17.099085    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-72229" event={"ID":"fc139aca-0b76-4bed-9bbb-9f020ebd6b62","Type":"ContainerStarted","Data":"86530f3dbe8bd75cce1f38ecce5dab1dea3de9ae4589123cea92104513dadcd7"}
Jul 27 18:43:17.099565 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:17.099191    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-72229"
Jul 27 18:43:17.121293 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:17.121235    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-72229" podStartSLOduration=2.667884831 podStartE2EDuration="11.121221472s" totalImagesPullingTime="3.913953659s" totalInitContainerRuntime="4.539382982s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:43:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:43:17.12015319 +0000 UTC m=+4539.263555240" watchObservedRunningTime="2026-07-27 18:43:17.121221472 +0000 UTC m=+4539.264623523"
Jul 27 18:43:18.103690 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:18.103661    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-72229"
Jul 27 18:43:18.105269 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:18.105222    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-72229" podUID="fc139aca-0b76-4bed-9bbb-9f020ebd6b62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.71:8080: connect: connection refused"
Jul 27 18:43:19.108520 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:19.108474    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-72229" podUID="fc139aca-0b76-4bed-9bbb-9f020ebd6b62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.71:8080: connect: connection refused"
Jul 27 18:43:24.112879 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:24.112852    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-72229"
Jul 27 18:43:24.113386 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:24.113360    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-72229" podUID="fc139aca-0b76-4bed-9bbb-9f020ebd6b62" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.71:8080: connect: connection refused"
Jul 27 18:43:34.113812 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:34.113775    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-72229"
Jul 27 18:43:47.542943 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:47.542910    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-72229"]
Jul 27 18:43:47.543392 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:47.543178    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-72229" podUID="fc139aca-0b76-4bed-9bbb-9f020ebd6b62" containerName="kserve-container" containerID="cri-o://d8e831d52e0d3ea54bedbbb122a9cb79ca1cb795dd135f8cec4c9ca585373a67" gracePeriod=30
Jul 27 18:43:47.543392 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:47.543208    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-72229" podUID="fc139aca-0b76-4bed-9bbb-9f020ebd6b62" containerName="kube-rbac-proxy" containerID="cri-o://86530f3dbe8bd75cce1f38ecce5dab1dea3de9ae4589123cea92104513dadcd7" gracePeriod=30
Jul 27 18:43:47.573304 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:47.573264    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:43:48.231940 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:48.231908    2573 generic.go:356] "Generic (PLEG): container finished" podID="fc139aca-0b76-4bed-9bbb-9f020ebd6b62" containerID="86530f3dbe8bd75cce1f38ecce5dab1dea3de9ae4589123cea92104513dadcd7" exitCode=2
Jul 27 18:43:48.232113 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:48.231978    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-72229" event={"ID":"fc139aca-0b76-4bed-9bbb-9f020ebd6b62","Type":"ContainerDied","Data":"86530f3dbe8bd75cce1f38ecce5dab1dea3de9ae4589123cea92104513dadcd7"}
Jul 27 18:43:49.109312 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:49.109271    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-72229" podUID="fc139aca-0b76-4bed-9bbb-9f020ebd6b62" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.71:8643/healthz\": dial tcp 10.133.0.71:8643: connect: connection refused"
Jul 27 18:43:52.824821 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:52.824781    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:43:52.924014 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:52.923972    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:43:54.109367 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:54.109328    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-72229" podUID="fc139aca-0b76-4bed-9bbb-9f020ebd6b62" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.71:8643/healthz\": dial tcp 10.133.0.71:8643: connect: connection refused"
Jul 27 18:43:59.108743 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:59.108699    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-72229" podUID="fc139aca-0b76-4bed-9bbb-9f020ebd6b62" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.71:8643/healthz\": dial tcp 10.133.0.71:8643: connect: connection refused"
Jul 27 18:43:59.109343 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:43:59.108837    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-72229"
Jul 27 18:44:04.109077 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:04.109035    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-72229" podUID="fc139aca-0b76-4bed-9bbb-9f020ebd6b62" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.71:8643/healthz\": dial tcp 10.133.0.71:8643: connect: connection refused"
Jul 27 18:44:09.109278 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:09.109206    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-72229" podUID="fc139aca-0b76-4bed-9bbb-9f020ebd6b62" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.71:8643/healthz\": dial tcp 10.133.0.71:8643: connect: connection refused"
Jul 27 18:44:14.109593 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:14.109540    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-72229" podUID="fc139aca-0b76-4bed-9bbb-9f020ebd6b62" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.71:8643/healthz\": dial tcp 10.133.0.71:8643: connect: connection refused"
Jul 27 18:44:18.188371 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:18.188349    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-72229"
Jul 27 18:44:18.280946 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:18.280901    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc139aca-0b76-4bed-9bbb-9f020ebd6b62-kube-rbac-proxy-sar-config\") pod \"fc139aca-0b76-4bed-9bbb-9f020ebd6b62\" (UID: \"fc139aca-0b76-4bed-9bbb-9f020ebd6b62\") "
Jul 27 18:44:18.281119 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:18.280963    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc139aca-0b76-4bed-9bbb-9f020ebd6b62-proxy-tls\") pod \"fc139aca-0b76-4bed-9bbb-9f020ebd6b62\" (UID: \"fc139aca-0b76-4bed-9bbb-9f020ebd6b62\") "
Jul 27 18:44:18.281119 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:18.280984    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc139aca-0b76-4bed-9bbb-9f020ebd6b62-cabundle-cert\") pod \"fc139aca-0b76-4bed-9bbb-9f020ebd6b62\" (UID: \"fc139aca-0b76-4bed-9bbb-9f020ebd6b62\") "
Jul 27 18:44:18.281119 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:18.281069    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n5hl2\" (UniqueName: \"kubernetes.io/projected/fc139aca-0b76-4bed-9bbb-9f020ebd6b62-kube-api-access-n5hl2\") pod \"fc139aca-0b76-4bed-9bbb-9f020ebd6b62\" (UID: \"fc139aca-0b76-4bed-9bbb-9f020ebd6b62\") "
Jul 27 18:44:18.281119 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:18.281094    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc139aca-0b76-4bed-9bbb-9f020ebd6b62-kserve-provision-location\") pod \"fc139aca-0b76-4bed-9bbb-9f020ebd6b62\" (UID: \"fc139aca-0b76-4bed-9bbb-9f020ebd6b62\") "
Jul 27 18:44:18.281523 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:18.281498    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc139aca-0b76-4bed-9bbb-9f020ebd6b62-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fc139aca-0b76-4bed-9bbb-9f020ebd6b62" (UID: "fc139aca-0b76-4bed-9bbb-9f020ebd6b62"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:44:18.281611 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:18.281576    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc139aca-0b76-4bed-9bbb-9f020ebd6b62-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fc139aca-0b76-4bed-9bbb-9f020ebd6b62" (UID: "fc139aca-0b76-4bed-9bbb-9f020ebd6b62"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:44:18.283051 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:18.283027    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc139aca-0b76-4bed-9bbb-9f020ebd6b62-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fc139aca-0b76-4bed-9bbb-9f020ebd6b62" (UID: "fc139aca-0b76-4bed-9bbb-9f020ebd6b62"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:44:18.283202 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:18.283178    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc139aca-0b76-4bed-9bbb-9f020ebd6b62-kube-api-access-n5hl2" (OuterVolumeSpecName: "kube-api-access-n5hl2") pod "fc139aca-0b76-4bed-9bbb-9f020ebd6b62" (UID: "fc139aca-0b76-4bed-9bbb-9f020ebd6b62"). InnerVolumeSpecName "kube-api-access-n5hl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:44:18.292358 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:18.292331    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc139aca-0b76-4bed-9bbb-9f020ebd6b62-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fc139aca-0b76-4bed-9bbb-9f020ebd6b62" (UID: "fc139aca-0b76-4bed-9bbb-9f020ebd6b62"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:44:18.361393 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:18.361303    2573 generic.go:356] "Generic (PLEG): container finished" podID="fc139aca-0b76-4bed-9bbb-9f020ebd6b62" containerID="d8e831d52e0d3ea54bedbbb122a9cb79ca1cb795dd135f8cec4c9ca585373a67" exitCode=137
Jul 27 18:44:18.361393 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:18.361359    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-72229"
Jul 27 18:44:18.361393 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:18.361365    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-72229" event={"ID":"fc139aca-0b76-4bed-9bbb-9f020ebd6b62","Type":"ContainerDied","Data":"d8e831d52e0d3ea54bedbbb122a9cb79ca1cb795dd135f8cec4c9ca585373a67"}
Jul 27 18:44:18.361660 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:18.361414    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-72229" event={"ID":"fc139aca-0b76-4bed-9bbb-9f020ebd6b62","Type":"ContainerDied","Data":"f924b0618e987df0e40a0959ff773c7f18763984fa70264f46bf8683147b1b72"}
Jul 27 18:44:18.361660 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:18.361439    2573 scope.go:117] "RemoveContainer" containerID="86530f3dbe8bd75cce1f38ecce5dab1dea3de9ae4589123cea92104513dadcd7"
Jul 27 18:44:18.370904 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:18.370883    2573 scope.go:117] "RemoveContainer" containerID="d8e831d52e0d3ea54bedbbb122a9cb79ca1cb795dd135f8cec4c9ca585373a67"
Jul 27 18:44:18.378585 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:18.378566    2573 scope.go:117] "RemoveContainer" containerID="0a1a7d9a89ae4a4ab05c56947742e78b20bd8c445e0f278de402fdd42f87a08e"
Jul 27 18:44:18.381751 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:18.381733    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc139aca-0b76-4bed-9bbb-9f020ebd6b62-proxy-tls\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:44:18.381840 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:18.381755    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc139aca-0b76-4bed-9bbb-9f020ebd6b62-cabundle-cert\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:44:18.381840 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:18.381768    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5hl2\" (UniqueName: \"kubernetes.io/projected/fc139aca-0b76-4bed-9bbb-9f020ebd6b62-kube-api-access-n5hl2\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:44:18.381840 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:18.381780    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc139aca-0b76-4bed-9bbb-9f020ebd6b62-kserve-provision-location\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:44:18.381840 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:18.381795    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc139aca-0b76-4bed-9bbb-9f020ebd6b62-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:44:18.384846 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:18.384825    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-72229"]
Jul 27 18:44:18.386131 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:18.386116    2573 scope.go:117] "RemoveContainer" containerID="86530f3dbe8bd75cce1f38ecce5dab1dea3de9ae4589123cea92104513dadcd7"
Jul 27 18:44:18.386407 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:44:18.386389    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86530f3dbe8bd75cce1f38ecce5dab1dea3de9ae4589123cea92104513dadcd7\": container with ID starting with 86530f3dbe8bd75cce1f38ecce5dab1dea3de9ae4589123cea92104513dadcd7 not found: ID does not exist" containerID="86530f3dbe8bd75cce1f38ecce5dab1dea3de9ae4589123cea92104513dadcd7"
Jul 27 18:44:18.386449 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:18.386417    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86530f3dbe8bd75cce1f38ecce5dab1dea3de9ae4589123cea92104513dadcd7"} err="failed to get container status \"86530f3dbe8bd75cce1f38ecce5dab1dea3de9ae4589123cea92104513dadcd7\": rpc error: code = NotFound desc = could not find container \"86530f3dbe8bd75cce1f38ecce5dab1dea3de9ae4589123cea92104513dadcd7\": container with ID starting with 86530f3dbe8bd75cce1f38ecce5dab1dea3de9ae4589123cea92104513dadcd7 not found: ID does not exist"
Jul 27 18:44:18.386449 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:18.386440    2573 scope.go:117] "RemoveContainer" containerID="d8e831d52e0d3ea54bedbbb122a9cb79ca1cb795dd135f8cec4c9ca585373a67"
Jul 27 18:44:18.386706 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:44:18.386688    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8e831d52e0d3ea54bedbbb122a9cb79ca1cb795dd135f8cec4c9ca585373a67\": container with ID starting with d8e831d52e0d3ea54bedbbb122a9cb79ca1cb795dd135f8cec4c9ca585373a67 not found: ID does not exist" containerID="d8e831d52e0d3ea54bedbbb122a9cb79ca1cb795dd135f8cec4c9ca585373a67"
Jul 27 18:44:18.386744 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:18.386713    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8e831d52e0d3ea54bedbbb122a9cb79ca1cb795dd135f8cec4c9ca585373a67"} err="failed to get container status \"d8e831d52e0d3ea54bedbbb122a9cb79ca1cb795dd135f8cec4c9ca585373a67\": rpc error: code = NotFound desc = could not find container \"d8e831d52e0d3ea54bedbbb122a9cb79ca1cb795dd135f8cec4c9ca585373a67\": container with ID starting with d8e831d52e0d3ea54bedbbb122a9cb79ca1cb795dd135f8cec4c9ca585373a67 not found: ID does not exist"
Jul 27 18:44:18.386744 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:18.386730    2573 scope.go:117] "RemoveContainer" containerID="0a1a7d9a89ae4a4ab05c56947742e78b20bd8c445e0f278de402fdd42f87a08e"
Jul 27 18:44:18.386975 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:44:18.386960    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a1a7d9a89ae4a4ab05c56947742e78b20bd8c445e0f278de402fdd42f87a08e\": container with ID starting with 0a1a7d9a89ae4a4ab05c56947742e78b20bd8c445e0f278de402fdd42f87a08e not found: ID does not exist" containerID="0a1a7d9a89ae4a4ab05c56947742e78b20bd8c445e0f278de402fdd42f87a08e"
Jul 27 18:44:18.387036 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:18.386981    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a1a7d9a89ae4a4ab05c56947742e78b20bd8c445e0f278de402fdd42f87a08e"} err="failed to get container status \"0a1a7d9a89ae4a4ab05c56947742e78b20bd8c445e0f278de402fdd42f87a08e\": rpc error: code = NotFound desc = could not find container \"0a1a7d9a89ae4a4ab05c56947742e78b20bd8c445e0f278de402fdd42f87a08e\": container with ID starting with 0a1a7d9a89ae4a4ab05c56947742e78b20bd8c445e0f278de402fdd42f87a08e not found: ID does not exist"
Jul 27 18:44:18.389440 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:18.389420    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-72229"]
Jul 27 18:44:18.469239 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:18.469204    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc139aca-0b76-4bed-9bbb-9f020ebd6b62" path="/var/lib/kubelet/pods/fc139aca-0b76-4bed-9bbb-9f020ebd6b62/volumes"
Jul 27 18:44:27.801184 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:27.801094    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:44:32.573671 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:32.573634    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:44:37.685355 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:37.685315    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:44:37.712457 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:37.712412    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2ggpq"]
Jul 27 18:44:37.712750 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:37.712737    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc139aca-0b76-4bed-9bbb-9f020ebd6b62" containerName="storage-initializer"
Jul 27 18:44:37.712795 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:37.712752    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc139aca-0b76-4bed-9bbb-9f020ebd6b62" containerName="storage-initializer"
Jul 27 18:44:37.712795 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:37.712768    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc139aca-0b76-4bed-9bbb-9f020ebd6b62" containerName="kserve-container"
Jul 27 18:44:37.712795 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:37.712774    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc139aca-0b76-4bed-9bbb-9f020ebd6b62" containerName="kserve-container"
Jul 27 18:44:37.712795 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:37.712785    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc139aca-0b76-4bed-9bbb-9f020ebd6b62" containerName="kube-rbac-proxy"
Jul 27 18:44:37.712795 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:37.712790    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc139aca-0b76-4bed-9bbb-9f020ebd6b62" containerName="kube-rbac-proxy"
Jul 27 18:44:37.712952 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:37.712833    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc139aca-0b76-4bed-9bbb-9f020ebd6b62" containerName="kserve-container"
Jul 27 18:44:37.712952 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:37.712844    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc139aca-0b76-4bed-9bbb-9f020ebd6b62" containerName="kube-rbac-proxy"
Jul 27 18:44:37.716959 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:37.716943    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2ggpq"
Jul 27 18:44:37.720052 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:37.720020    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-kube-rbac-proxy-sar-config\""
Jul 27 18:44:37.720052 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:37.720036    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-predictor-serving-cert\""
Jul 27 18:44:37.720284 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:37.720099    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 18:44:37.720284 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:37.720167    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 18:44:37.721477 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:37.721457    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 18:44:37.721477 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:37.721467    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z6ld7\""
Jul 27 18:44:37.721477 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:37.721476    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 18:44:37.725977 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:37.725956    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2ggpq"]
Jul 27 18:44:37.843032 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:37.842987    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b3f8816-7aa9-47d0-b482-f27eccc74496-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-2ggpq\" (UID: \"3b3f8816-7aa9-47d0-b482-f27eccc74496\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2ggpq"
Jul 27 18:44:37.843032 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:37.843036    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55x6z\" (UniqueName: \"kubernetes.io/projected/3b3f8816-7aa9-47d0-b482-f27eccc74496-kube-api-access-55x6z\") pod \"isvc-triton-predictor-5d7db58676-2ggpq\" (UID: \"3b3f8816-7aa9-47d0-b482-f27eccc74496\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2ggpq"
Jul 27 18:44:37.843325 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:37.843075    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b3f8816-7aa9-47d0-b482-f27eccc74496-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-2ggpq\" (UID: \"3b3f8816-7aa9-47d0-b482-f27eccc74496\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2ggpq"
Jul 27 18:44:37.843325 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:37.843101    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b3f8816-7aa9-47d0-b482-f27eccc74496-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-2ggpq\" (UID: \"3b3f8816-7aa9-47d0-b482-f27eccc74496\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2ggpq"
Jul 27 18:44:37.843325 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:37.843191    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b3f8816-7aa9-47d0-b482-f27eccc74496-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-2ggpq\" (UID: \"3b3f8816-7aa9-47d0-b482-f27eccc74496\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2ggpq"
Jul 27 18:44:37.940749 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:37.940660    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:44:37.944303 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:37.944277    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b3f8816-7aa9-47d0-b482-f27eccc74496-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-2ggpq\" (UID: \"3b3f8816-7aa9-47d0-b482-f27eccc74496\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2ggpq"
Jul 27 18:44:37.944443 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:37.944357    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b3f8816-7aa9-47d0-b482-f27eccc74496-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-2ggpq\" (UID: \"3b3f8816-7aa9-47d0-b482-f27eccc74496\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2ggpq"
Jul 27 18:44:37.944443 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:37.944388    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55x6z\" (UniqueName: \"kubernetes.io/projected/3b3f8816-7aa9-47d0-b482-f27eccc74496-kube-api-access-55x6z\") pod \"isvc-triton-predictor-5d7db58676-2ggpq\" (UID: \"3b3f8816-7aa9-47d0-b482-f27eccc74496\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2ggpq"
Jul 27 18:44:37.944443 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:37.944428    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b3f8816-7aa9-47d0-b482-f27eccc74496-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-2ggpq\" (UID: \"3b3f8816-7aa9-47d0-b482-f27eccc74496\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2ggpq"
Jul 27 18:44:37.944577 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:37.944453    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b3f8816-7aa9-47d0-b482-f27eccc74496-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-2ggpq\" (UID: \"3b3f8816-7aa9-47d0-b482-f27eccc74496\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2ggpq"
Jul 27 18:44:37.944782 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:37.944761    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b3f8816-7aa9-47d0-b482-f27eccc74496-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-2ggpq\" (UID: \"3b3f8816-7aa9-47d0-b482-f27eccc74496\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2ggpq"
Jul 27 18:44:37.945094 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:37.945075    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b3f8816-7aa9-47d0-b482-f27eccc74496-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-2ggpq\" (UID: \"3b3f8816-7aa9-47d0-b482-f27eccc74496\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2ggpq"
Jul 27 18:44:37.945157 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:37.945100    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b3f8816-7aa9-47d0-b482-f27eccc74496-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-2ggpq\" (UID: \"3b3f8816-7aa9-47d0-b482-f27eccc74496\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2ggpq"
Jul 27 18:44:37.946733 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:37.946710    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b3f8816-7aa9-47d0-b482-f27eccc74496-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-2ggpq\" (UID: \"3b3f8816-7aa9-47d0-b482-f27eccc74496\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2ggpq"
Jul 27 18:44:37.953889 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:37.953866    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55x6z\" (UniqueName: \"kubernetes.io/projected/3b3f8816-7aa9-47d0-b482-f27eccc74496-kube-api-access-55x6z\") pod \"isvc-triton-predictor-5d7db58676-2ggpq\" (UID: \"3b3f8816-7aa9-47d0-b482-f27eccc74496\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2ggpq"
Jul 27 18:44:38.028498 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:38.028462    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2ggpq"
Jul 27 18:44:38.161458 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:38.161432    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2ggpq"]
Jul 27 18:44:38.163521 ip-10-0-140-0 kubenswrapper[2573]: W0727 18:44:38.163495    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b3f8816_7aa9_47d0_b482_f27eccc74496.slice/crio-88ba1b2d0369464354edcbce087338a5075db479fbf32ae9f9dbf05b4c46ab81 WatchSource:0}: Error finding container 88ba1b2d0369464354edcbce087338a5075db479fbf32ae9f9dbf05b4c46ab81: Status 404 returned error can't find the container with id 88ba1b2d0369464354edcbce087338a5075db479fbf32ae9f9dbf05b4c46ab81
Jul 27 18:44:38.447044 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:38.446956    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2ggpq" event={"ID":"3b3f8816-7aa9-47d0-b482-f27eccc74496","Type":"ContainerStarted","Data":"c8daee0a292e4e8314e737ac7d8cf153ffca9fe2ed60b70c5570f5e63571fcdf"}
Jul 27 18:44:38.447044 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:38.447033    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2ggpq" event={"ID":"3b3f8816-7aa9-47d0-b482-f27eccc74496","Type":"ContainerStarted","Data":"88ba1b2d0369464354edcbce087338a5075db479fbf32ae9f9dbf05b4c46ab81"}
Jul 27 18:44:42.466015 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:42.465979    2573 generic.go:356] "Generic (PLEG): container finished" podID="3b3f8816-7aa9-47d0-b482-f27eccc74496" containerID="c8daee0a292e4e8314e737ac7d8cf153ffca9fe2ed60b70c5570f5e63571fcdf" exitCode=0
Jul 27 18:44:42.469543 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:44:42.469507    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2ggpq" event={"ID":"3b3f8816-7aa9-47d0-b482-f27eccc74496","Type":"ContainerDied","Data":"c8daee0a292e4e8314e737ac7d8cf153ffca9fe2ed60b70c5570f5e63571fcdf"}
Jul 27 18:46:37.099849 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:46:37.099812    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2ggpq" event={"ID":"3b3f8816-7aa9-47d0-b482-f27eccc74496","Type":"ContainerStarted","Data":"78d258a2fd20bc257e6ec5c71a2ed5aaa1e1f489265cbb3e9120de26523c2dca"}
Jul 27 18:46:37.100365 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:46:37.099878    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2ggpq" event={"ID":"3b3f8816-7aa9-47d0-b482-f27eccc74496","Type":"ContainerStarted","Data":"261ff9e66c312a853c05ec64804c4308795634548e97585df7c0af46d38e4aee"}
Jul 27 18:46:37.100365 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:46:37.100018    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2ggpq"
Jul 27 18:46:37.128961 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:46:37.128913    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2ggpq" podStartSLOduration=2.175715072 podStartE2EDuration="2m0.128897572s" totalImagesPullingTime="1m54.220620685s" totalInitContainerRuntime="3.732561815s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:44:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:46:37.127537793 +0000 UTC m=+4739.270939847" watchObservedRunningTime="2026-07-27 18:46:37.128897572 +0000 UTC m=+4739.272299622"
Jul 27 18:46:38.104830 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:46:38.104797    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2ggpq"
Jul 27 18:46:38.106130 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:46:38.106106    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2ggpq" podUID="3b3f8816-7aa9-47d0-b482-f27eccc74496" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.72:8080: connect: connection refused"
Jul 27 18:46:39.109150 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:46:39.109107    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2ggpq" podUID="3b3f8816-7aa9-47d0-b482-f27eccc74496" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.72:8080: connect: connection refused"
Jul 27 18:46:44.113577 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:46:44.113547    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2ggpq"
Jul 27 18:46:44.114344 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:46:44.114324    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2ggpq"
Jul 27 18:46:49.285440 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:46:49.285411    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2ggpq"]
Jul 27 18:46:49.285883 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:46:49.285655    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2ggpq" podUID="3b3f8816-7aa9-47d0-b482-f27eccc74496" containerName="kserve-container" containerID="cri-o://261ff9e66c312a853c05ec64804c4308795634548e97585df7c0af46d38e4aee" gracePeriod=30
Jul 27 18:46:49.285883 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:46:49.285684    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2ggpq" podUID="3b3f8816-7aa9-47d0-b482-f27eccc74496" containerName="kube-rbac-proxy" containerID="cri-o://78d258a2fd20bc257e6ec5c71a2ed5aaa1e1f489265cbb3e9120de26523c2dca" gracePeriod=30
Jul 27 18:46:49.332426 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:46:49.332399    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:46:50.157353 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:46:50.157317    2573 generic.go:356] "Generic (PLEG): container finished" podID="3b3f8816-7aa9-47d0-b482-f27eccc74496" containerID="78d258a2fd20bc257e6ec5c71a2ed5aaa1e1f489265cbb3e9120de26523c2dca" exitCode=2
Jul 27 18:46:50.157652 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:46:50.157392    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2ggpq" event={"ID":"3b3f8816-7aa9-47d0-b482-f27eccc74496","Type":"ContainerDied","Data":"78d258a2fd20bc257e6ec5c71a2ed5aaa1e1f489265cbb3e9120de26523c2dca"}
Jul 27 18:46:51.766021 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:46:51.765942    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2ggpq"
Jul 27 18:46:51.854794 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:46:51.854755    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-55x6z\" (UniqueName: \"kubernetes.io/projected/3b3f8816-7aa9-47d0-b482-f27eccc74496-kube-api-access-55x6z\") pod \"3b3f8816-7aa9-47d0-b482-f27eccc74496\" (UID: \"3b3f8816-7aa9-47d0-b482-f27eccc74496\") "
Jul 27 18:46:51.854794 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:46:51.854796    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b3f8816-7aa9-47d0-b482-f27eccc74496-kube-rbac-proxy-sar-config\") pod \"3b3f8816-7aa9-47d0-b482-f27eccc74496\" (UID: \"3b3f8816-7aa9-47d0-b482-f27eccc74496\") "
Jul 27 18:46:51.855082 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:46:51.854861    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b3f8816-7aa9-47d0-b482-f27eccc74496-proxy-tls\") pod \"3b3f8816-7aa9-47d0-b482-f27eccc74496\" (UID: \"3b3f8816-7aa9-47d0-b482-f27eccc74496\") "
Jul 27 18:46:51.855082 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:46:51.854923    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b3f8816-7aa9-47d0-b482-f27eccc74496-kserve-provision-location\") pod \"3b3f8816-7aa9-47d0-b482-f27eccc74496\" (UID: \"3b3f8816-7aa9-47d0-b482-f27eccc74496\") "
Jul 27 18:46:51.855082 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:46:51.854945    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b3f8816-7aa9-47d0-b482-f27eccc74496-cabundle-cert\") pod \"3b3f8816-7aa9-47d0-b482-f27eccc74496\" (UID: \"3b3f8816-7aa9-47d0-b482-f27eccc74496\") "
Jul 27 18:46:51.855322 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:46:51.855289    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b3f8816-7aa9-47d0-b482-f27eccc74496-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3b3f8816-7aa9-47d0-b482-f27eccc74496" (UID: "3b3f8816-7aa9-47d0-b482-f27eccc74496"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:46:51.855460 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:46:51.855392    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b3f8816-7aa9-47d0-b482-f27eccc74496-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3b3f8816-7aa9-47d0-b482-f27eccc74496" (UID: "3b3f8816-7aa9-47d0-b482-f27eccc74496"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:46:51.855460 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:46:51.855402    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b3f8816-7aa9-47d0-b482-f27eccc74496-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3b3f8816-7aa9-47d0-b482-f27eccc74496" (UID: "3b3f8816-7aa9-47d0-b482-f27eccc74496"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:46:51.857036 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:46:51.857005    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b3f8816-7aa9-47d0-b482-f27eccc74496-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3b3f8816-7aa9-47d0-b482-f27eccc74496" (UID: "3b3f8816-7aa9-47d0-b482-f27eccc74496"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:46:51.857143 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:46:51.857062    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b3f8816-7aa9-47d0-b482-f27eccc74496-kube-api-access-55x6z" (OuterVolumeSpecName: "kube-api-access-55x6z") pod "3b3f8816-7aa9-47d0-b482-f27eccc74496" (UID: "3b3f8816-7aa9-47d0-b482-f27eccc74496"). InnerVolumeSpecName "kube-api-access-55x6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:46:51.955593 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:46:51.955556    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55x6z\" (UniqueName: \"kubernetes.io/projected/3b3f8816-7aa9-47d0-b482-f27eccc74496-kube-api-access-55x6z\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:46:51.955593 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:46:51.955586    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b3f8816-7aa9-47d0-b482-f27eccc74496-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:46:51.955593 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:46:51.955596    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b3f8816-7aa9-47d0-b482-f27eccc74496-proxy-tls\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:46:51.955903 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:46:51.955606    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b3f8816-7aa9-47d0-b482-f27eccc74496-kserve-provision-location\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:46:51.955903 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:46:51.955615    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b3f8816-7aa9-47d0-b482-f27eccc74496-cabundle-cert\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:46:52.167803 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:46:52.167766    2573 generic.go:356] "Generic (PLEG): container finished" podID="3b3f8816-7aa9-47d0-b482-f27eccc74496" containerID="261ff9e66c312a853c05ec64804c4308795634548e97585df7c0af46d38e4aee" exitCode=0
Jul 27 18:46:52.168101 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:46:52.167833    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2ggpq"
Jul 27 18:46:52.168101 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:46:52.167858    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2ggpq" event={"ID":"3b3f8816-7aa9-47d0-b482-f27eccc74496","Type":"ContainerDied","Data":"261ff9e66c312a853c05ec64804c4308795634548e97585df7c0af46d38e4aee"}
Jul 27 18:46:52.168101 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:46:52.167918    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2ggpq" event={"ID":"3b3f8816-7aa9-47d0-b482-f27eccc74496","Type":"ContainerDied","Data":"88ba1b2d0369464354edcbce087338a5075db479fbf32ae9f9dbf05b4c46ab81"}
Jul 27 18:46:52.168101 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:46:52.167946    2573 scope.go:117] "RemoveContainer" containerID="78d258a2fd20bc257e6ec5c71a2ed5aaa1e1f489265cbb3e9120de26523c2dca"
Jul 27 18:46:52.176373 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:46:52.176356    2573 scope.go:117] "RemoveContainer" containerID="261ff9e66c312a853c05ec64804c4308795634548e97585df7c0af46d38e4aee"
Jul 27 18:46:52.184359 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:46:52.184316    2573 scope.go:117] "RemoveContainer" containerID="c8daee0a292e4e8314e737ac7d8cf153ffca9fe2ed60b70c5570f5e63571fcdf"
Jul 27 18:46:52.191196 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:46:52.191170    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2ggpq"]
Jul 27 18:46:52.192741 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:46:52.192721    2573 scope.go:117] "RemoveContainer" containerID="78d258a2fd20bc257e6ec5c71a2ed5aaa1e1f489265cbb3e9120de26523c2dca"
Jul 27 18:46:52.193042 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:46:52.193015    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78d258a2fd20bc257e6ec5c71a2ed5aaa1e1f489265cbb3e9120de26523c2dca\": container with ID starting with 78d258a2fd20bc257e6ec5c71a2ed5aaa1e1f489265cbb3e9120de26523c2dca not found: ID does not exist" containerID="78d258a2fd20bc257e6ec5c71a2ed5aaa1e1f489265cbb3e9120de26523c2dca"
Jul 27 18:46:52.193180 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:46:52.193047    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78d258a2fd20bc257e6ec5c71a2ed5aaa1e1f489265cbb3e9120de26523c2dca"} err="failed to get container status \"78d258a2fd20bc257e6ec5c71a2ed5aaa1e1f489265cbb3e9120de26523c2dca\": rpc error: code = NotFound desc = could not find container \"78d258a2fd20bc257e6ec5c71a2ed5aaa1e1f489265cbb3e9120de26523c2dca\": container with ID starting with 78d258a2fd20bc257e6ec5c71a2ed5aaa1e1f489265cbb3e9120de26523c2dca not found: ID does not exist"
Jul 27 18:46:52.193180 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:46:52.193071    2573 scope.go:117] "RemoveContainer" containerID="261ff9e66c312a853c05ec64804c4308795634548e97585df7c0af46d38e4aee"
Jul 27 18:46:52.193452 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:46:52.193425    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"261ff9e66c312a853c05ec64804c4308795634548e97585df7c0af46d38e4aee\": container with ID starting with 261ff9e66c312a853c05ec64804c4308795634548e97585df7c0af46d38e4aee not found: ID does not exist" containerID="261ff9e66c312a853c05ec64804c4308795634548e97585df7c0af46d38e4aee"
Jul 27 18:46:52.193573 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:46:52.193458    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"261ff9e66c312a853c05ec64804c4308795634548e97585df7c0af46d38e4aee"} err="failed to get container status \"261ff9e66c312a853c05ec64804c4308795634548e97585df7c0af46d38e4aee\": rpc error: code = NotFound desc = could not find container \"261ff9e66c312a853c05ec64804c4308795634548e97585df7c0af46d38e4aee\": container with ID starting with 261ff9e66c312a853c05ec64804c4308795634548e97585df7c0af46d38e4aee not found: ID does not exist"
Jul 27 18:46:52.193573 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:46:52.193479    2573 scope.go:117] "RemoveContainer" containerID="c8daee0a292e4e8314e737ac7d8cf153ffca9fe2ed60b70c5570f5e63571fcdf"
Jul 27 18:46:52.193975 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:46:52.193952    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8daee0a292e4e8314e737ac7d8cf153ffca9fe2ed60b70c5570f5e63571fcdf\": container with ID starting with c8daee0a292e4e8314e737ac7d8cf153ffca9fe2ed60b70c5570f5e63571fcdf not found: ID does not exist" containerID="c8daee0a292e4e8314e737ac7d8cf153ffca9fe2ed60b70c5570f5e63571fcdf"
Jul 27 18:46:52.194053 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:46:52.193984    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8daee0a292e4e8314e737ac7d8cf153ffca9fe2ed60b70c5570f5e63571fcdf"} err="failed to get container status \"c8daee0a292e4e8314e737ac7d8cf153ffca9fe2ed60b70c5570f5e63571fcdf\": rpc error: code = NotFound desc = could not find container \"c8daee0a292e4e8314e737ac7d8cf153ffca9fe2ed60b70c5570f5e63571fcdf\": container with ID starting with c8daee0a292e4e8314e737ac7d8cf153ffca9fe2ed60b70c5570f5e63571fcdf not found: ID does not exist"
Jul 27 18:46:52.195553 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:46:52.195535    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-2ggpq"]
Jul 27 18:46:52.468546 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:46:52.468471    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b3f8816-7aa9-47d0-b482-f27eccc74496" path="/var/lib/kubelet/pods/3b3f8816-7aa9-47d0-b482-f27eccc74496/volumes"
Jul 27 18:46:54.415428 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:46:54.415389    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:46:54.888305 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:46:54.888268    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:47:38.649032 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:47:38.649006    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vc2tz_87dac797-f8d2-4d36-999f-f8ce855dbeff/ovn-acl-logging/0.log"
Jul 27 18:47:38.658412 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:47:38.658390    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vc2tz_87dac797-f8d2-4d36-999f-f8ce855dbeff/ovn-acl-logging/0.log"
Jul 27 18:48:24.605402 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:48:24.605369    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:48:26.456568 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:48:26.456534    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:48:29.671032 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:48:29.670960    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:48:29.733893 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:48:29.733861    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tmb4x"]
Jul 27 18:48:29.734222 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:48:29.734209    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b3f8816-7aa9-47d0-b482-f27eccc74496" containerName="storage-initializer"
Jul 27 18:48:29.734294 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:48:29.734225    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b3f8816-7aa9-47d0-b482-f27eccc74496" containerName="storage-initializer"
Jul 27 18:48:29.734294 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:48:29.734243    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b3f8816-7aa9-47d0-b482-f27eccc74496" containerName="kserve-container"
Jul 27 18:48:29.734294 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:48:29.734262    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b3f8816-7aa9-47d0-b482-f27eccc74496" containerName="kserve-container"
Jul 27 18:48:29.734294 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:48:29.734273    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b3f8816-7aa9-47d0-b482-f27eccc74496" containerName="kube-rbac-proxy"
Jul 27 18:48:29.734294 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:48:29.734280    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b3f8816-7aa9-47d0-b482-f27eccc74496" containerName="kube-rbac-proxy"
Jul 27 18:48:29.734456 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:48:29.734328    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b3f8816-7aa9-47d0-b482-f27eccc74496" containerName="kube-rbac-proxy"
Jul 27 18:48:29.734456 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:48:29.734338    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b3f8816-7aa9-47d0-b482-f27eccc74496" containerName="kserve-container"
Jul 27 18:48:29.736464 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:48:29.736446    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tmb4x"
Jul 27 18:48:29.739339 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:48:29.739306    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z6ld7\""
Jul 27 18:48:29.739613 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:48:29.739590    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
Jul 27 18:48:29.739687 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:48:29.739615    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 18:48:29.740797 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:48:29.740773    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 18:48:29.741073 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:48:29.741054    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Jul 27 18:48:29.741448 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:48:29.741429    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 18:48:29.742190 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:48:29.742160    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 18:48:29.747312 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:48:29.747287    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tmb4x"]
Jul 27 18:48:29.843191 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:48:29.843148    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll7gh\" (UniqueName: \"kubernetes.io/projected/f70606b9-60d2-494f-bd4c-7d5bb09ecddf-kube-api-access-ll7gh\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tmb4x\" (UID: \"f70606b9-60d2-494f-bd4c-7d5bb09ecddf\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tmb4x"
Jul 27 18:48:29.843401 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:48:29.843208    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f70606b9-60d2-494f-bd4c-7d5bb09ecddf-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tmb4x\" (UID: \"f70606b9-60d2-494f-bd4c-7d5bb09ecddf\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tmb4x"
Jul 27 18:48:29.843401 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:48:29.843295    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f70606b9-60d2-494f-bd4c-7d5bb09ecddf-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tmb4x\" (UID: \"f70606b9-60d2-494f-bd4c-7d5bb09ecddf\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tmb4x"
Jul 27 18:48:29.843401 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:48:29.843356    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f70606b9-60d2-494f-bd4c-7d5bb09ecddf-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tmb4x\" (UID: \"f70606b9-60d2-494f-bd4c-7d5bb09ecddf\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tmb4x"
Jul 27 18:48:29.843531 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:48:29.843400    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f70606b9-60d2-494f-bd4c-7d5bb09ecddf-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tmb4x\" (UID: \"f70606b9-60d2-494f-bd4c-7d5bb09ecddf\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tmb4x"
Jul 27 18:48:29.853915 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:48:29.853882    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:48:29.944225 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:48:29.944136    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f70606b9-60d2-494f-bd4c-7d5bb09ecddf-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tmb4x\" (UID: \"f70606b9-60d2-494f-bd4c-7d5bb09ecddf\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tmb4x"
Jul 27 18:48:29.944225 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:48:29.944185    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f70606b9-60d2-494f-bd4c-7d5bb09ecddf-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tmb4x\" (UID: \"f70606b9-60d2-494f-bd4c-7d5bb09ecddf\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tmb4x"
Jul 27 18:48:29.944469 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:48:29.944372    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll7gh\" (UniqueName: \"kubernetes.io/projected/f70606b9-60d2-494f-bd4c-7d5bb09ecddf-kube-api-access-ll7gh\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tmb4x\" (UID: \"f70606b9-60d2-494f-bd4c-7d5bb09ecddf\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tmb4x"
Jul 27 18:48:29.944469 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:48:29.944433    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f70606b9-60d2-494f-bd4c-7d5bb09ecddf-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tmb4x\" (UID: \"f70606b9-60d2-494f-bd4c-7d5bb09ecddf\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tmb4x"
Jul 27 18:48:29.944469 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:48:29.944464    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f70606b9-60d2-494f-bd4c-7d5bb09ecddf-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tmb4x\" (UID: \"f70606b9-60d2-494f-bd4c-7d5bb09ecddf\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tmb4x"
Jul 27 18:48:29.944622 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:48:29.944557    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f70606b9-60d2-494f-bd4c-7d5bb09ecddf-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tmb4x\" (UID: \"f70606b9-60d2-494f-bd4c-7d5bb09ecddf\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tmb4x"
Jul 27 18:48:29.945100 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:48:29.945081    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f70606b9-60d2-494f-bd4c-7d5bb09ecddf-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tmb4x\" (UID: \"f70606b9-60d2-494f-bd4c-7d5bb09ecddf\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tmb4x"
Jul 27 18:48:29.945170 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:48:29.945112    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f70606b9-60d2-494f-bd4c-7d5bb09ecddf-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tmb4x\" (UID: \"f70606b9-60d2-494f-bd4c-7d5bb09ecddf\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tmb4x"
Jul 27 18:48:29.946666 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:48:29.946646    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f70606b9-60d2-494f-bd4c-7d5bb09ecddf-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tmb4x\" (UID: \"f70606b9-60d2-494f-bd4c-7d5bb09ecddf\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tmb4x"
Jul 27 18:48:29.953011 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:48:29.952989    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll7gh\" (UniqueName: \"kubernetes.io/projected/f70606b9-60d2-494f-bd4c-7d5bb09ecddf-kube-api-access-ll7gh\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tmb4x\" (UID: \"f70606b9-60d2-494f-bd4c-7d5bb09ecddf\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tmb4x"
Jul 27 18:48:30.053449 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:48:30.053405    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tmb4x"
Jul 27 18:48:30.222285 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:48:30.222240    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tmb4x"]
Jul 27 18:48:30.225126 ip-10-0-140-0 kubenswrapper[2573]: W0727 18:48:30.225096    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf70606b9_60d2_494f_bd4c_7d5bb09ecddf.slice/crio-ca273b1c825b28009d71abd31bf4b92c367f6334773e3f3f3087832919df8ad4 WatchSource:0}: Error finding container ca273b1c825b28009d71abd31bf4b92c367f6334773e3f3f3087832919df8ad4: Status 404 returned error can't find the container with id ca273b1c825b28009d71abd31bf4b92c367f6334773e3f3f3087832919df8ad4
Jul 27 18:48:30.578032 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:48:30.577943    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tmb4x" event={"ID":"f70606b9-60d2-494f-bd4c-7d5bb09ecddf","Type":"ContainerStarted","Data":"1a0433f70ae695cdc092c9cfb0cf3fb2c4dab6555744bc1cc7273728cf2bae8c"}
Jul 27 18:48:30.578032 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:48:30.578012    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tmb4x" event={"ID":"f70606b9-60d2-494f-bd4c-7d5bb09ecddf","Type":"ContainerStarted","Data":"ca273b1c825b28009d71abd31bf4b92c367f6334773e3f3f3087832919df8ad4"}
Jul 27 18:48:34.596373 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:48:34.596285    2573 generic.go:356] "Generic (PLEG): container finished" podID="f70606b9-60d2-494f-bd4c-7d5bb09ecddf" containerID="1a0433f70ae695cdc092c9cfb0cf3fb2c4dab6555744bc1cc7273728cf2bae8c" exitCode=0
Jul 27 18:48:34.596373 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:48:34.596361    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tmb4x" event={"ID":"f70606b9-60d2-494f-bd4c-7d5bb09ecddf","Type":"ContainerDied","Data":"1a0433f70ae695cdc092c9cfb0cf3fb2c4dab6555744bc1cc7273728cf2bae8c"}
Jul 27 18:48:35.602238 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:48:35.602200    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tmb4x" event={"ID":"f70606b9-60d2-494f-bd4c-7d5bb09ecddf","Type":"ContainerStarted","Data":"92cf288b9677a808ca5c6252deea4612fd28870d575bc201998a4d1f7a806f0d"}
Jul 27 18:48:35.602631 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:48:35.602305    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tmb4x" event={"ID":"f70606b9-60d2-494f-bd4c-7d5bb09ecddf","Type":"ContainerStarted","Data":"fcb1753b151044e5368e44a50f0cd415993597d9823de76517787db850fe9028"}
Jul 27 18:48:35.602631 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:48:35.602560    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tmb4x"
Jul 27 18:48:35.625209 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:48:35.625155    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tmb4x" podStartSLOduration=2.7183014439999997 podStartE2EDuration="6.625140069s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.906838625s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:48:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:48:35.624814498 +0000 UTC m=+4857.768216549" watchObservedRunningTime="2026-07-27 18:48:35.625140069 +0000 UTC m=+4857.768542139"
Jul 27 18:48:36.608443 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:48:36.608411    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tmb4x"
Jul 27 18:48:42.617471 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:48:42.617440    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tmb4x"
Jul 27 18:49:12.621878 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:12.621847    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tmb4x"
Jul 27 18:49:19.773854 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:19.773821    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tmb4x"]
Jul 27 18:49:19.774344 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:19.774055    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tmb4x" podUID="f70606b9-60d2-494f-bd4c-7d5bb09ecddf" containerName="kserve-container" containerID="cri-o://fcb1753b151044e5368e44a50f0cd415993597d9823de76517787db850fe9028" gracePeriod=30
Jul 27 18:49:19.774344 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:19.774148    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tmb4x" podUID="f70606b9-60d2-494f-bd4c-7d5bb09ecddf" containerName="kube-rbac-proxy" containerID="cri-o://92cf288b9677a808ca5c6252deea4612fd28870d575bc201998a4d1f7a806f0d" gracePeriod=30
Jul 27 18:49:19.794469 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:19.794437    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:49:20.810003 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:20.809967    2573 generic.go:356] "Generic (PLEG): container finished" podID="f70606b9-60d2-494f-bd4c-7d5bb09ecddf" containerID="92cf288b9677a808ca5c6252deea4612fd28870d575bc201998a4d1f7a806f0d" exitCode=2
Jul 27 18:49:20.810003 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:20.810006    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tmb4x" event={"ID":"f70606b9-60d2-494f-bd4c-7d5bb09ecddf","Type":"ContainerDied","Data":"92cf288b9677a808ca5c6252deea4612fd28870d575bc201998a4d1f7a806f0d"}
Jul 27 18:49:22.613562 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:22.613515    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tmb4x" podUID="f70606b9-60d2-494f-bd4c-7d5bb09ecddf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.73:8643/healthz\": dial tcp 10.133.0.73:8643: connect: connection refused"
Jul 27 18:49:22.618918 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:22.618891    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tmb4x" podUID="f70606b9-60d2-494f-bd4c-7d5bb09ecddf" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.73:8080/v2/models/isvc-xgboost-v2-mlserver/ready\": dial tcp 10.133.0.73:8080: connect: connection refused"
Jul 27 18:49:24.931601 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:24.931568    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:49:24.972128 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:24.972094    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-g4kfr"]
Jul 27 18:49:24.974750 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:24.974734    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-g4kfr"
Jul 27 18:49:24.978267 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:24.978235    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Jul 27 18:49:24.978388 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:24.978234    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
Jul 27 18:49:24.993568 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:24.993544    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-g4kfr"]
Jul 27 18:49:25.088330 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:25.088286    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrc2h\" (UniqueName: \"kubernetes.io/projected/103da4f9-3705-4bf7-b9a5-0a34178b57dd-kube-api-access-vrc2h\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-g4kfr\" (UID: \"103da4f9-3705-4bf7-b9a5-0a34178b57dd\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-g4kfr"
Jul 27 18:49:25.088538 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:25.088384    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/103da4f9-3705-4bf7-b9a5-0a34178b57dd-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-g4kfr\" (UID: \"103da4f9-3705-4bf7-b9a5-0a34178b57dd\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-g4kfr"
Jul 27 18:49:25.088538 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:25.088451    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/103da4f9-3705-4bf7-b9a5-0a34178b57dd-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-g4kfr\" (UID: \"103da4f9-3705-4bf7-b9a5-0a34178b57dd\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-g4kfr"
Jul 27 18:49:25.088538 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:25.088474    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/103da4f9-3705-4bf7-b9a5-0a34178b57dd-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-g4kfr\" (UID: \"103da4f9-3705-4bf7-b9a5-0a34178b57dd\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-g4kfr"
Jul 27 18:49:25.088538 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:25.088514    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/103da4f9-3705-4bf7-b9a5-0a34178b57dd-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-g4kfr\" (UID: \"103da4f9-3705-4bf7-b9a5-0a34178b57dd\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-g4kfr"
Jul 27 18:49:25.111412 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:25.111369    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:49:25.189792 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:25.189699    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/103da4f9-3705-4bf7-b9a5-0a34178b57dd-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-g4kfr\" (UID: \"103da4f9-3705-4bf7-b9a5-0a34178b57dd\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-g4kfr"
Jul 27 18:49:25.189792 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:25.189743    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/103da4f9-3705-4bf7-b9a5-0a34178b57dd-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-g4kfr\" (UID: \"103da4f9-3705-4bf7-b9a5-0a34178b57dd\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-g4kfr"
Jul 27 18:49:25.189792 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:25.189771    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/103da4f9-3705-4bf7-b9a5-0a34178b57dd-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-g4kfr\" (UID: \"103da4f9-3705-4bf7-b9a5-0a34178b57dd\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-g4kfr"
Jul 27 18:49:25.190041 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:25.189829    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrc2h\" (UniqueName: \"kubernetes.io/projected/103da4f9-3705-4bf7-b9a5-0a34178b57dd-kube-api-access-vrc2h\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-g4kfr\" (UID: \"103da4f9-3705-4bf7-b9a5-0a34178b57dd\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-g4kfr"
Jul 27 18:49:25.190041 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:25.189864    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/103da4f9-3705-4bf7-b9a5-0a34178b57dd-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-g4kfr\" (UID: \"103da4f9-3705-4bf7-b9a5-0a34178b57dd\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-g4kfr"
Jul 27 18:49:25.190340 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:25.190317    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/103da4f9-3705-4bf7-b9a5-0a34178b57dd-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-g4kfr\" (UID: \"103da4f9-3705-4bf7-b9a5-0a34178b57dd\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-g4kfr"
Jul 27 18:49:25.190525 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:25.190507    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/103da4f9-3705-4bf7-b9a5-0a34178b57dd-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-g4kfr\" (UID: \"103da4f9-3705-4bf7-b9a5-0a34178b57dd\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-g4kfr"
Jul 27 18:49:25.190597 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:25.190530    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/103da4f9-3705-4bf7-b9a5-0a34178b57dd-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-g4kfr\" (UID: \"103da4f9-3705-4bf7-b9a5-0a34178b57dd\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-g4kfr"
Jul 27 18:49:25.192386 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:25.192367    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/103da4f9-3705-4bf7-b9a5-0a34178b57dd-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-g4kfr\" (UID: \"103da4f9-3705-4bf7-b9a5-0a34178b57dd\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-g4kfr"
Jul 27 18:49:25.201436 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:25.201407    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrc2h\" (UniqueName: \"kubernetes.io/projected/103da4f9-3705-4bf7-b9a5-0a34178b57dd-kube-api-access-vrc2h\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-g4kfr\" (UID: \"103da4f9-3705-4bf7-b9a5-0a34178b57dd\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-g4kfr"
Jul 27 18:49:25.285084 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:25.285044    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-g4kfr"
Jul 27 18:49:25.420205 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:25.420177    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-g4kfr"]
Jul 27 18:49:25.422327 ip-10-0-140-0 kubenswrapper[2573]: W0727 18:49:25.422302    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod103da4f9_3705_4bf7_b9a5_0a34178b57dd.slice/crio-c07e7eba5dfa2e1a0898adba6592f2c3e18485fb51255c0336c2fa9d7908d9c1 WatchSource:0}: Error finding container c07e7eba5dfa2e1a0898adba6592f2c3e18485fb51255c0336c2fa9d7908d9c1: Status 404 returned error can't find the container with id c07e7eba5dfa2e1a0898adba6592f2c3e18485fb51255c0336c2fa9d7908d9c1
Jul 27 18:49:25.833498 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:25.833411    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-g4kfr" event={"ID":"103da4f9-3705-4bf7-b9a5-0a34178b57dd","Type":"ContainerStarted","Data":"6d153a5b76cae7c795b6024b044cec9fd7db56d59b514fb60e1ca81481a8dbde"}
Jul 27 18:49:25.833498 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:25.833466    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-g4kfr" event={"ID":"103da4f9-3705-4bf7-b9a5-0a34178b57dd","Type":"ContainerStarted","Data":"c07e7eba5dfa2e1a0898adba6592f2c3e18485fb51255c0336c2fa9d7908d9c1"}
Jul 27 18:49:26.630695 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:26.630667    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tmb4x"
Jul 27 18:49:26.704200 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:26.704111    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f70606b9-60d2-494f-bd4c-7d5bb09ecddf-kserve-provision-location\") pod \"f70606b9-60d2-494f-bd4c-7d5bb09ecddf\" (UID: \"f70606b9-60d2-494f-bd4c-7d5bb09ecddf\") "
Jul 27 18:49:26.704200 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:26.704162    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ll7gh\" (UniqueName: \"kubernetes.io/projected/f70606b9-60d2-494f-bd4c-7d5bb09ecddf-kube-api-access-ll7gh\") pod \"f70606b9-60d2-494f-bd4c-7d5bb09ecddf\" (UID: \"f70606b9-60d2-494f-bd4c-7d5bb09ecddf\") "
Jul 27 18:49:26.704537 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:26.704204    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f70606b9-60d2-494f-bd4c-7d5bb09ecddf-cabundle-cert\") pod \"f70606b9-60d2-494f-bd4c-7d5bb09ecddf\" (UID: \"f70606b9-60d2-494f-bd4c-7d5bb09ecddf\") "
Jul 27 18:49:26.704537 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:26.704240    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f70606b9-60d2-494f-bd4c-7d5bb09ecddf-kube-rbac-proxy-sar-config\") pod \"f70606b9-60d2-494f-bd4c-7d5bb09ecddf\" (UID: \"f70606b9-60d2-494f-bd4c-7d5bb09ecddf\") "
Jul 27 18:49:26.704537 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:26.704331    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f70606b9-60d2-494f-bd4c-7d5bb09ecddf-proxy-tls\") pod \"f70606b9-60d2-494f-bd4c-7d5bb09ecddf\" (UID: \"f70606b9-60d2-494f-bd4c-7d5bb09ecddf\") "
Jul 27 18:49:26.704537 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:26.704465    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f70606b9-60d2-494f-bd4c-7d5bb09ecddf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f70606b9-60d2-494f-bd4c-7d5bb09ecddf" (UID: "f70606b9-60d2-494f-bd4c-7d5bb09ecddf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:49:26.704730 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:26.704630    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f70606b9-60d2-494f-bd4c-7d5bb09ecddf-kserve-provision-location\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:49:26.704774 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:26.704735    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f70606b9-60d2-494f-bd4c-7d5bb09ecddf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f70606b9-60d2-494f-bd4c-7d5bb09ecddf" (UID: "f70606b9-60d2-494f-bd4c-7d5bb09ecddf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:49:26.704823 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:26.704802    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f70606b9-60d2-494f-bd4c-7d5bb09ecddf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f70606b9-60d2-494f-bd4c-7d5bb09ecddf" (UID: "f70606b9-60d2-494f-bd4c-7d5bb09ecddf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:49:26.706447 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:26.706423    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f70606b9-60d2-494f-bd4c-7d5bb09ecddf-kube-api-access-ll7gh" (OuterVolumeSpecName: "kube-api-access-ll7gh") pod "f70606b9-60d2-494f-bd4c-7d5bb09ecddf" (UID: "f70606b9-60d2-494f-bd4c-7d5bb09ecddf"). InnerVolumeSpecName "kube-api-access-ll7gh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:49:26.706559 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:26.706542    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f70606b9-60d2-494f-bd4c-7d5bb09ecddf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f70606b9-60d2-494f-bd4c-7d5bb09ecddf" (UID: "f70606b9-60d2-494f-bd4c-7d5bb09ecddf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:49:26.805207 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:26.805170    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f70606b9-60d2-494f-bd4c-7d5bb09ecddf-proxy-tls\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:49:26.805207 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:26.805202    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll7gh\" (UniqueName: \"kubernetes.io/projected/f70606b9-60d2-494f-bd4c-7d5bb09ecddf-kube-api-access-ll7gh\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:49:26.805207 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:26.805215    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f70606b9-60d2-494f-bd4c-7d5bb09ecddf-cabundle-cert\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:49:26.805437 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:26.805223    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f70606b9-60d2-494f-bd4c-7d5bb09ecddf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:49:26.839775 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:26.839737    2573 generic.go:356] "Generic (PLEG): container finished" podID="f70606b9-60d2-494f-bd4c-7d5bb09ecddf" containerID="fcb1753b151044e5368e44a50f0cd415993597d9823de76517787db850fe9028" exitCode=0
Jul 27 18:49:26.839912 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:26.839804    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tmb4x"
Jul 27 18:49:26.839912 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:26.839829    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tmb4x" event={"ID":"f70606b9-60d2-494f-bd4c-7d5bb09ecddf","Type":"ContainerDied","Data":"fcb1753b151044e5368e44a50f0cd415993597d9823de76517787db850fe9028"}
Jul 27 18:49:26.839912 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:26.839893    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tmb4x" event={"ID":"f70606b9-60d2-494f-bd4c-7d5bb09ecddf","Type":"ContainerDied","Data":"ca273b1c825b28009d71abd31bf4b92c367f6334773e3f3f3087832919df8ad4"}
Jul 27 18:49:26.840021 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:26.839920    2573 scope.go:117] "RemoveContainer" containerID="92cf288b9677a808ca5c6252deea4612fd28870d575bc201998a4d1f7a806f0d"
Jul 27 18:49:26.848357 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:26.848341    2573 scope.go:117] "RemoveContainer" containerID="fcb1753b151044e5368e44a50f0cd415993597d9823de76517787db850fe9028"
Jul 27 18:49:26.856486 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:26.856468    2573 scope.go:117] "RemoveContainer" containerID="1a0433f70ae695cdc092c9cfb0cf3fb2c4dab6555744bc1cc7273728cf2bae8c"
Jul 27 18:49:26.863538 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:26.863509    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tmb4x"]
Jul 27 18:49:26.865481 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:26.865450    2573 scope.go:117] "RemoveContainer" containerID="92cf288b9677a808ca5c6252deea4612fd28870d575bc201998a4d1f7a806f0d"
Jul 27 18:49:26.865784 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:49:26.865766    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92cf288b9677a808ca5c6252deea4612fd28870d575bc201998a4d1f7a806f0d\": container with ID starting with 92cf288b9677a808ca5c6252deea4612fd28870d575bc201998a4d1f7a806f0d not found: ID does not exist" containerID="92cf288b9677a808ca5c6252deea4612fd28870d575bc201998a4d1f7a806f0d"
Jul 27 18:49:26.865864 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:26.865798    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92cf288b9677a808ca5c6252deea4612fd28870d575bc201998a4d1f7a806f0d"} err="failed to get container status \"92cf288b9677a808ca5c6252deea4612fd28870d575bc201998a4d1f7a806f0d\": rpc error: code = NotFound desc = could not find container \"92cf288b9677a808ca5c6252deea4612fd28870d575bc201998a4d1f7a806f0d\": container with ID starting with 92cf288b9677a808ca5c6252deea4612fd28870d575bc201998a4d1f7a806f0d not found: ID does not exist"
Jul 27 18:49:26.865864 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:26.865824    2573 scope.go:117] "RemoveContainer" containerID="fcb1753b151044e5368e44a50f0cd415993597d9823de76517787db850fe9028"
Jul 27 18:49:26.866141 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:49:26.866124    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcb1753b151044e5368e44a50f0cd415993597d9823de76517787db850fe9028\": container with ID starting with fcb1753b151044e5368e44a50f0cd415993597d9823de76517787db850fe9028 not found: ID does not exist" containerID="fcb1753b151044e5368e44a50f0cd415993597d9823de76517787db850fe9028"
Jul 27 18:49:26.866185 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:26.866142    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-tmb4x"]
Jul 27 18:49:26.866185 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:26.866147    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcb1753b151044e5368e44a50f0cd415993597d9823de76517787db850fe9028"} err="failed to get container status \"fcb1753b151044e5368e44a50f0cd415993597d9823de76517787db850fe9028\": rpc error: code = NotFound desc = could not find container \"fcb1753b151044e5368e44a50f0cd415993597d9823de76517787db850fe9028\": container with ID starting with fcb1753b151044e5368e44a50f0cd415993597d9823de76517787db850fe9028 not found: ID does not exist"
Jul 27 18:49:26.866185 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:26.866164    2573 scope.go:117] "RemoveContainer" containerID="1a0433f70ae695cdc092c9cfb0cf3fb2c4dab6555744bc1cc7273728cf2bae8c"
Jul 27 18:49:26.866548 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:49:26.866527    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a0433f70ae695cdc092c9cfb0cf3fb2c4dab6555744bc1cc7273728cf2bae8c\": container with ID starting with 1a0433f70ae695cdc092c9cfb0cf3fb2c4dab6555744bc1cc7273728cf2bae8c not found: ID does not exist" containerID="1a0433f70ae695cdc092c9cfb0cf3fb2c4dab6555744bc1cc7273728cf2bae8c"
Jul 27 18:49:26.866634 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:26.866550    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a0433f70ae695cdc092c9cfb0cf3fb2c4dab6555744bc1cc7273728cf2bae8c"} err="failed to get container status \"1a0433f70ae695cdc092c9cfb0cf3fb2c4dab6555744bc1cc7273728cf2bae8c\": rpc error: code = NotFound desc = could not find container \"1a0433f70ae695cdc092c9cfb0cf3fb2c4dab6555744bc1cc7273728cf2bae8c\": container with ID starting with 1a0433f70ae695cdc092c9cfb0cf3fb2c4dab6555744bc1cc7273728cf2bae8c not found: ID does not exist"
Jul 27 18:49:28.469104 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:28.469073    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f70606b9-60d2-494f-bd4c-7d5bb09ecddf" path="/var/lib/kubelet/pods/f70606b9-60d2-494f-bd4c-7d5bb09ecddf/volumes"
Jul 27 18:49:29.854829 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:29.854748    2573 generic.go:356] "Generic (PLEG): container finished" podID="103da4f9-3705-4bf7-b9a5-0a34178b57dd" containerID="6d153a5b76cae7c795b6024b044cec9fd7db56d59b514fb60e1ca81481a8dbde" exitCode=0
Jul 27 18:49:29.855203 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:29.854820    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-g4kfr" event={"ID":"103da4f9-3705-4bf7-b9a5-0a34178b57dd","Type":"ContainerDied","Data":"6d153a5b76cae7c795b6024b044cec9fd7db56d59b514fb60e1ca81481a8dbde"}
Jul 27 18:49:30.861773 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:30.861744    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-g4kfr" event={"ID":"103da4f9-3705-4bf7-b9a5-0a34178b57dd","Type":"ContainerStarted","Data":"3f9a7c340380e694ff658972c22ee0a9330636c960c87ad7b99bb1ba8113588c"}
Jul 27 18:49:30.862163 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:30.861803    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-g4kfr" event={"ID":"103da4f9-3705-4bf7-b9a5-0a34178b57dd","Type":"ContainerStarted","Data":"50dc4e5198c29e445dca43ba93761b1b14b764fcc98c014ae7d247b9471451a3"}
Jul 27 18:49:30.862163 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:30.861992    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-g4kfr"
Jul 27 18:49:30.862163 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:30.862038    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-g4kfr"
Jul 27 18:49:30.885129 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:30.885078    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-g4kfr" podStartSLOduration=2.965147596 podStartE2EDuration="6.885061156s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.91991356s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:49:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:49:30.88352347 +0000 UTC m=+4913.026925521" watchObservedRunningTime="2026-07-27 18:49:30.885061156 +0000 UTC m=+4913.028463207"
Jul 27 18:49:36.872183 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:49:36.872152    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-g4kfr"
Jul 27 18:50:06.877098 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:50:06.877065    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-g4kfr"
Jul 27 18:50:15.032343 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:50:15.032312    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-g4kfr"]
Jul 27 18:50:15.033018 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:50:15.032549    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-g4kfr" podUID="103da4f9-3705-4bf7-b9a5-0a34178b57dd" containerName="kserve-container" containerID="cri-o://50dc4e5198c29e445dca43ba93761b1b14b764fcc98c014ae7d247b9471451a3" gracePeriod=30
Jul 27 18:50:15.033018 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:50:15.032620    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-g4kfr" podUID="103da4f9-3705-4bf7-b9a5-0a34178b57dd" containerName="kube-rbac-proxy" containerID="cri-o://3f9a7c340380e694ff658972c22ee0a9330636c960c87ad7b99bb1ba8113588c" gracePeriod=30
Jul 27 18:50:15.052349 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:50:15.052320    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:50:16.059548 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:50:16.059515    2573 generic.go:356] "Generic (PLEG): container finished" podID="103da4f9-3705-4bf7-b9a5-0a34178b57dd" containerID="3f9a7c340380e694ff658972c22ee0a9330636c960c87ad7b99bb1ba8113588c" exitCode=2
Jul 27 18:50:16.059927 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:50:16.059589    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-g4kfr" event={"ID":"103da4f9-3705-4bf7-b9a5-0a34178b57dd","Type":"ContainerDied","Data":"3f9a7c340380e694ff658972c22ee0a9330636c960c87ad7b99bb1ba8113588c"}
Jul 27 18:50:16.866564 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:50:16.866524    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-g4kfr" podUID="103da4f9-3705-4bf7-b9a5-0a34178b57dd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.74:8643/healthz\": dial tcp 10.133.0.74:8643: connect: connection refused"
Jul 27 18:50:20.180488 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:50:20.180448    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:50:20.315622 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:50:20.315590    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:50:21.783658 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:50:21.783629    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-g4kfr"
Jul 27 18:50:21.861795 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:50:21.861720    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrc2h\" (UniqueName: \"kubernetes.io/projected/103da4f9-3705-4bf7-b9a5-0a34178b57dd-kube-api-access-vrc2h\") pod \"103da4f9-3705-4bf7-b9a5-0a34178b57dd\" (UID: \"103da4f9-3705-4bf7-b9a5-0a34178b57dd\") "
Jul 27 18:50:21.861795 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:50:21.861778    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/103da4f9-3705-4bf7-b9a5-0a34178b57dd-kserve-provision-location\") pod \"103da4f9-3705-4bf7-b9a5-0a34178b57dd\" (UID: \"103da4f9-3705-4bf7-b9a5-0a34178b57dd\") "
Jul 27 18:50:21.862015 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:50:21.861813    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/103da4f9-3705-4bf7-b9a5-0a34178b57dd-proxy-tls\") pod \"103da4f9-3705-4bf7-b9a5-0a34178b57dd\" (UID: \"103da4f9-3705-4bf7-b9a5-0a34178b57dd\") "
Jul 27 18:50:21.862015 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:50:21.861869    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/103da4f9-3705-4bf7-b9a5-0a34178b57dd-kube-rbac-proxy-sar-config\") pod \"103da4f9-3705-4bf7-b9a5-0a34178b57dd\" (UID: \"103da4f9-3705-4bf7-b9a5-0a34178b57dd\") "
Jul 27 18:50:21.862015 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:50:21.861909    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/103da4f9-3705-4bf7-b9a5-0a34178b57dd-cabundle-cert\") pod \"103da4f9-3705-4bf7-b9a5-0a34178b57dd\" (UID: \"103da4f9-3705-4bf7-b9a5-0a34178b57dd\") "
Jul 27 18:50:21.862195 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:50:21.862121    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103da4f9-3705-4bf7-b9a5-0a34178b57dd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "103da4f9-3705-4bf7-b9a5-0a34178b57dd" (UID: "103da4f9-3705-4bf7-b9a5-0a34178b57dd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:50:21.862297 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:50:21.862243    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/103da4f9-3705-4bf7-b9a5-0a34178b57dd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "103da4f9-3705-4bf7-b9a5-0a34178b57dd" (UID: "103da4f9-3705-4bf7-b9a5-0a34178b57dd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:50:21.862357 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:50:21.862310    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/103da4f9-3705-4bf7-b9a5-0a34178b57dd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "103da4f9-3705-4bf7-b9a5-0a34178b57dd" (UID: "103da4f9-3705-4bf7-b9a5-0a34178b57dd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:50:21.863992 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:50:21.863968    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/103da4f9-3705-4bf7-b9a5-0a34178b57dd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "103da4f9-3705-4bf7-b9a5-0a34178b57dd" (UID: "103da4f9-3705-4bf7-b9a5-0a34178b57dd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:50:21.863992 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:50:21.863971    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/103da4f9-3705-4bf7-b9a5-0a34178b57dd-kube-api-access-vrc2h" (OuterVolumeSpecName: "kube-api-access-vrc2h") pod "103da4f9-3705-4bf7-b9a5-0a34178b57dd" (UID: "103da4f9-3705-4bf7-b9a5-0a34178b57dd"). InnerVolumeSpecName "kube-api-access-vrc2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:50:21.962768 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:50:21.962731    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrc2h\" (UniqueName: \"kubernetes.io/projected/103da4f9-3705-4bf7-b9a5-0a34178b57dd-kube-api-access-vrc2h\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:50:21.962768 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:50:21.962762    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/103da4f9-3705-4bf7-b9a5-0a34178b57dd-kserve-provision-location\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:50:21.962768 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:50:21.962773    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/103da4f9-3705-4bf7-b9a5-0a34178b57dd-proxy-tls\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:50:21.963008 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:50:21.962783    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/103da4f9-3705-4bf7-b9a5-0a34178b57dd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:50:21.963008 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:50:21.962792    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/103da4f9-3705-4bf7-b9a5-0a34178b57dd-cabundle-cert\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:50:22.087943 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:50:22.087911    2573 generic.go:356] "Generic (PLEG): container finished" podID="103da4f9-3705-4bf7-b9a5-0a34178b57dd" containerID="50dc4e5198c29e445dca43ba93761b1b14b764fcc98c014ae7d247b9471451a3" exitCode=0
Jul 27 18:50:22.088132 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:50:22.087971    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-g4kfr"
Jul 27 18:50:22.088132 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:50:22.087988    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-g4kfr" event={"ID":"103da4f9-3705-4bf7-b9a5-0a34178b57dd","Type":"ContainerDied","Data":"50dc4e5198c29e445dca43ba93761b1b14b764fcc98c014ae7d247b9471451a3"}
Jul 27 18:50:22.088132 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:50:22.088048    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-g4kfr" event={"ID":"103da4f9-3705-4bf7-b9a5-0a34178b57dd","Type":"ContainerDied","Data":"c07e7eba5dfa2e1a0898adba6592f2c3e18485fb51255c0336c2fa9d7908d9c1"}
Jul 27 18:50:22.088132 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:50:22.088077    2573 scope.go:117] "RemoveContainer" containerID="3f9a7c340380e694ff658972c22ee0a9330636c960c87ad7b99bb1ba8113588c"
Jul 27 18:50:22.096787 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:50:22.096767    2573 scope.go:117] "RemoveContainer" containerID="50dc4e5198c29e445dca43ba93761b1b14b764fcc98c014ae7d247b9471451a3"
Jul 27 18:50:22.104476 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:50:22.104456    2573 scope.go:117] "RemoveContainer" containerID="6d153a5b76cae7c795b6024b044cec9fd7db56d59b514fb60e1ca81481a8dbde"
Jul 27 18:50:22.111024 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:50:22.110996    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-g4kfr"]
Jul 27 18:50:22.113911 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:50:22.113888    2573 scope.go:117] "RemoveContainer" containerID="3f9a7c340380e694ff658972c22ee0a9330636c960c87ad7b99bb1ba8113588c"
Jul 27 18:50:22.114212 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:50:22.114194    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-g4kfr"]
Jul 27 18:50:22.114374 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:50:22.114202    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f9a7c340380e694ff658972c22ee0a9330636c960c87ad7b99bb1ba8113588c\": container with ID starting with 3f9a7c340380e694ff658972c22ee0a9330636c960c87ad7b99bb1ba8113588c not found: ID does not exist" containerID="3f9a7c340380e694ff658972c22ee0a9330636c960c87ad7b99bb1ba8113588c"
Jul 27 18:50:22.114374 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:50:22.114234    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f9a7c340380e694ff658972c22ee0a9330636c960c87ad7b99bb1ba8113588c"} err="failed to get container status \"3f9a7c340380e694ff658972c22ee0a9330636c960c87ad7b99bb1ba8113588c\": rpc error: code = NotFound desc = could not find container \"3f9a7c340380e694ff658972c22ee0a9330636c960c87ad7b99bb1ba8113588c\": container with ID starting with 3f9a7c340380e694ff658972c22ee0a9330636c960c87ad7b99bb1ba8113588c not found: ID does not exist"
Jul 27 18:50:22.114374 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:50:22.114267    2573 scope.go:117] "RemoveContainer" containerID="50dc4e5198c29e445dca43ba93761b1b14b764fcc98c014ae7d247b9471451a3"
Jul 27 18:50:22.114545 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:50:22.114528    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50dc4e5198c29e445dca43ba93761b1b14b764fcc98c014ae7d247b9471451a3\": container with ID starting with 50dc4e5198c29e445dca43ba93761b1b14b764fcc98c014ae7d247b9471451a3 not found: ID does not exist" containerID="50dc4e5198c29e445dca43ba93761b1b14b764fcc98c014ae7d247b9471451a3"
Jul 27 18:50:22.114586 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:50:22.114553    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50dc4e5198c29e445dca43ba93761b1b14b764fcc98c014ae7d247b9471451a3"} err="failed to get container status \"50dc4e5198c29e445dca43ba93761b1b14b764fcc98c014ae7d247b9471451a3\": rpc error: code = NotFound desc = could not find container \"50dc4e5198c29e445dca43ba93761b1b14b764fcc98c014ae7d247b9471451a3\": container with ID starting with 50dc4e5198c29e445dca43ba93761b1b14b764fcc98c014ae7d247b9471451a3 not found: ID does not exist"
Jul 27 18:50:22.114586 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:50:22.114569    2573 scope.go:117] "RemoveContainer" containerID="6d153a5b76cae7c795b6024b044cec9fd7db56d59b514fb60e1ca81481a8dbde"
Jul 27 18:50:22.114834 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:50:22.114817    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d153a5b76cae7c795b6024b044cec9fd7db56d59b514fb60e1ca81481a8dbde\": container with ID starting with 6d153a5b76cae7c795b6024b044cec9fd7db56d59b514fb60e1ca81481a8dbde not found: ID does not exist" containerID="6d153a5b76cae7c795b6024b044cec9fd7db56d59b514fb60e1ca81481a8dbde"
Jul 27 18:50:22.114875 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:50:22.114843    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d153a5b76cae7c795b6024b044cec9fd7db56d59b514fb60e1ca81481a8dbde"} err="failed to get container status \"6d153a5b76cae7c795b6024b044cec9fd7db56d59b514fb60e1ca81481a8dbde\": rpc error: code = NotFound desc = could not find container \"6d153a5b76cae7c795b6024b044cec9fd7db56d59b514fb60e1ca81481a8dbde\": container with ID starting with 6d153a5b76cae7c795b6024b044cec9fd7db56d59b514fb60e1ca81481a8dbde not found: ID does not exist"
Jul 27 18:50:22.468779 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:50:22.468747    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="103da4f9-3705-4bf7-b9a5-0a34178b57dd" path="/var/lib/kubelet/pods/103da4f9-3705-4bf7-b9a5-0a34178b57dd/volumes"
Jul 27 18:51:34.112937 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:51:34.112841    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:51:40.358451 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:51:40.358414    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:51:45.463586 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:51:45.463540    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:51:45.510894 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:51:45.510846    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-q2rks"]
Jul 27 18:51:45.511572 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:51:45.511547    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f70606b9-60d2-494f-bd4c-7d5bb09ecddf" containerName="kube-rbac-proxy"
Jul 27 18:51:45.511760 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:51:45.511746    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f70606b9-60d2-494f-bd4c-7d5bb09ecddf" containerName="kube-rbac-proxy"
Jul 27 18:51:45.511873 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:51:45.511861    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="103da4f9-3705-4bf7-b9a5-0a34178b57dd" containerName="kube-rbac-proxy"
Jul 27 18:51:45.511955 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:51:45.511946    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="103da4f9-3705-4bf7-b9a5-0a34178b57dd" containerName="kube-rbac-proxy"
Jul 27 18:51:45.512044 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:51:45.512034    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="103da4f9-3705-4bf7-b9a5-0a34178b57dd" containerName="storage-initializer"
Jul 27 18:51:45.512116 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:51:45.512108    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="103da4f9-3705-4bf7-b9a5-0a34178b57dd" containerName="storage-initializer"
Jul 27 18:51:45.512202 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:51:45.512193    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f70606b9-60d2-494f-bd4c-7d5bb09ecddf" containerName="kserve-container"
Jul 27 18:51:45.512303 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:51:45.512292    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f70606b9-60d2-494f-bd4c-7d5bb09ecddf" containerName="kserve-container"
Jul 27 18:51:45.512413 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:51:45.512403    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f70606b9-60d2-494f-bd4c-7d5bb09ecddf" containerName="storage-initializer"
Jul 27 18:51:45.512489 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:51:45.512480    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f70606b9-60d2-494f-bd4c-7d5bb09ecddf" containerName="storage-initializer"
Jul 27 18:51:45.512571 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:51:45.512562    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="103da4f9-3705-4bf7-b9a5-0a34178b57dd" containerName="kserve-container"
Jul 27 18:51:45.512648 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:51:45.512639    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="103da4f9-3705-4bf7-b9a5-0a34178b57dd" containerName="kserve-container"
Jul 27 18:51:45.512886 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:51:45.512872    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="103da4f9-3705-4bf7-b9a5-0a34178b57dd" containerName="kserve-container"
Jul 27 18:51:45.512990 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:51:45.512980    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f70606b9-60d2-494f-bd4c-7d5bb09ecddf" containerName="kube-rbac-proxy"
Jul 27 18:51:45.513061 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:51:45.513053    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f70606b9-60d2-494f-bd4c-7d5bb09ecddf" containerName="kserve-container"
Jul 27 18:51:45.513142 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:51:45.513131    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="103da4f9-3705-4bf7-b9a5-0a34178b57dd" containerName="kube-rbac-proxy"
Jul 27 18:51:45.517055 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:51:45.517033    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-q2rks"
Jul 27 18:51:45.520144 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:51:45.520123    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 27 18:51:45.520328 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:51:45.520308    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 27 18:51:45.520484 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:51:45.520359    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 27 18:51:45.520484 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:51:45.520160    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
Jul 27 18:51:45.520652 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:51:45.520122    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 27 18:51:45.521572 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:51:45.521549    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z6ld7\""
Jul 27 18:51:45.521852 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:51:45.521832    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 27 18:51:45.523650 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:51:45.523628    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-q2rks"]
Jul 27 18:51:45.606823 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:51:45.606783    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d87d\" (UniqueName: \"kubernetes.io/projected/f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba-kube-api-access-7d87d\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-q2rks\" (UID: \"f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-q2rks"
Jul 27 18:51:45.606823 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:51:45.606824    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-q2rks\" (UID: \"f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-q2rks"
Jul 27 18:51:45.607075 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:51:45.606848    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-q2rks\" (UID: \"f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-q2rks"
Jul 27 18:51:45.607075 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:51:45.606868    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-q2rks\" (UID: \"f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-q2rks"
Jul 27 18:51:45.607075 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:51:45.606978    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-q2rks\" (UID: \"f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-q2rks"
Jul 27 18:51:45.689412 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:51:45.689380    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:51:45.707527 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:51:45.707498    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-q2rks\" (UID: \"f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-q2rks"
Jul 27 18:51:45.707671 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:51:45.707542    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d87d\" (UniqueName: \"kubernetes.io/projected/f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba-kube-api-access-7d87d\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-q2rks\" (UID: \"f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-q2rks"
Jul 27 18:51:45.707671 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:51:45.707562    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-q2rks\" (UID: \"f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-q2rks"
Jul 27 18:51:45.707671 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:51:45.707586    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-q2rks\" (UID: \"f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-q2rks"
Jul 27 18:51:45.707671 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:51:45.707613    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-q2rks\" (UID: \"f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-q2rks"
Jul 27 18:51:45.707986 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:51:45.707961    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-q2rks\" (UID: \"f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-q2rks"
Jul 27 18:51:45.708275 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:51:45.708226    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-q2rks\" (UID: \"f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-q2rks"
Jul 27 18:51:45.708338 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:51:45.708234    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-q2rks\" (UID: \"f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-q2rks"
Jul 27 18:51:45.710151 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:51:45.710127    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-q2rks\" (UID: \"f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-q2rks"
Jul 27 18:51:45.717003 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:51:45.716933    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d87d\" (UniqueName: \"kubernetes.io/projected/f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba-kube-api-access-7d87d\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-q2rks\" (UID: \"f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-q2rks"
Jul 27 18:51:45.829442 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:51:45.829400    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-q2rks"
Jul 27 18:51:45.962517 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:51:45.962492    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-q2rks"]
Jul 27 18:51:45.964695 ip-10-0-140-0 kubenswrapper[2573]: W0727 18:51:45.964666    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0bb9ff3_4fa7_46bb_abce_ba84f323a3ba.slice/crio-390dfda7b642a26abc825d3be4806f5f4840b6b625837334ece2842a0616eab6 WatchSource:0}: Error finding container 390dfda7b642a26abc825d3be4806f5f4840b6b625837334ece2842a0616eab6: Status 404 returned error can't find the container with id 390dfda7b642a26abc825d3be4806f5f4840b6b625837334ece2842a0616eab6
Jul 27 18:51:46.442530 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:51:46.442490    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-q2rks" event={"ID":"f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba","Type":"ContainerStarted","Data":"9192df5f9ef26fd08bf728e776850bb312e08b9090f496336776809962b06dbd"}
Jul 27 18:51:46.442717 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:51:46.442552    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-q2rks" event={"ID":"f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba","Type":"ContainerStarted","Data":"390dfda7b642a26abc825d3be4806f5f4840b6b625837334ece2842a0616eab6"}
Jul 27 18:51:50.461799 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:51:50.461767    2573 generic.go:356] "Generic (PLEG): container finished" podID="f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba" containerID="9192df5f9ef26fd08bf728e776850bb312e08b9090f496336776809962b06dbd" exitCode=0
Jul 27 18:51:50.462327 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:51:50.461845    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-q2rks" event={"ID":"f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba","Type":"ContainerDied","Data":"9192df5f9ef26fd08bf728e776850bb312e08b9090f496336776809962b06dbd"}
Jul 27 18:51:51.469139 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:51:51.469103    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-q2rks" event={"ID":"f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba","Type":"ContainerStarted","Data":"3c545eca3d57a432f1352e36a98b534ef817fcdd839c3f19849bd3f3f2c152f8"}
Jul 27 18:51:51.469560 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:51:51.469163    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-q2rks" event={"ID":"f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba","Type":"ContainerStarted","Data":"465ecc051cc4794e8b7e55a9f4bfebd4dd2bb8664ea662c9cde5b2d04ee8469e"}
Jul 27 18:51:51.469560 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:51:51.469363    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-q2rks"
Jul 27 18:51:51.492433 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:51:51.492388    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-q2rks" podStartSLOduration=2.575994442 podStartE2EDuration="6.492373548s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.916379106s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:51:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:51:51.49181686 +0000 UTC m=+5053.635218926" watchObservedRunningTime="2026-07-27 18:51:51.492373548 +0000 UTC m=+5053.635775600"
Jul 27 18:51:52.472792 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:51:52.472764    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-q2rks"
Jul 27 18:51:58.482029 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:51:58.481998    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-q2rks"
Jul 27 18:52:28.486980 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:52:28.486924    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-q2rks" podUID="f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Jul 27 18:52:38.484930 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:52:38.484901    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-q2rks"
Jul 27 18:52:38.676731 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:52:38.676707    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vc2tz_87dac797-f8d2-4d36-999f-f8ce855dbeff/ovn-acl-logging/0.log"
Jul 27 18:52:38.685744 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:52:38.685722    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vc2tz_87dac797-f8d2-4d36-999f-f8ce855dbeff/ovn-acl-logging/0.log"
Jul 27 18:52:45.602628 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:52:45.602581    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-q2rks"]
Jul 27 18:52:45.603139 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:52:45.602866    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-q2rks" podUID="f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba" containerName="kserve-container" containerID="cri-o://465ecc051cc4794e8b7e55a9f4bfebd4dd2bb8664ea662c9cde5b2d04ee8469e" gracePeriod=30
Jul 27 18:52:45.603139 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:52:45.602930    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-q2rks" podUID="f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba" containerName="kube-rbac-proxy" containerID="cri-o://3c545eca3d57a432f1352e36a98b534ef817fcdd839c3f19849bd3f3f2c152f8" gracePeriod=30
Jul 27 18:52:45.622566 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:52:45.622538    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:52:46.710311 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:52:46.710239    2573 generic.go:356] "Generic (PLEG): container finished" podID="f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba" containerID="3c545eca3d57a432f1352e36a98b534ef817fcdd839c3f19849bd3f3f2c152f8" exitCode=2
Jul 27 18:52:46.710704 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:52:46.710316    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-q2rks" event={"ID":"f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba","Type":"ContainerDied","Data":"3c545eca3d57a432f1352e36a98b534ef817fcdd839c3f19849bd3f3f2c152f8"}
Jul 27 18:52:48.478364 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:52:48.478327    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-q2rks" podUID="f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.75:8643/healthz\": dial tcp 10.133.0.75:8643: connect: connection refused"
Jul 27 18:52:50.710563 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:52:50.710530    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:52:50.875662 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:52:50.875631    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:52:53.557144 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:52:53.557116    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-q2rks"
Jul 27 18:52:53.680075 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:52:53.680033    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7d87d\" (UniqueName: \"kubernetes.io/projected/f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba-kube-api-access-7d87d\") pod \"f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba\" (UID: \"f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba\") "
Jul 27 18:52:53.680309 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:52:53.680096    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba-cabundle-cert\") pod \"f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba\" (UID: \"f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba\") "
Jul 27 18:52:53.680309 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:52:53.680132    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba-kube-rbac-proxy-sar-config\") pod \"f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba\" (UID: \"f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba\") "
Jul 27 18:52:53.680309 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:52:53.680155    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba-kserve-provision-location\") pod \"f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba\" (UID: \"f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba\") "
Jul 27 18:52:53.680497 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:52:53.680351    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba-proxy-tls\") pod \"f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba\" (UID: \"f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba\") "
Jul 27 18:52:53.680554 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:52:53.680525    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba" (UID: "f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:52:53.680554 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:52:53.680537    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba" (UID: "f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:52:53.680642 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:52:53.680536    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba" (UID: "f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 18:52:53.680683 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:52:53.680651    2573 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:52:53.680683 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:52:53.680665    2573 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba-kserve-provision-location\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:52:53.680683 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:52:53.680674    2573 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba-cabundle-cert\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:52:53.682287 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:52:53.682263    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba-kube-api-access-7d87d" (OuterVolumeSpecName: "kube-api-access-7d87d") pod "f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba" (UID: "f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba"). InnerVolumeSpecName "kube-api-access-7d87d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:52:53.682287 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:52:53.682264    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba" (UID: "f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:52:53.743386 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:52:53.743349    2573 generic.go:356] "Generic (PLEG): container finished" podID="f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba" containerID="465ecc051cc4794e8b7e55a9f4bfebd4dd2bb8664ea662c9cde5b2d04ee8469e" exitCode=0
Jul 27 18:52:53.743564 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:52:53.743406    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-q2rks"
Jul 27 18:52:53.743564 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:52:53.743434    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-q2rks" event={"ID":"f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba","Type":"ContainerDied","Data":"465ecc051cc4794e8b7e55a9f4bfebd4dd2bb8664ea662c9cde5b2d04ee8469e"}
Jul 27 18:52:53.743564 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:52:53.743489    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-q2rks" event={"ID":"f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba","Type":"ContainerDied","Data":"390dfda7b642a26abc825d3be4806f5f4840b6b625837334ece2842a0616eab6"}
Jul 27 18:52:53.743564 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:52:53.743515    2573 scope.go:117] "RemoveContainer" containerID="3c545eca3d57a432f1352e36a98b534ef817fcdd839c3f19849bd3f3f2c152f8"
Jul 27 18:52:53.752828 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:52:53.752811    2573 scope.go:117] "RemoveContainer" containerID="465ecc051cc4794e8b7e55a9f4bfebd4dd2bb8664ea662c9cde5b2d04ee8469e"
Jul 27 18:52:53.760227 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:52:53.760208    2573 scope.go:117] "RemoveContainer" containerID="9192df5f9ef26fd08bf728e776850bb312e08b9090f496336776809962b06dbd"
Jul 27 18:52:53.766397 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:52:53.766374    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-q2rks"]
Jul 27 18:52:53.768127 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:52:53.768111    2573 scope.go:117] "RemoveContainer" containerID="3c545eca3d57a432f1352e36a98b534ef817fcdd839c3f19849bd3f3f2c152f8"
Jul 27 18:52:53.768413 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:52:53.768395    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c545eca3d57a432f1352e36a98b534ef817fcdd839c3f19849bd3f3f2c152f8\": container with ID starting with 3c545eca3d57a432f1352e36a98b534ef817fcdd839c3f19849bd3f3f2c152f8 not found: ID does not exist" containerID="3c545eca3d57a432f1352e36a98b534ef817fcdd839c3f19849bd3f3f2c152f8"
Jul 27 18:52:53.768474 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:52:53.768422    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c545eca3d57a432f1352e36a98b534ef817fcdd839c3f19849bd3f3f2c152f8"} err="failed to get container status \"3c545eca3d57a432f1352e36a98b534ef817fcdd839c3f19849bd3f3f2c152f8\": rpc error: code = NotFound desc = could not find container \"3c545eca3d57a432f1352e36a98b534ef817fcdd839c3f19849bd3f3f2c152f8\": container with ID starting with 3c545eca3d57a432f1352e36a98b534ef817fcdd839c3f19849bd3f3f2c152f8 not found: ID does not exist"
Jul 27 18:52:53.768474 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:52:53.768440    2573 scope.go:117] "RemoveContainer" containerID="465ecc051cc4794e8b7e55a9f4bfebd4dd2bb8664ea662c9cde5b2d04ee8469e"
Jul 27 18:52:53.768696 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:52:53.768678    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"465ecc051cc4794e8b7e55a9f4bfebd4dd2bb8664ea662c9cde5b2d04ee8469e\": container with ID starting with 465ecc051cc4794e8b7e55a9f4bfebd4dd2bb8664ea662c9cde5b2d04ee8469e not found: ID does not exist" containerID="465ecc051cc4794e8b7e55a9f4bfebd4dd2bb8664ea662c9cde5b2d04ee8469e"
Jul 27 18:52:53.768737 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:52:53.768700    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"465ecc051cc4794e8b7e55a9f4bfebd4dd2bb8664ea662c9cde5b2d04ee8469e"} err="failed to get container status \"465ecc051cc4794e8b7e55a9f4bfebd4dd2bb8664ea662c9cde5b2d04ee8469e\": rpc error: code = NotFound desc = could not find container \"465ecc051cc4794e8b7e55a9f4bfebd4dd2bb8664ea662c9cde5b2d04ee8469e\": container with ID starting with 465ecc051cc4794e8b7e55a9f4bfebd4dd2bb8664ea662c9cde5b2d04ee8469e not found: ID does not exist"
Jul 27 18:52:53.768737 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:52:53.768714    2573 scope.go:117] "RemoveContainer" containerID="9192df5f9ef26fd08bf728e776850bb312e08b9090f496336776809962b06dbd"
Jul 27 18:52:53.768967 ip-10-0-140-0 kubenswrapper[2573]: E0727 18:52:53.768951    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9192df5f9ef26fd08bf728e776850bb312e08b9090f496336776809962b06dbd\": container with ID starting with 9192df5f9ef26fd08bf728e776850bb312e08b9090f496336776809962b06dbd not found: ID does not exist" containerID="9192df5f9ef26fd08bf728e776850bb312e08b9090f496336776809962b06dbd"
Jul 27 18:52:53.769008 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:52:53.768971    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9192df5f9ef26fd08bf728e776850bb312e08b9090f496336776809962b06dbd"} err="failed to get container status \"9192df5f9ef26fd08bf728e776850bb312e08b9090f496336776809962b06dbd\": rpc error: code = NotFound desc = could not find container \"9192df5f9ef26fd08bf728e776850bb312e08b9090f496336776809962b06dbd\": container with ID starting with 9192df5f9ef26fd08bf728e776850bb312e08b9090f496336776809962b06dbd not found: ID does not exist"
Jul 27 18:52:53.772105 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:52:53.772080    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-q2rks"]
Jul 27 18:52:53.781027 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:52:53.781006    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7d87d\" (UniqueName: \"kubernetes.io/projected/f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba-kube-api-access-7d87d\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:52:53.781106 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:52:53.781027    2573 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba-proxy-tls\") on node \"ip-10-0-140-0.ec2.internal\" DevicePath \"\""
Jul 27 18:52:54.469689 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:52:54.469610    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba" path="/var/lib/kubelet/pods/f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba/volumes"
Jul 27 18:52:54.478651 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:52:54.478622    2573 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-q2rks" podUID="f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.75:8643/healthz\": context deadline exceeded"
Jul 27 18:53:14.680492 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:53:14.680455    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:53:41.587625 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:53:41.587593    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:54:10.887488 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:54:10.887449    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:54:15.986141 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:54:15.986104    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:54:16.106573 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:54:16.106543    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:55:46.198567 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:55:46.198533    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:55:46.323231 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:55:46.323197    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:55:46.848048 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:55:46.848007    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:57:16.348567 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:57:16.348506    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:57:17.385576 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:57:17.385538    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:57:17.788673 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:57:17.788640    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:57:27.422586 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:57:27.422557    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:57:28.834263 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:57:28.834213    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:57:28.935417 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:57:28.935384    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:57:37.418916 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:57:37.418881    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:57:38.705186 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:57:38.703400    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vc2tz_87dac797-f8d2-4d36-999f-f8ce855dbeff/ovn-acl-logging/0.log"
Jul 27 18:57:38.713739 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:57:38.713713    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vc2tz_87dac797-f8d2-4d36-999f-f8ce855dbeff/ovn-acl-logging/0.log"
Jul 27 18:57:57.290436 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:57:57.290396    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:57:57.345978 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:57:57.345951    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:58:58.612234 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:58:58.612203    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:58:59.714041 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:58:59.713966    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:59:00.099478 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:59:00.099397    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:59:09.677977 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:59:09.677941    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:59:10.735419 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:59:10.735388    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 18:59:10.941022 ip-10-0-140-0 kubenswrapper[2573]: I0727 18:59:10.940980    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 19:00:40.875087 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:00:40.875014    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 19:00:41.873880 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:00:41.873843    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 19:00:42.384319 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:00:42.384288    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 19:00:51.895290 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:00:51.895257    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 19:00:53.619404 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:00:53.619370    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 19:00:53.720012 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:00:53.719980    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 19:01:29.774874 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:29.774837    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 19:01:29.900272 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:29.900217    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-grcz9/must-gather-dhbhf"]
Jul 27 19:01:29.900581 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:29.900568    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba" containerName="kube-rbac-proxy"
Jul 27 19:01:29.900630 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:29.900584    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba" containerName="kube-rbac-proxy"
Jul 27 19:01:29.900630 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:29.900596    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba" containerName="storage-initializer"
Jul 27 19:01:29.900630 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:29.900601    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba" containerName="storage-initializer"
Jul 27 19:01:29.900630 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:29.900610    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba" containerName="kserve-container"
Jul 27 19:01:29.900630 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:29.900616    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba" containerName="kserve-container"
Jul 27 19:01:29.900786 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:29.900671    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba" containerName="kube-rbac-proxy"
Jul 27 19:01:29.900786 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:29.900681    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0bb9ff3-4fa7-46bb-abce-ba84f323a3ba" containerName="kserve-container"
Jul 27 19:01:29.903688 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:29.903670    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-grcz9/must-gather-dhbhf"
Jul 27 19:01:29.906433 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:29.906409    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-grcz9\"/\"kube-root-ca.crt\""
Jul 27 19:01:29.906564 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:29.906485    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-grcz9\"/\"default-dockercfg-5fzhb\""
Jul 27 19:01:29.907805 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:29.907788    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-grcz9\"/\"openshift-service-ca.crt\""
Jul 27 19:01:29.911961 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:29.911941    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-grcz9/must-gather-dhbhf"]
Jul 27 19:01:29.913094 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:29.913070    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 19:01:29.985060 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:29.985026    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h456h\" (UniqueName: \"kubernetes.io/projected/546f0b4f-5b40-4275-bc5c-8cf7264150f6-kube-api-access-h456h\") pod \"must-gather-dhbhf\" (UID: \"546f0b4f-5b40-4275-bc5c-8cf7264150f6\") " pod="openshift-must-gather-grcz9/must-gather-dhbhf"
Jul 27 19:01:29.985243 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:29.985087    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/546f0b4f-5b40-4275-bc5c-8cf7264150f6-must-gather-output\") pod \"must-gather-dhbhf\" (UID: \"546f0b4f-5b40-4275-bc5c-8cf7264150f6\") " pod="openshift-must-gather-grcz9/must-gather-dhbhf"
Jul 27 19:01:30.025497 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:30.025424    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 19:01:30.086201 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:30.086169    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/546f0b4f-5b40-4275-bc5c-8cf7264150f6-must-gather-output\") pod \"must-gather-dhbhf\" (UID: \"546f0b4f-5b40-4275-bc5c-8cf7264150f6\") " pod="openshift-must-gather-grcz9/must-gather-dhbhf"
Jul 27 19:01:30.086327 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:30.086224    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h456h\" (UniqueName: \"kubernetes.io/projected/546f0b4f-5b40-4275-bc5c-8cf7264150f6-kube-api-access-h456h\") pod \"must-gather-dhbhf\" (UID: \"546f0b4f-5b40-4275-bc5c-8cf7264150f6\") " pod="openshift-must-gather-grcz9/must-gather-dhbhf"
Jul 27 19:01:30.086598 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:30.086576    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/546f0b4f-5b40-4275-bc5c-8cf7264150f6-must-gather-output\") pod \"must-gather-dhbhf\" (UID: \"546f0b4f-5b40-4275-bc5c-8cf7264150f6\") " pod="openshift-must-gather-grcz9/must-gather-dhbhf"
Jul 27 19:01:30.107336 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:30.107307    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h456h\" (UniqueName: \"kubernetes.io/projected/546f0b4f-5b40-4275-bc5c-8cf7264150f6-kube-api-access-h456h\") pod \"must-gather-dhbhf\" (UID: \"546f0b4f-5b40-4275-bc5c-8cf7264150f6\") " pod="openshift-must-gather-grcz9/must-gather-dhbhf"
Jul 27 19:01:30.226366 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:30.226330    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-grcz9/must-gather-dhbhf"
Jul 27 19:01:30.355174 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:30.355145    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-grcz9/must-gather-dhbhf"]
Jul 27 19:01:30.358008 ip-10-0-140-0 kubenswrapper[2573]: W0727 19:01:30.357978    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod546f0b4f_5b40_4275_bc5c_8cf7264150f6.slice/crio-28405479f571848ef68d239da300a680a88e4aee4b4dc16e65a2047588dcf1e0 WatchSource:0}: Error finding container 28405479f571848ef68d239da300a680a88e4aee4b4dc16e65a2047588dcf1e0: Status 404 returned error can't find the container with id 28405479f571848ef68d239da300a680a88e4aee4b4dc16e65a2047588dcf1e0
Jul 27 19:01:30.360079 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:30.360064    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 19:01:30.953548 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:30.953510    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-grcz9/must-gather-dhbhf" event={"ID":"546f0b4f-5b40-4275-bc5c-8cf7264150f6","Type":"ContainerStarted","Data":"28405479f571848ef68d239da300a680a88e4aee4b4dc16e65a2047588dcf1e0"}
Jul 27 19:01:31.961422 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:31.961382    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-grcz9/must-gather-dhbhf" event={"ID":"546f0b4f-5b40-4275-bc5c-8cf7264150f6","Type":"ContainerStarted","Data":"d8c524d0357d1dcf3eafe1a35a1b59a488cd9ec2485d43ba2320fbad50ca93aa"}
Jul 27 19:01:31.962001 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:31.961974    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-grcz9/must-gather-dhbhf" event={"ID":"546f0b4f-5b40-4275-bc5c-8cf7264150f6","Type":"ContainerStarted","Data":"323412f15d6d9012d1d15caeb3b4b1694d266076bb550a3c1a720b073c0f8e2e"}
Jul 27 19:01:31.980240 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:31.980183    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-grcz9/must-gather-dhbhf" podStartSLOduration=2.127411089 podStartE2EDuration="2.980165378s" totalImagesPullingTime="852.754289ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:01:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:01:31.979158408 +0000 UTC m=+5634.122560460" watchObservedRunningTime="2026-07-27 19:01:31.980165378 +0000 UTC m=+5634.123567428"
Jul 27 19:01:32.802729 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:32.802702    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zr64w_f3460441-b43e-430b-b5ec-21ea720fbd7f/global-pull-secret-syncer/0.log"
Jul 27 19:01:32.860985 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:32.860948    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log"
Jul 27 19:01:33.050179 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:33.050149    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-0.ec2.internal_99d4260fbe93baf5233f885e6c10aaab/haproxy/0.log"
Jul 27 19:01:35.476778 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:35.476734    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 19:01:35.531554 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:35.531511    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-67cbg_c6c489eb-66d0-484b-83b4-af557ac8a997/konnectivity-agent/0.log" limit=1024
Jul 27 19:01:37.066310 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:37.066278    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_00a563aa-d5f9-4c94-86b9-9c013e1473b0/alertmanager/0.log"
Jul 27 19:01:37.092405 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:37.092379    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_00a563aa-d5f9-4c94-86b9-9c013e1473b0/config-reloader/0.log"
Jul 27 19:01:37.116747 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:37.116708    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_00a563aa-d5f9-4c94-86b9-9c013e1473b0/kube-rbac-proxy-web/0.log"
Jul 27 19:01:37.146917 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:37.146881    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_00a563aa-d5f9-4c94-86b9-9c013e1473b0/kube-rbac-proxy/0.log"
Jul 27 19:01:37.173500 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:37.173469    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_00a563aa-d5f9-4c94-86b9-9c013e1473b0/kube-rbac-proxy-metric/0.log"
Jul 27 19:01:37.205587 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:37.205554    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_00a563aa-d5f9-4c94-86b9-9c013e1473b0/prom-label-proxy/0.log"
Jul 27 19:01:37.230464 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:37.230430    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_00a563aa-d5f9-4c94-86b9-9c013e1473b0/init-config-reloader/0.log"
Jul 27 19:01:37.310847 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:37.310810    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-vx9kx_7ab3e588-fe39-4747-8d1f-9b1ce44ffed4/cluster-monitoring-operator/0.log"
Jul 27 19:01:37.510299 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:37.510236    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-pwwlv_d89a1662-c5db-4f1a-8466-2006b605667b/monitoring-plugin/0.log"
Jul 27 19:01:37.623048 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:37.623016    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-chc8q_e4cabfc1-9cab-4eb9-8e32-7dd66c04d572/node-exporter/0.log"
Jul 27 19:01:37.651929 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:37.651894    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-chc8q_e4cabfc1-9cab-4eb9-8e32-7dd66c04d572/kube-rbac-proxy/0.log"
Jul 27 19:01:37.672075 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:37.672051    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-chc8q_e4cabfc1-9cab-4eb9-8e32-7dd66c04d572/init-textfile/0.log"
Jul 27 19:01:37.777170 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:37.777083    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-rxfs4_61385c00-c295-4f64-a5c8-d5f0dd2ce6af/kube-rbac-proxy-main/0.log"
Jul 27 19:01:37.799494 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:37.799456    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-rxfs4_61385c00-c295-4f64-a5c8-d5f0dd2ce6af/kube-rbac-proxy-self/0.log"
Jul 27 19:01:37.820468 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:37.820437    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-rxfs4_61385c00-c295-4f64-a5c8-d5f0dd2ce6af/openshift-state-metrics/0.log"
Jul 27 19:01:38.040990 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:38.040858    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-sscsd_8cf5e605-7a6e-4cf6-af0c-5e7b19038568/prometheus-operator/0.log"
Jul 27 19:01:38.058111 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:38.058078    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-sscsd_8cf5e605-7a6e-4cf6-af0c-5e7b19038568/kube-rbac-proxy/0.log"
Jul 27 19:01:38.082717 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:38.082684    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-9xsht_da10c088-ba88-4668-8e2b-eda1f18d6203/prometheus-operator-admission-webhook/0.log"
Jul 27 19:01:41.335451 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:41.335416    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-grcz9/perf-node-gather-daemonset-ht2zm"]
Jul 27 19:01:41.341826 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:41.341794    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-grcz9/perf-node-gather-daemonset-ht2zm"
Jul 27 19:01:41.345886 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:41.345856    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-grcz9/perf-node-gather-daemonset-ht2zm"]
Jul 27 19:01:41.490508 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:41.490466    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b5e91af2-91ed-48a7-a2f3-ef515a8ac747-proc\") pod \"perf-node-gather-daemonset-ht2zm\" (UID: \"b5e91af2-91ed-48a7-a2f3-ef515a8ac747\") " pod="openshift-must-gather-grcz9/perf-node-gather-daemonset-ht2zm"
Jul 27 19:01:41.490710 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:41.490527    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b5e91af2-91ed-48a7-a2f3-ef515a8ac747-podres\") pod \"perf-node-gather-daemonset-ht2zm\" (UID: \"b5e91af2-91ed-48a7-a2f3-ef515a8ac747\") " pod="openshift-must-gather-grcz9/perf-node-gather-daemonset-ht2zm"
Jul 27 19:01:41.490710 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:41.490557    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5e91af2-91ed-48a7-a2f3-ef515a8ac747-sys\") pod \"perf-node-gather-daemonset-ht2zm\" (UID: \"b5e91af2-91ed-48a7-a2f3-ef515a8ac747\") " pod="openshift-must-gather-grcz9/perf-node-gather-daemonset-ht2zm"
Jul 27 19:01:41.490710 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:41.490660    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b5e91af2-91ed-48a7-a2f3-ef515a8ac747-lib-modules\") pod \"perf-node-gather-daemonset-ht2zm\" (UID: \"b5e91af2-91ed-48a7-a2f3-ef515a8ac747\") " pod="openshift-must-gather-grcz9/perf-node-gather-daemonset-ht2zm"
Jul 27 19:01:41.490710 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:41.490689    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68thg\" (UniqueName: \"kubernetes.io/projected/b5e91af2-91ed-48a7-a2f3-ef515a8ac747-kube-api-access-68thg\") pod \"perf-node-gather-daemonset-ht2zm\" (UID: \"b5e91af2-91ed-48a7-a2f3-ef515a8ac747\") " pod="openshift-must-gather-grcz9/perf-node-gather-daemonset-ht2zm"
Jul 27 19:01:41.591853 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:41.591764    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b5e91af2-91ed-48a7-a2f3-ef515a8ac747-lib-modules\") pod \"perf-node-gather-daemonset-ht2zm\" (UID: \"b5e91af2-91ed-48a7-a2f3-ef515a8ac747\") " pod="openshift-must-gather-grcz9/perf-node-gather-daemonset-ht2zm"
Jul 27 19:01:41.592084 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:41.592055    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b5e91af2-91ed-48a7-a2f3-ef515a8ac747-lib-modules\") pod \"perf-node-gather-daemonset-ht2zm\" (UID: \"b5e91af2-91ed-48a7-a2f3-ef515a8ac747\") " pod="openshift-must-gather-grcz9/perf-node-gather-daemonset-ht2zm"
Jul 27 19:01:41.592191 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:41.592061    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68thg\" (UniqueName: \"kubernetes.io/projected/b5e91af2-91ed-48a7-a2f3-ef515a8ac747-kube-api-access-68thg\") pod \"perf-node-gather-daemonset-ht2zm\" (UID: \"b5e91af2-91ed-48a7-a2f3-ef515a8ac747\") " pod="openshift-must-gather-grcz9/perf-node-gather-daemonset-ht2zm"
Jul 27 19:01:41.592395 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:41.592367    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b5e91af2-91ed-48a7-a2f3-ef515a8ac747-proc\") pod \"perf-node-gather-daemonset-ht2zm\" (UID: \"b5e91af2-91ed-48a7-a2f3-ef515a8ac747\") " pod="openshift-must-gather-grcz9/perf-node-gather-daemonset-ht2zm"
Jul 27 19:01:41.592531 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:41.592445    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b5e91af2-91ed-48a7-a2f3-ef515a8ac747-podres\") pod \"perf-node-gather-daemonset-ht2zm\" (UID: \"b5e91af2-91ed-48a7-a2f3-ef515a8ac747\") " pod="openshift-must-gather-grcz9/perf-node-gather-daemonset-ht2zm"
Jul 27 19:01:41.592531 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:41.592476    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5e91af2-91ed-48a7-a2f3-ef515a8ac747-sys\") pod \"perf-node-gather-daemonset-ht2zm\" (UID: \"b5e91af2-91ed-48a7-a2f3-ef515a8ac747\") " pod="openshift-must-gather-grcz9/perf-node-gather-daemonset-ht2zm"
Jul 27 19:01:41.592531 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:41.592499    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b5e91af2-91ed-48a7-a2f3-ef515a8ac747-proc\") pod \"perf-node-gather-daemonset-ht2zm\" (UID: \"b5e91af2-91ed-48a7-a2f3-ef515a8ac747\") " pod="openshift-must-gather-grcz9/perf-node-gather-daemonset-ht2zm"
Jul 27 19:01:41.592709 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:41.592608    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5e91af2-91ed-48a7-a2f3-ef515a8ac747-sys\") pod \"perf-node-gather-daemonset-ht2zm\" (UID: \"b5e91af2-91ed-48a7-a2f3-ef515a8ac747\") " pod="openshift-must-gather-grcz9/perf-node-gather-daemonset-ht2zm"
Jul 27 19:01:41.592709 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:41.592681    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b5e91af2-91ed-48a7-a2f3-ef515a8ac747-podres\") pod \"perf-node-gather-daemonset-ht2zm\" (UID: \"b5e91af2-91ed-48a7-a2f3-ef515a8ac747\") " pod="openshift-must-gather-grcz9/perf-node-gather-daemonset-ht2zm"
Jul 27 19:01:41.601585 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:41.601521    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68thg\" (UniqueName: \"kubernetes.io/projected/b5e91af2-91ed-48a7-a2f3-ef515a8ac747-kube-api-access-68thg\") pod \"perf-node-gather-daemonset-ht2zm\" (UID: \"b5e91af2-91ed-48a7-a2f3-ef515a8ac747\") " pod="openshift-must-gather-grcz9/perf-node-gather-daemonset-ht2zm"
Jul 27 19:01:41.654678 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:41.654647    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-grcz9/perf-node-gather-daemonset-ht2zm"
Jul 27 19:01:41.828173 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:41.828094    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-grcz9/perf-node-gather-daemonset-ht2zm"]
Jul 27 19:01:41.832180 ip-10-0-140-0 kubenswrapper[2573]: W0727 19:01:41.832120    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb5e91af2_91ed_48a7_a2f3_ef515a8ac747.slice/crio-91609178f7803a4d942bdaf58d1563271fec01a5db3b3f433d7b81c511493dcd WatchSource:0}: Error finding container 91609178f7803a4d942bdaf58d1563271fec01a5db3b3f433d7b81c511493dcd: Status 404 returned error can't find the container with id 91609178f7803a4d942bdaf58d1563271fec01a5db3b3f433d7b81c511493dcd
Jul 27 19:01:42.020442 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:42.020404    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-grcz9/perf-node-gather-daemonset-ht2zm" event={"ID":"b5e91af2-91ed-48a7-a2f3-ef515a8ac747","Type":"ContainerStarted","Data":"e91c57a02f31571fadd34f5860a59e723bcb60dee4757b6135e3ba42867034d1"}
Jul 27 19:01:42.020611 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:42.020468    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-grcz9/perf-node-gather-daemonset-ht2zm" event={"ID":"b5e91af2-91ed-48a7-a2f3-ef515a8ac747","Type":"ContainerStarted","Data":"91609178f7803a4d942bdaf58d1563271fec01a5db3b3f433d7b81c511493dcd"}
Jul 27 19:01:42.020611 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:42.020500    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-grcz9/perf-node-gather-daemonset-ht2zm"
Jul 27 19:01:42.039433 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:42.039365    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-grcz9/perf-node-gather-daemonset-ht2zm" podStartSLOduration=1.039345071 podStartE2EDuration="1.039345071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:01:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:01:42.037049161 +0000 UTC m=+5644.180451215" watchObservedRunningTime="2026-07-27 19:01:42.039345071 +0000 UTC m=+5644.182747124"
Jul 27 19:01:42.040831 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:42.040808    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-php8z_b4f69c89-1bf8-483d-8854-324c9a551e60/dns/0.log"
Jul 27 19:01:42.065521 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:42.065497    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-php8z_b4f69c89-1bf8-483d-8854-324c9a551e60/kube-rbac-proxy/0.log"
Jul 27 19:01:42.193999 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:42.193965    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-95qjs_e19a218b-06c1-4547-9f14-00d201dc6030/dns-node-resolver/0.log"
Jul 27 19:01:42.664337 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:42.664304    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-74648bb95f-zz4vg_0e5f0900-1fec-4aa8-9372-45cb418b7bdd/registry/0.log"
Jul 27 19:01:42.695312 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:42.695276    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-74648bb95f-zz4vg_0e5f0900-1fec-4aa8-9372-45cb418b7bdd/registry/1.log"
Jul 27 19:01:42.774734 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:42.774697    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zb877_2802ee72-4be9-423a-a9bd-102c54f96d06/node-ca/0.log"
Jul 27 19:01:43.806289 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:43.806241    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6nssx_567f3aaf-ffed-47b5-97cf-2451d54b0db7/serve-healthcheck-canary/0.log"
Jul 27 19:01:44.203989 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:44.203932    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8ctdw_fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4/kube-rbac-proxy/0.log"
Jul 27 19:01:44.221047 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:44.221027    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8ctdw_fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4/exporter/0.log"
Jul 27 19:01:44.238366 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:44.238338    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8ctdw_fd6ca9f1-b1b1-4258-964a-7a1c1276b3c4/extractor/0.log"
Jul 27 19:01:46.725808 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:46.725768    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-84fc959947-j9x2v_492d3c37-e7bb-4598-84b8-544d13e956ba/manager/0.log"
Jul 27 19:01:46.742607 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:46.742580    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-jjfkl_562877e9-4748-48fb-8fe6-d27f9f13f93c/server/0.log"
Jul 27 19:01:48.035772 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:48.035742    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-grcz9/perf-node-gather-daemonset-ht2zm"
Jul 27 19:01:51.623660 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:51.623584    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-dcbmp_93ed2402-e28d-4671-b8bd-f7993a1afdd0/migrator/0.log"
Jul 27 19:01:51.642776 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:51.642752    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-dcbmp_93ed2402-e28d-4671-b8bd-f7993a1afdd0/graceful-termination/0.log"
Jul 27 19:01:53.285190 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:53.285166    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sgsbl_1869a1a5-58a8-4839-a479-679491d1f97c/kube-multus-additional-cni-plugins/0.log"
Jul 27 19:01:53.303578 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:53.303553    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sgsbl_1869a1a5-58a8-4839-a479-679491d1f97c/egress-router-binary-copy/0.log"
Jul 27 19:01:53.322666 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:53.322645    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sgsbl_1869a1a5-58a8-4839-a479-679491d1f97c/cni-plugins/0.log"
Jul 27 19:01:53.346603 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:53.346579    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sgsbl_1869a1a5-58a8-4839-a479-679491d1f97c/bond-cni-plugin/0.log"
Jul 27 19:01:53.368086 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:53.368064    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sgsbl_1869a1a5-58a8-4839-a479-679491d1f97c/routeoverride-cni/0.log"
Jul 27 19:01:53.391436 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:53.391403    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sgsbl_1869a1a5-58a8-4839-a479-679491d1f97c/whereabouts-cni-bincopy/0.log"
Jul 27 19:01:53.410478 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:53.410450    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sgsbl_1869a1a5-58a8-4839-a479-679491d1f97c/whereabouts-cni/0.log"
Jul 27 19:01:53.503826 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:53.503787    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-p97fd_cf35f819-6812-444e-a55c-b9fc24bc396e/kube-multus/0.log"
Jul 27 19:01:53.607291 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:53.607202    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zxdxx_2fedc794-294b-44d7-ba67-55dbad330693/network-metrics-daemon/0.log"
Jul 27 19:01:53.621858 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:53.621820    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zxdxx_2fedc794-294b-44d7-ba67-55dbad330693/kube-rbac-proxy/0.log"
Jul 27 19:01:54.878453 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:54.878425    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vc2tz_87dac797-f8d2-4d36-999f-f8ce855dbeff/ovn-controller/0.log"
Jul 27 19:01:54.894006 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:54.893971    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vc2tz_87dac797-f8d2-4d36-999f-f8ce855dbeff/ovn-acl-logging/0.log"
Jul 27 19:01:54.894204 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:54.894143    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vc2tz_87dac797-f8d2-4d36-999f-f8ce855dbeff/ovn-acl-logging/1.log"
Jul 27 19:01:54.911730 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:54.911699    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vc2tz_87dac797-f8d2-4d36-999f-f8ce855dbeff/kube-rbac-proxy-node/0.log"
Jul 27 19:01:54.951297 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:54.951267    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vc2tz_87dac797-f8d2-4d36-999f-f8ce855dbeff/kube-rbac-proxy-ovn-metrics/0.log"
Jul 27 19:01:54.966373 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:54.966347    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vc2tz_87dac797-f8d2-4d36-999f-f8ce855dbeff/northd/0.log"
Jul 27 19:01:54.983106 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:54.983085    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vc2tz_87dac797-f8d2-4d36-999f-f8ce855dbeff/nbdb/0.log"
Jul 27 19:01:55.003265 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:55.003222    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vc2tz_87dac797-f8d2-4d36-999f-f8ce855dbeff/sbdb/0.log"
Jul 27 19:01:55.056143 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:55.056110    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vc2tz_87dac797-f8d2-4d36-999f-f8ce855dbeff/ovnkube-controller/0.log"
Jul 27 19:01:55.977293 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:55.977268    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-7d88cb55d8-fvcgl_953d5e73-7bd2-454e-a2c5-90c411bcbd7d/check-endpoints/0.log"
Jul 27 19:01:56.024898 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:56.024869    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-sqfbc_1f0e6c35-0c8a-427b-9647-00055e7fa320/network-check-target-container/0.log"
Jul 27 19:01:56.963132 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:56.963102    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-qgh42_d9b682b7-4f6d-4f8e-ae7a-d6a6034859d2/iptables-alerter/0.log"
Jul 27 19:01:57.592855 ip-10-0-140-0 kubenswrapper[2573]: I0727 19:01:57.592830    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-rmfml_5682e309-a980-4018-8f8d-c942f245b0f1/tuned/0.log"