Sep 17 17:32:24.051473 ip-10-0-133-139 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 17 17:32:24.051487 ip-10-0-133-139 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 17 17:32:24.051497 ip-10-0-133-139 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 17 17:32:24.051735 ip-10-0-133-139 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 17 17:32:35.418951 ip-10-0-133-139 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 17 17:32:35.418968 ip-10-0-133-139 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 9bb32bff6b474454b63cde4f561be0b3 --
Sep 17 17:34:22.362154 ip-10-0-133-139 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 17:34:22.787198 ip-10-0-133-139 kubenswrapper[2585]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:34:22.787198 ip-10-0-133-139 kubenswrapper[2585]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 17:34:22.787198 ip-10-0-133-139 kubenswrapper[2585]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:34:22.787198 ip-10-0-133-139 kubenswrapper[2585]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 17:34:22.787198 ip-10-0-133-139 kubenswrapper[2585]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 17:34:22.790177 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.790087    2585 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 17:34:22.793396 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793381    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:34:22.793396 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793396    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:34:22.793456 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793399    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:34:22.793456 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793402    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:34:22.793456 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793405    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:34:22.793456 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793408    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:34:22.793456 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793411    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:34:22.793456 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793414    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:34:22.793456 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793416    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:34:22.793456 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793419    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:34:22.793456 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793421    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:34:22.793456 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793424    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:34:22.793456 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793427    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:34:22.793456 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793429    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:34:22.793456 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793432    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:34:22.793456 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793434    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:34:22.793456 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793437    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:34:22.793456 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793462    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:34:22.793832 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793466    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:34:22.793832 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793469    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:34:22.793832 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793472    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:34:22.793832 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793475    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:34:22.793832 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793478    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:34:22.793832 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793480    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:34:22.793832 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793483    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:34:22.793832 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793485    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:34:22.793832 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793488    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:34:22.793832 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793491    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:34:22.793832 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793493    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:34:22.793832 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793496    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:34:22.793832 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793498    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:34:22.793832 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793501    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:34:22.793832 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793503    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:34:22.793832 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793505    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:34:22.793832 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793508    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:34:22.793832 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793510    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:34:22.793832 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793512    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:34:22.793832 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793515    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:34:22.794327 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793517    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:34:22.794327 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793520    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:34:22.794327 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793522    2585 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:34:22.794327 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793524    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:34:22.794327 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793527    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:34:22.794327 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793531    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:34:22.794327 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793533    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:34:22.794327 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793536    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:34:22.794327 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793538    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:34:22.794327 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793541    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:34:22.794327 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793543    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:34:22.794327 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793545    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:34:22.794327 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793548    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:34:22.794327 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793551    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:34:22.794327 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793553    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:34:22.794327 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793556    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:34:22.794327 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793558    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:34:22.794327 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793561    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:34:22.794327 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793563    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:34:22.794327 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793566    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:34:22.794805 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793568    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:34:22.794805 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793572    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:34:22.794805 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793574    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:34:22.794805 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793577    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:34:22.794805 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793582    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:34:22.794805 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793586    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:34:22.794805 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793588    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:34:22.794805 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793591    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:34:22.794805 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793594    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:34:22.794805 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793597    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:34:22.794805 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793600    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:34:22.794805 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793603    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:34:22.794805 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793607    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:34:22.794805 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793610    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:34:22.794805 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793613    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:34:22.794805 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793616    2585 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:34:22.794805 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793619    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:34:22.794805 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793621    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:34:22.794805 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793624    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:34:22.795265 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793626    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:34:22.795265 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793629    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:34:22.795265 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793632    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:34:22.795265 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793635    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:34:22.795265 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793638    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:34:22.795265 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793642    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:34:22.795265 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793645    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:34:22.795265 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793648    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:34:22.795265 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793651    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:34:22.795265 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793653    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:34:22.795265 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793655    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:34:22.795265 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793658    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:34:22.795265 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793661    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:34:22.795265 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793663    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:34:22.795265 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793665    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:34:22.795265 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793668    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:34:22.795265 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793670    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:34:22.795265 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793673    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:34:22.795265 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793676    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:34:22.795757 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793678    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:34:22.795757 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793680    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:34:22.795757 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.793683    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:34:22.795757 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794369    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:34:22.795757 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794376    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:34:22.795757 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794379    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:34:22.795757 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794382    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:34:22.795757 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794384    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:34:22.795757 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794387    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:34:22.795757 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794390    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:34:22.795757 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794396    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:34:22.795757 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794400    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:34:22.795757 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794403    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:34:22.795757 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794406    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:34:22.795757 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794408    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:34:22.795757 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794411    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:34:22.795757 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794414    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:34:22.795757 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794417    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:34:22.795757 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794420    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:34:22.795757 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794423    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:34:22.796255 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794425    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:34:22.796255 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794428    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:34:22.796255 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794431    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:34:22.796255 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794433    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:34:22.796255 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794436    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:34:22.796255 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794438    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:34:22.796255 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794441    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:34:22.796255 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794444    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:34:22.796255 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794447    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:34:22.796255 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794449    2585 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:34:22.796255 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794452    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:34:22.796255 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794454    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:34:22.796255 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794458    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:34:22.796255 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794461    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:34:22.796255 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794463    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:34:22.796255 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794466    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:34:22.796255 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794469    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:34:22.796255 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794471    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:34:22.796255 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794474    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:34:22.796255 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794476    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:34:22.796737 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794479    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:34:22.796737 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794482    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:34:22.796737 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794484    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:34:22.796737 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794487    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:34:22.796737 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794489    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:34:22.796737 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794492    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:34:22.796737 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794494    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:34:22.796737 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794497    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:34:22.796737 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794499    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:34:22.796737 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794502    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:34:22.796737 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794504    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:34:22.796737 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794507    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:34:22.796737 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794509    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:34:22.796737 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794512    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:34:22.796737 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794514    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:34:22.796737 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794517    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:34:22.796737 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794519    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:34:22.796737 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794522    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:34:22.796737 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794525    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:34:22.796737 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794527    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:34:22.797220 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794530    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:34:22.797220 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794532    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:34:22.797220 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794534    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:34:22.797220 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794537    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:34:22.797220 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794539    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:34:22.797220 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794543    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:34:22.797220 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794546    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:34:22.797220 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794548    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:34:22.797220 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794551    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:34:22.797220 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794553    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:34:22.797220 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794556    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:34:22.797220 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794559    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:34:22.797220 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794562    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:34:22.797220 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794565    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:34:22.797220 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794569    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:34:22.797220 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794573    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:34:22.797220 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794576    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:34:22.797220 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794579    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:34:22.797220 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794582    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:34:22.797673 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794585    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:34:22.797673 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794587    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:34:22.797673 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794590    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:34:22.797673 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794592    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:34:22.797673 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794595    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:34:22.797673 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794597    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:34:22.797673 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794600    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:34:22.797673 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794602    2585 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:34:22.797673 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794605    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:34:22.797673 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794608    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:34:22.797673 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794610    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:34:22.797673 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794613    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:34:22.797673 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794615    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:34:22.797673 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794618    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:34:22.797673 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794621    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:34:22.797673 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794623    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:34:22.797673 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794626    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:34:22.797673 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794630    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:34:22.797673 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794633    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:34:22.798147 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794637    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:34:22.798147 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794640    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:34:22.798147 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794643    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:34:22.798147 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.794646    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:34:22.798147 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795001    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:34:22.798147 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795006    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:34:22.798147 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795011    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:34:22.798147 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795015    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:34:22.798147 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795018    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:34:22.798147 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795021    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:34:22.798147 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795024    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:34:22.798147 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795027    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:34:22.798147 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795029    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:34:22.798147 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795033    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:34:22.798147 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795035    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:34:22.798147 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795038    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:34:22.798147 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795041    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:34:22.798147 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795043    2585 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:34:22.798147 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795046    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:34:22.798592 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795049    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:34:22.798592 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795053    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:34:22.798592 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795056    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:34:22.798592 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795059    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:34:22.798592 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795062    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:34:22.798592 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795065    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:34:22.798592 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795068    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:34:22.798592 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795071    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:34:22.798592 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795074    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:34:22.798592 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795076    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:34:22.798592 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795079    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:34:22.798592 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795082    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:34:22.798592 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795085    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:34:22.798592 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795089    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:34:22.798592 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795092    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:34:22.798592 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795094    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:34:22.798592 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795097    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:34:22.798592 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795100    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:34:22.798592 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795102    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:34:22.799080 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795105    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:34:22.799080 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795108    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:34:22.799080 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795110    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:34:22.799080 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795113    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:34:22.799080 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795116    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:34:22.799080 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795118    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:34:22.799080 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795121    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:34:22.799080 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795124    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:34:22.799080 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795127    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:34:22.799080 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795129    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:34:22.799080 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795132    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:34:22.799080 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795134    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:34:22.799080 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795137    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:34:22.799080 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795139    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:34:22.799080 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795141    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:34:22.799080 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795144    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:34:22.799080 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795146    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:34:22.799080 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795151    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:34:22.799080 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795154    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:34:22.799080 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795156    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:34:22.799557 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795159    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:34:22.799557 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795162    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:34:22.799557 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795165    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:34:22.799557 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795168    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:34:22.799557 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795171    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:34:22.799557 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795173    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:34:22.799557 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795176    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:34:22.799557 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795179    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:34:22.799557 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795181    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:34:22.799557 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795184    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:34:22.799557 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795186    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:34:22.799557 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795189    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:34:22.799557 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795192    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:34:22.799557 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795194    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:34:22.799557 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795197    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:34:22.799557 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795199    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:34:22.799557 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795202    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:34:22.799557 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795205    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:34:22.799557 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795207    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:34:22.800009 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795210    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:34:22.800009 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795213    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:34:22.800009 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795216    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:34:22.800009 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795218    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:34:22.800009 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795221    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:34:22.800009 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795223    2585 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:34:22.800009 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795226    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:34:22.800009 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795229    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:34:22.800009 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795231    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:34:22.800009 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795233    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:34:22.800009 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795237    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:34:22.800009 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795240    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:34:22.800009 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795243    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:34:22.800009 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795245    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:34:22.800009 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795247    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:34:22.800009 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795250    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:34:22.800009 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795252    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:34:22.800009 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795255    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:34:22.800009 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795258    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:34:22.800009 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795262    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:34:22.800517 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795264    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:34:22.800517 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795268    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:34:22.800517 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795270    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:34:22.800517 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795273    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:34:22.800517 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795276    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:34:22.800517 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795278    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:34:22.800517 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795348    2585 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 17:34:22.800517 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795394    2585 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 17:34:22.800517 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795400    2585 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 17:34:22.800517 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795404    2585 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 17:34:22.800517 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795408    2585 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 17:34:22.800517 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795411    2585 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 17:34:22.800517 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795416    2585 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 17:34:22.800517 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795420    2585 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 17:34:22.800517 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795423    2585 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 17:34:22.800517 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795426    2585 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 17:34:22.800517 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795429    2585 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 17:34:22.800517 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795432    2585 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 17:34:22.800517 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795435    2585 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 17:34:22.800517 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795438    2585 flags.go:64] FLAG: --cgroup-root=""
Sep 17 17:34:22.800517 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795441    2585 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 17:34:22.800517 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795444    2585 flags.go:64] FLAG: --client-ca-file=""
Sep 17 17:34:22.801039 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795448    2585 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 17:34:22.801039 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795451    2585 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 17:34:22.801039 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795458    2585 flags.go:64] FLAG: --cluster-domain=""
Sep 17 17:34:22.801039 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795461    2585 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 17:34:22.801039 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795464    2585 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 17:34:22.801039 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795467    2585 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 17:34:22.801039 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795470    2585 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 17:34:22.801039 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795474    2585 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 17:34:22.801039 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795477    2585 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 17:34:22.801039 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795481    2585 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 17:34:22.801039 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795484    2585 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 17:34:22.801039 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795488    2585 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 17:34:22.801039 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795491    2585 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 17:34:22.801039 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795494    2585 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 17:34:22.801039 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795497    2585 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 17:34:22.801039 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795500    2585 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 17:34:22.801039 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795504    2585 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 17:34:22.801039 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795507    2585 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 17:34:22.801039 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795510    2585 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 17:34:22.801039 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795513    2585 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 17:34:22.801039 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795515    2585 flags.go:64] FLAG: --enable-server="true"
Sep 17 17:34:22.801039 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795518    2585 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 17:34:22.801039 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795522    2585 flags.go:64] FLAG: --event-burst="100"
Sep 17 17:34:22.801039 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795525    2585 flags.go:64] FLAG: --event-qps="50"
Sep 17 17:34:22.801039 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795528    2585 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 17:34:22.801662 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795531    2585 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 17:34:22.801662 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795534    2585 flags.go:64] FLAG: --eviction-hard=""
Sep 17 17:34:22.801662 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795537    2585 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 17:34:22.801662 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795540    2585 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 17:34:22.801662 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795543    2585 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 17:34:22.801662 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795546    2585 flags.go:64] FLAG: --eviction-soft=""
Sep 17 17:34:22.801662 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795548    2585 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 17:34:22.801662 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795552    2585 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 17:34:22.801662 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795555    2585 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 17:34:22.801662 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795558    2585 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 17:34:22.801662 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795561    2585 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 17:34:22.801662 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795564    2585 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 17:34:22.801662 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795567    2585 flags.go:64] FLAG: --feature-gates=""
Sep 17 17:34:22.801662 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795571    2585 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 17:34:22.801662 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795574    2585 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 17:34:22.801662 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795577    2585 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 17:34:22.801662 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795580    2585 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 17:34:22.801662 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795583    2585 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 17:34:22.801662 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795586    2585 flags.go:64] FLAG: --help="false"
Sep 17 17:34:22.801662 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795589    2585 flags.go:64] FLAG: --hostname-override="ip-10-0-133-139.ec2.internal"
Sep 17 17:34:22.801662 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795592    2585 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 17:34:22.801662 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795595    2585 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 17:34:22.801662 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795598    2585 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 17:34:22.802256 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795602    2585 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 17:34:22.802256 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795605    2585 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 17:34:22.802256 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795608    2585 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 17:34:22.802256 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795611    2585 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 17:34:22.802256 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795614    2585 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 17:34:22.802256 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795617    2585 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 17:34:22.802256 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795619    2585 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 17:34:22.802256 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795622    2585 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 17:34:22.802256 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795625    2585 flags.go:64] FLAG: --kube-reserved=""
Sep 17 17:34:22.802256 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795628    2585 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 17:34:22.802256 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795631    2585 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 17:34:22.802256 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795634    2585 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 17:34:22.802256 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795637    2585 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 17:34:22.802256 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795640    2585 flags.go:64] FLAG: --lock-file=""
Sep 17 17:34:22.802256 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795642    2585 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 17:34:22.802256 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795645    2585 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 17:34:22.802256 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795650    2585 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 17:34:22.802256 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795655    2585 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 17:34:22.802256 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795658    2585 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 17:34:22.802256 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795660    2585 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 17:34:22.802256 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795663    2585 flags.go:64] FLAG: --logging-format="text"
Sep 17 17:34:22.802256 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795666    2585 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 17:34:22.802256 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795670    2585 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 17:34:22.802256 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795672    2585 flags.go:64] FLAG: --manifest-url=""
Sep 17 17:34:22.802832 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795675    2585 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 17:34:22.802832 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795679    2585 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 17:34:22.802832 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795682    2585 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 17:34:22.802832 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795686    2585 flags.go:64] FLAG: --max-pods="110"
Sep 17 17:34:22.802832 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795689    2585 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 17:34:22.802832 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795692    2585 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 17:34:22.802832 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795695    2585 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 17:34:22.802832 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795698    2585 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 17:34:22.802832 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795701    2585 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 17:34:22.802832 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795704    2585 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 17:34:22.802832 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795707    2585 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 17:34:22.802832 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795713    2585 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 17:34:22.802832 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795716    2585 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 17:34:22.802832 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795719    2585 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 17:34:22.802832 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795722    2585 flags.go:64] FLAG: --pod-cidr=""
Sep 17 17:34:22.802832 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795725    2585 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 17:34:22.802832 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795730    2585 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 17:34:22.802832 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795732    2585 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 17:34:22.802832 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795735    2585 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 17:34:22.802832 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795738    2585 flags.go:64] FLAG: --port="10250"
Sep 17 17:34:22.802832 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795741    2585 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 17:34:22.802832 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795744    2585 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f445820191d942be"
Sep 17 17:34:22.802832 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795747    2585 flags.go:64] FLAG: --qos-reserved=""
Sep 17 17:34:22.802832 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795750    2585 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 17:34:22.803781 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795754    2585 flags.go:64] FLAG: --register-node="true"
Sep 17 17:34:22.803781 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795757    2585 flags.go:64] FLAG: --register-with-taints=""
Sep 17 17:34:22.803781 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795761    2585 flags.go:64] FLAG: --registry-burst="10"
Sep 17 17:34:22.803781 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795764    2585 flags.go:64] FLAG: --registry-qps="5"
Sep 17 17:34:22.803781 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795767    2585 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 17:34:22.803781 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795770    2585 flags.go:64] FLAG: --reserved-memory=""
Sep 17 17:34:22.803781 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795774    2585 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 17:34:22.803781 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795777    2585 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 17:34:22.803781 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795780    2585 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 17:34:22.803781 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795783    2585 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 17:34:22.803781 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795786    2585 flags.go:64] FLAG: --runonce="false"
Sep 17 17:34:22.803781 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795788    2585 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 17:34:22.803781 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795791    2585 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 17:34:22.803781 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795794    2585 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 17:34:22.803781 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795797    2585 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 17:34:22.803781 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795800    2585 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 17:34:22.803781 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795804    2585 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 17:34:22.803781 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795806    2585 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 17:34:22.803781 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795810    2585 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 17:34:22.803781 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795812    2585 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 17:34:22.803781 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795815    2585 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 17:34:22.803781 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795818    2585 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 17:34:22.803781 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795820    2585 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 17:34:22.803781 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795823    2585 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 17:34:22.803781 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795826    2585 flags.go:64] FLAG: --system-cgroups=""
Sep 17 17:34:22.804670 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795829    2585 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 17:34:22.804670 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795834    2585 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 17:34:22.804670 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795837    2585 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 17:34:22.804670 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795839    2585 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 17:34:22.804670 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795844    2585 flags.go:64] FLAG: --tls-min-version=""
Sep 17 17:34:22.804670 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795846    2585 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 17:34:22.804670 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795849    2585 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 17:34:22.804670 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795853    2585 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 17:34:22.804670 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795856    2585 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 17:34:22.804670 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795858    2585 flags.go:64] FLAG: --v="2"
Sep 17 17:34:22.804670 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795862    2585 flags.go:64] FLAG: --version="false"
Sep 17 17:34:22.804670 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795866    2585 flags.go:64] FLAG: --vmodule=""
Sep 17 17:34:22.804670 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795869    2585 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 17:34:22.804670 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.795872    2585 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 17:34:22.804670 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795996    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:34:22.804670 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.795999    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:34:22.804670 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796003    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:34:22.804670 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796007    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:34:22.804670 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796010    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:34:22.804670 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796012    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:34:22.804670 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796016    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:34:22.804670 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796019    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:34:22.804670 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796021    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:34:22.805560 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796024    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:34:22.805560 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796027    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:34:22.805560 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796030    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:34:22.805560 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796032    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:34:22.805560 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796036    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:34:22.805560 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796040    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:34:22.805560 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796043    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:34:22.805560 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796045    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:34:22.805560 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796048    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:34:22.805560 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796051    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:34:22.805560 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796053    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:34:22.805560 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796056    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:34:22.805560 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796058    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:34:22.805560 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796061    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:34:22.805560 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796063    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:34:22.805560 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796066    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:34:22.805560 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796070    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:34:22.805560 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796073    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:34:22.805560 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796075    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:34:22.806375 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796078    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:34:22.806375 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796080    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:34:22.806375 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796083    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:34:22.806375 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796087    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:34:22.806375 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796090    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:34:22.806375 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796092    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:34:22.806375 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796095    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:34:22.806375 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796099    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:34:22.806375 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796101    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:34:22.806375 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796104    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:34:22.806375 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796106    2585 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:34:22.806375 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796109    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:34:22.806375 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796111    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:34:22.806375 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796114    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:34:22.806375 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796117    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:34:22.806375 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796120    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:34:22.806375 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796122    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:34:22.806375 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796125    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:34:22.806375 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796128    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:34:22.806375 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796130    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:34:22.806957 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796133    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:34:22.806957 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796135    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:34:22.806957 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796138    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:34:22.806957 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796140    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:34:22.806957 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796143    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:34:22.806957 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796146    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:34:22.806957 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796148    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:34:22.806957 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796151    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:34:22.806957 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796153    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:34:22.806957 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796157    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:34:22.806957 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796160    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:34:22.806957 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796164    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:34:22.806957 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796167    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:34:22.806957 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796170    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:34:22.806957 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796173    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:34:22.806957 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796175    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:34:22.806957 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796178    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:34:22.806957 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796181    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:34:22.806957 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796183    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:34:22.806957 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796187    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:34:22.807468 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796190    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:34:22.807468 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796192    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:34:22.807468 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796195    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:34:22.807468 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796197    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:34:22.807468 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796200    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:34:22.807468 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796202    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:34:22.807468 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796205    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:34:22.807468 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796208    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:34:22.807468 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796210    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:34:22.807468 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796213    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:34:22.807468 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796216    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:34:22.807468 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796218    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:34:22.807468 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796221    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:34:22.807468 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796223    2585 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:34:22.807468 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796226    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:34:22.807468 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796228    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:34:22.807468 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796231    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:34:22.807468 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796234    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:34:22.807468 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796236    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:34:22.807468 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796239    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:34:22.808321 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796241    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:34:22.808321 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796245    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:34:22.808321 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796247    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:34:22.808321 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796250    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:34:22.808321 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796252    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:34:22.808321 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796255    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:34:22.808321 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796257    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:34:22.808321 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796260    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:34:22.808321 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796263    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:34:22.808321 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796266    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:34:22.808321 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.796268    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:34:22.808321 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.796971    2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:34:22.808321 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.806462    2585 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 17:34:22.808321 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.806481    2585 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 17:34:22.808321 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.807418    2585 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 17:34:22.808978 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.807438    2585 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 17:34:22.808978 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807527    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:34:22.808978 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807536    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:34:22.808978 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807542    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:34:22.808978 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807547    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:34:22.808978 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807552    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:34:22.808978 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807556    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:34:22.808978 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807561    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:34:22.808978 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807566    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:34:22.808978 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807572    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:34:22.808978 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807577    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:34:22.808978 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807581    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:34:22.808978 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807586    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:34:22.808978 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807590    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:34:22.808978 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807595    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:34:22.808978 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807600    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:34:22.808978 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807604    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:34:22.808978 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807609    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:34:22.808978 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807613    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:34:22.809480 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807617    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:34:22.809480 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807621    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:34:22.809480 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807625    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:34:22.809480 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807629    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:34:22.809480 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807633    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:34:22.809480 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807637    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:34:22.809480 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807641    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:34:22.809480 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807645    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:34:22.809480 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807649    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:34:22.809480 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807653    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:34:22.809480 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807661    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:34:22.809480 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807668    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:34:22.809480 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807673    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:34:22.809480 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807677    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:34:22.809480 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807682    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:34:22.809480 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807686    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:34:22.809480 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807690    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:34:22.809480 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807695    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:34:22.809480 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807700    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:34:22.809480 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807704    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:34:22.810228 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807708    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:34:22.810228 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807712    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:34:22.810228 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807717    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:34:22.810228 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807721    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:34:22.810228 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807725    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:34:22.810228 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807731    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:34:22.810228 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807736    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:34:22.810228 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807741    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:34:22.810228 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807746    2585 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:34:22.810228 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807750    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:34:22.810228 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807755    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:34:22.810228 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807759    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:34:22.810228 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807764    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:34:22.810228 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807769    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:34:22.810228 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807773    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:34:22.810228 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807778    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:34:22.810228 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807782    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:34:22.810228 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807786    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:34:22.810228 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807790    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:34:22.810228 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807794    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:34:22.811051 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807798    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:34:22.811051 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807803    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:34:22.811051 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807807    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:34:22.811051 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807812    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:34:22.811051 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807816    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:34:22.811051 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807820    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:34:22.811051 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807825    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:34:22.811051 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807829    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:34:22.811051 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807834    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:34:22.811051 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807838    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:34:22.811051 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807842    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:34:22.811051 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807846    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:34:22.811051 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807850    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:34:22.811051 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807855    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:34:22.811051 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807859    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:34:22.811051 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807862    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:34:22.811051 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807867    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:34:22.811051 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807871    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:34:22.811051 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807875    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:34:22.811051 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807879    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:34:22.811613 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807883    2585 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:34:22.811613 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807905    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:34:22.811613 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807909    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:34:22.811613 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807913    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:34:22.811613 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807917    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:34:22.811613 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807921    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:34:22.811613 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807924    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:34:22.811613 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807928    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:34:22.811613 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807932    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:34:22.811613 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807937    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:34:22.811613 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807942    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:34:22.811613 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807948    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:34:22.811613 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807955    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:34:22.811613 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807959    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:34:22.811613 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807963    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:34:22.811613 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807968    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:34:22.811613 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807972    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:34:22.811613 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807976    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:34:22.811613 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807980    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:34:22.812223 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807984    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:34:22.812223 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.807988    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:34:22.812223 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.807996    2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:34:22.812223 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808178    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 17:34:22.812223 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808187    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 17:34:22.812223 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808193    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 17:34:22.812223 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808197    2585 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 17:34:22.812223 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808202    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 17:34:22.812223 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808206    2585 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 17:34:22.812223 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808210    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 17:34:22.812223 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808214    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 17:34:22.812223 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808218    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 17:34:22.812223 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808222    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 17:34:22.812223 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808226    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 17:34:22.812223 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808230    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 17:34:22.812592 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808235    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 17:34:22.812592 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808239    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 17:34:22.812592 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808243    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 17:34:22.812592 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808248    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 17:34:22.812592 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808252    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 17:34:22.812592 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808257    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 17:34:22.812592 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808261    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 17:34:22.812592 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808265    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 17:34:22.812592 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808269    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 17:34:22.812592 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808274    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 17:34:22.812592 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808278    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 17:34:22.812592 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808282    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 17:34:22.812592 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808286    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 17:34:22.812592 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808290    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 17:34:22.812592 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808294    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 17:34:22.812592 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808298    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 17:34:22.812592 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808302    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 17:34:22.812592 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808306    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 17:34:22.812592 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808310    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 17:34:22.812592 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808314    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 17:34:22.813257 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808318    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 17:34:22.813257 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808322    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 17:34:22.813257 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808325    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 17:34:22.813257 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808329    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 17:34:22.813257 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808333    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 17:34:22.813257 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808337    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 17:34:22.813257 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808340    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 17:34:22.813257 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808344    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 17:34:22.813257 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808349    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 17:34:22.813257 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808353    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 17:34:22.813257 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808356    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 17:34:22.813257 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808361    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 17:34:22.813257 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808365    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 17:34:22.813257 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808371    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 17:34:22.813257 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808375    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 17:34:22.813257 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808379    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 17:34:22.813257 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808383    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 17:34:22.813257 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808388    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 17:34:22.813257 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808392    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 17:34:22.813257 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808395    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 17:34:22.813757 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808400    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 17:34:22.813757 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808404    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 17:34:22.813757 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808408    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 17:34:22.813757 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808413    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 17:34:22.813757 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808419    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 17:34:22.813757 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808426    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 17:34:22.813757 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808431    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 17:34:22.813757 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808435    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 17:34:22.813757 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808439    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 17:34:22.813757 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808444    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 17:34:22.813757 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808448    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 17:34:22.813757 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808452    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 17:34:22.813757 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808456    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 17:34:22.813757 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808460    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 17:34:22.813757 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808464    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 17:34:22.813757 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808468    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 17:34:22.813757 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808473    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 17:34:22.813757 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808476    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 17:34:22.813757 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808480    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 17:34:22.813757 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808485    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 17:34:22.814292 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808489    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 17:34:22.814292 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808493    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 17:34:22.814292 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808497    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 17:34:22.814292 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808501    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 17:34:22.814292 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808505    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 17:34:22.814292 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808509    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 17:34:22.814292 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808514    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 17:34:22.814292 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808518    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 17:34:22.814292 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808522    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 17:34:22.814292 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808526    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 17:34:22.814292 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808533    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 17:34:22.814292 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808539    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 17:34:22.814292 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808544    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 17:34:22.814292 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808549    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 17:34:22.814292 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808554    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 17:34:22.814292 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808558    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 17:34:22.814292 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808563    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 17:34:22.814292 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808568    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 17:34:22.814729 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808574    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 17:34:22.814729 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808579    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 17:34:22.814729 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808584    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 17:34:22.814729 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808588    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 17:34:22.814729 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808592    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 17:34:22.814729 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808596    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 17:34:22.814729 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808600    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 17:34:22.814729 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808604    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 17:34:22.814729 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:22.808608    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 17:34:22.814729 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.808616    2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 17:34:22.814729 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.808789    2585 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 17:34:22.814729 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.811921    2585 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 17:34:22.814729 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.813057    2585 server.go:1019] "Starting client certificate rotation"
Sep 17 17:34:22.814729 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.813157    2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 17:34:22.815106 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.813187    2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 17:34:22.816019 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.816003    2585 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 17:34:22.817163 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.817142    2585 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 17:34:22.830567 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.830551    2585 log.go:25] "Validated CRI v1 runtime API"
Sep 17 17:34:22.832242 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.832228    2585 log.go:25] "Validated CRI v1 image API"
Sep 17 17:34:22.832996 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.832977    2585 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 17:34:22.836445 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.836427    2585 fs.go:135] Filesystem UUIDs: map[0877ec44-e228-4935-8364-d4e2f435eaa2:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8b528c79-c9a3-4078-bbcd-5b235b3a575f:/dev/nvme0n1p4]
Sep 17 17:34:22.836495 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.836445    2585 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 17:34:22.840443 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.840422    2585 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 17:34:22.843256 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.843154    2585 manager.go:216] Machine: {Timestamp:2026-09-17 17:34:22.841873904 +0000 UTC m=+0.373268035 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3216992 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24d89eb163fb17a32286f34d56368c SystemUUID:ec24d89e-b163-fb17-a322-86f34d56368c BootID:9bb32bff-6b47-4454-b63c-de4f561be0b3 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e4:0f:f3:51 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e4:0f:f3:51 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:36:47:e5:96:78:92 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 17:34:22.843256 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.843254    2585 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 17:34:22.843366 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.843329    2585 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 17:34:22.844562 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.844540    2585 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 17:34:22.844688 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.844566    2585 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-139.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 17:34:22.845412 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.845402    2585 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 17:34:22.845486 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.845415    2585 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 17:34:22.845486 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.845422    2585 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 17:34:22.845486 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.845437    2585 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 17:34:22.845562 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.845504    2585 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 17:34:22.850264 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.850253    2585 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 17:34:22.850378 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.850368    2585 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 17:34:22.850411 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.850398    2585 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 17:34:22.850411 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.850408    2585 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 17:34:22.850466 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.850423    2585 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 17:34:22.850466 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.850433    2585 kubelet.go:398] "Adding apiserver pod source"
Sep 17 17:34:22.850466 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.850441    2585 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 17:34:22.850804 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.850792    2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 17:34:22.850843 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.850817    2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 17:34:22.853110 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.853094    2585 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 17:34:22.856099 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.856084    2585 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 17:34:22.856162 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.856113    2585 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 17:34:22.857475 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.857462    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 17:34:22.857512 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.857480    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 17:34:22.857512 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.857486    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 17:34:22.857512 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.857492    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 17:34:22.857512 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.857501    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 17:34:22.857512 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.857506    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 17:34:22.857638 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.857520    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 17:34:22.857638 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.857529    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 17:34:22.857638 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.857538    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 17:34:22.857638 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.857554    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 17:34:22.857638 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.857568    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 17:34:22.857638 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.857574    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 17:34:22.859299 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.859287    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 17:34:22.859299 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.859298    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 17:34:22.860986 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:22.860964    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-139.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 17:34:22.861054 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:22.860997    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 17:34:22.863115 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.863104    2585 server.go:1268] "Started kubelet"
Sep 17 17:34:22.863222 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.863187    2585 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 17:34:22.863289 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.863214    2585 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 17:34:22.863326 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.863307    2585 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 17:34:22.863970 ip-10-0-133-139 systemd[1]: Started Kubernetes Kubelet.
Sep 17 17:34:22.864823 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.864799    2585 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 17:34:22.866565 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.866548    2585 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 17:34:22.868030 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:22.867883    2585 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 17:34:22.868320 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.868306    2585 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tv6kk"
Sep 17 17:34:22.877984 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:22.873685    2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-139.ec2.internal.18d62c2a7962007b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-139.ec2.internal,UID:ip-10-0-133-139.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-139.ec2.internal,},FirstTimestamp:2026-09-17 17:34:22.863065211 +0000 UTC m=+0.394459342,LastTimestamp:2026-09-17 17:34:22.863065211 +0000 UTC m=+0.394459342,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-139.ec2.internal,}"
Sep 17 17:34:22.878138 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.878121    2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 17:34:22.878205 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.878132    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-139.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 17:34:22.878205 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.878187    2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tv6kk"
Sep 17 17:34:22.878758 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.878743    2585 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 17:34:22.879217 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:22.879084    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-139.ec2.internal\" not found"
Sep 17 17:34:22.879983 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.879966    2585 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 17:34:22.880080 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.879986    2585 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 17:34:22.880481 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.880462    2585 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 17:34:22.880586 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.880539    2585 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 17:34:22.880586 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.880546    2585 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 17:34:22.880903 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.880873    2585 factory.go:55] Registering systemd factory
Sep 17 17:34:22.880981 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.880914    2585 factory.go:223] Registration of the systemd container factory successfully
Sep 17 17:34:22.881232 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.881217    2585 factory.go:153] Registering CRI-O factory
Sep 17 17:34:22.881301 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.881237    2585 factory.go:223] Registration of the crio container factory successfully
Sep 17 17:34:22.881355 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.881321    2585 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 17:34:22.881355 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.881347    2585 factory.go:103] Registering Raw factory
Sep 17 17:34:22.881447 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.881358    2585 manager.go:1196] Started watching for new ooms in manager
Sep 17 17:34:22.882010 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.881996    2585 manager.go:349] Starting recovery of all containers
Sep 17 17:34:22.889047 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.889018    2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 17:34:22.891432 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.891406    2585 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:34:22.895312 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:22.895295    2585 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-133-139.ec2.internal\" not found" node="ip-10-0-133-139.ec2.internal"
Sep 17 17:34:22.895764 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.895751    2585 manager.go:354] Recovery completed
Sep 17 17:34:22.901074 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.901061    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:34:22.902843 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.902824    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-139.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:34:22.902925 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.902859    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-139.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:34:22.902925 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.902871    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-139.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:34:22.903395 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.903378    2585 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 17:34:22.903395 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.903393    2585 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 17:34:22.903527 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.903394    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:34:22.903527 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.903409    2585 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 17:34:22.904133 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.904117    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-139.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:34:22.904192 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.904146    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-139.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:34:22.904192 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.904159    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-139.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:34:22.905427 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.905415    2585 policy_none.go:49] "None policy: Start"
Sep 17 17:34:22.905475 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.905429    2585 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 17:34:22.905475 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.905441    2585 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 17:34:22.907445 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.907434    2585 policy_none.go:47] "Start"
Sep 17 17:34:22.933496 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.933484    2585 manager.go:337] "Starting Device Plugin manager"
Sep 17 17:34:23.041055 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:22.933517    2585 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 17:34:23.041055 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.933526    2585 server.go:87] "Starting device plugin registration server"
Sep 17 17:34:23.041055 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.933855    2585 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 17:34:23.041055 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.933868    2585 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 17:34:23.041055 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.933997    2585 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 17:34:23.041055 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.934072    2585 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 17:34:23.041055 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.934080    2585 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 17:34:23.041055 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:22.935214    2585 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 17:34:23.041055 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:22.935256    2585 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-139.ec2.internal\" not found"
Sep 17 17:34:23.041055 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.981137    2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 17:34:23.041055 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.981182    2585 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 17:34:23.041055 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.981204    2585 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 17:34:23.041055 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:22.981234    2585 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 17:34:23.041055 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:22.983738    2585 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:34:23.041055 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:23.034302    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:34:23.041055 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:23.035176    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-139.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:34:23.041055 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:23.035201    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-139.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:34:23.041055 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:23.035210    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-139.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:34:23.041055 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:23.035233    2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-139.ec2.internal"
Sep 17 17:34:23.044375 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:23.044360    2585 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-139.ec2.internal"
Sep 17 17:34:23.044450 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:23.044380    2585 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-139.ec2.internal\": node \"ip-10-0-133-139.ec2.internal\" not found"
Sep 17 17:34:23.064498 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:23.064478    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-139.ec2.internal\" not found"
Sep 17 17:34:23.081764 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:23.081745    2585 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-139.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-139.ec2.internal"]
Sep 17 17:34:23.081826 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:23.081801    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:34:23.083117 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:23.083102    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-139.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:34:23.083172 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:23.083130    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-139.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:34:23.083172 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:23.083140    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-139.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:34:23.085436 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:23.085425    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:34:23.085549 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:23.085533    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-139.ec2.internal"
Sep 17 17:34:23.085595 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:23.085562    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:34:23.086120 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:23.086104    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-139.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:34:23.086180 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:23.086126    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-139.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:34:23.086180 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:23.086142    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-139.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:34:23.086180 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:23.086104    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-139.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:34:23.086269 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:23.086203    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-139.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:34:23.086269 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:23.086214    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-139.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:34:23.088262 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:23.088247    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-139.ec2.internal"
Sep 17 17:34:23.088299 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:23.088273    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 17:34:23.088978 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:23.088964    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-139.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 17:34:23.089042 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:23.088988    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-139.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 17:34:23.089042 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:23.089002    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-139.ec2.internal" event="NodeHasSufficientPID"
Sep 17 17:34:23.104394 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:23.104374    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-139.ec2.internal\" not found" node="ip-10-0-133-139.ec2.internal"
Sep 17 17:34:23.108615 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:23.108599    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-139.ec2.internal\" not found" node="ip-10-0-133-139.ec2.internal"
Sep 17 17:34:23.165279 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:23.165260    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-139.ec2.internal\" not found"
Sep 17 17:34:23.181853 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:23.181833    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/359ad2cc4231b18d20008ab45c907b6d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-139.ec2.internal\" (UID: \"359ad2cc4231b18d20008ab45c907b6d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-139.ec2.internal"
Sep 17 17:34:23.181917 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:23.181858    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/359ad2cc4231b18d20008ab45c907b6d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-139.ec2.internal\" (UID: \"359ad2cc4231b18d20008ab45c907b6d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-139.ec2.internal"
Sep 17 17:34:23.181917 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:23.181874    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/aba6d40ed12a7528810e76c18b9ddea9-config\") pod \"kube-apiserver-proxy-ip-10-0-133-139.ec2.internal\" (UID: \"aba6d40ed12a7528810e76c18b9ddea9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-139.ec2.internal"
Sep 17 17:34:23.265487 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:23.265468    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-139.ec2.internal\" not found"
Sep 17 17:34:23.282743 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:23.282727    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/359ad2cc4231b18d20008ab45c907b6d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-139.ec2.internal\" (UID: \"359ad2cc4231b18d20008ab45c907b6d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-139.ec2.internal"
Sep 17 17:34:23.282792 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:23.282749    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/359ad2cc4231b18d20008ab45c907b6d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-139.ec2.internal\" (UID: \"359ad2cc4231b18d20008ab45c907b6d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-139.ec2.internal"
Sep 17 17:34:23.282792 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:23.282765    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/aba6d40ed12a7528810e76c18b9ddea9-config\") pod \"kube-apiserver-proxy-ip-10-0-133-139.ec2.internal\" (UID: \"aba6d40ed12a7528810e76c18b9ddea9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-139.ec2.internal"
Sep 17 17:34:23.282852 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:23.282822    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/aba6d40ed12a7528810e76c18b9ddea9-config\") pod \"kube-apiserver-proxy-ip-10-0-133-139.ec2.internal\" (UID: \"aba6d40ed12a7528810e76c18b9ddea9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-139.ec2.internal"
Sep 17 17:34:23.282852 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:23.282833    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/359ad2cc4231b18d20008ab45c907b6d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-139.ec2.internal\" (UID: \"359ad2cc4231b18d20008ab45c907b6d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-139.ec2.internal"
Sep 17 17:34:23.282923 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:23.282839    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/359ad2cc4231b18d20008ab45c907b6d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-139.ec2.internal\" (UID: \"359ad2cc4231b18d20008ab45c907b6d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-139.ec2.internal"
Sep 17 17:34:23.366208 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:23.366141    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-139.ec2.internal\" not found"
Sep 17 17:34:23.408576 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:23.408547    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-139.ec2.internal"
Sep 17 17:34:23.412376 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:23.412362    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-139.ec2.internal"
Sep 17 17:34:23.466620 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:23.466591    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-139.ec2.internal\" not found"
Sep 17 17:34:23.567060 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:23.567030    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-139.ec2.internal\" not found"
Sep 17 17:34:23.667531 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:23.667503    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-139.ec2.internal\" not found"
Sep 17 17:34:23.767957 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:23.767931    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-139.ec2.internal\" not found"
Sep 17 17:34:24.930981 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:24.930956    2585 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 17:34:24.931409 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:24.931089    2585 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 17:34:24.931409 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:24.931108    2585 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 17:34:24.986386 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:24.986366    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-139.ec2.internal\" not found"
Sep 17 17:34:25.003556 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:25.003526    2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 17:29:23 +0000 UTC" deadline="2028-05-29 16:36:48.117177674 +0000 UTC"
Sep 17 17:34:25.003556 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:25.003553    2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14879h2m23.113627575s"
Sep 17 17:34:25.086755 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:25.086697    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-139.ec2.internal\" not found"
Sep 17 17:34:25.108527 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:25.108480    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaba6d40ed12a7528810e76c18b9ddea9.slice/crio-862fcf55cc5761e4abf49caa25de3c68ad6bcc46864674cc20ea8826dce99700 WatchSource:0}: Error finding container 862fcf55cc5761e4abf49caa25de3c68ad6bcc46864674cc20ea8826dce99700: Status 404 returned error can't find the container with id 862fcf55cc5761e4abf49caa25de3c68ad6bcc46864674cc20ea8826dce99700
Sep 17 17:34:25.109589 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:25.109564    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod359ad2cc4231b18d20008ab45c907b6d.slice/crio-edd4c47f3bf070aa4d79ee2e6449874679aed0c9198490f1819658be9ae194ed WatchSource:0}: Error finding container edd4c47f3bf070aa4d79ee2e6449874679aed0c9198490f1819658be9ae194ed: Status 404 returned error can't find the container with id edd4c47f3bf070aa4d79ee2e6449874679aed0c9198490f1819658be9ae194ed
Sep 17 17:34:25.114097 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:25.114081    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:34:25.187266 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:25.187244    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-139.ec2.internal\" not found"
Sep 17 17:34:25.216276 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:25.216258    2585 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:34:25.287524 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:25.287479    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-139.ec2.internal\" not found"
Sep 17 17:34:25.297895 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:25.297877    2585 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:34:25.299245 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:25.299233    2585 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-139.ec2.internal"
Sep 17 17:34:25.315394 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:25.315373    2585 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 17:34:25.316446 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:25.316431    2585 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-139.ec2.internal"
Sep 17 17:34:25.327731 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:25.327714    2585 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 17:34:25.746479 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:25.746449    2585 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:34:25.829125 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:25.828967    2585 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 17:34:25.968949 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:25.968919    2585 apiserver.go:52] "Watching apiserver"
Sep 17 17:34:25.976298 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:25.976277    2585 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 17:34:25.977548 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:25.977526    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-mglss","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-139.ec2.internal","openshift-multus/multus-additional-cni-plugins-9tfrw","openshift-multus/multus-d972p","openshift-network-diagnostics/network-check-target-z8nx5","kube-system/konnectivity-agent-jpg4z","openshift-cluster-node-tuning-operator/tuned-pcc9f","openshift-multus/network-metrics-daemon-p7vh6","openshift-network-operator/iptables-alerter-lt9dx","openshift-ovn-kubernetes/ovnkube-node-rrf79","kube-system/global-pull-secret-syncer-7spmx","kube-system/kube-apiserver-proxy-ip-10-0-133-139.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgp7g"]
Sep 17 17:34:25.979989 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:25.979967    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pcc9f"
Sep 17 17:34:25.982216 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:25.982192    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9tfrw"
Sep 17 17:34:25.984273 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:25.984256    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 17:34:25.984727 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:25.984711    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-d972p"
Sep 17 17:34:25.986318 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:25.986298    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 17:34:25.987047 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:25.986842    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z8nx5"
Sep 17 17:34:25.987047 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:25.986914    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z8nx5" podUID="7bcabfd1-e72e-4866-9190-6bcce1f7ce64"
Sep 17 17:34:25.988446 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:25.988425    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 17:34:25.988721 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:25.988702    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 17:34:25.988810 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:25.988723    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-dw7vm\""
Sep 17 17:34:25.988810 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:25.988745    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:34:25.988810 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:25.988758    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 17:34:25.988991 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:25.988702    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 17:34:25.988991 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:25.988763    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 17:34:25.990602 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:25.990580    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-fpbpx\""
Sep 17 17:34:25.990801 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:25.990786    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jpg4z"
Sep 17 17:34:25.991892 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:25.991873    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6n86b\""
Sep 17 17:34:25.993149 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:25.993130    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mglss"
Sep 17 17:34:25.993643 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:25.993620    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4r7lt\""
Sep 17 17:34:25.993824 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:25.993803    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 17:34:25.993996 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:25.993980    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 17:34:25.995338 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:25.995313    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 17:34:25.995908 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:25.995580    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p7vh6"
Sep 17 17:34:25.995908 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:25.995659    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p7vh6" podUID="39af04b6-713b-406b-952d-7b2ef2b2b952"
Sep 17 17:34:25.997089 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:25.997039    2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 17:34:25.997303 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:25.997286    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 17:34:25.997755 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:25.997739    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lt9dx"
Sep 17 17:34:25.998198 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:25.998178    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 17:34:26.000248 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.000229    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.000358 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.000338    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-px5tz\""
Sep 17 17:34:26.000423 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.000413    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 17:34:26.000589 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.000571    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 17:34:26.000724 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.000710    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 17:34:26.001196 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.001177    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-r2m68\""
Sep 17 17:34:26.002579 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.002562    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7spmx"
Sep 17 17:34:26.002670 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:26.002644    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7spmx" podUID="dbb6518b-5a9b-4a33-89a1-71c9e4db70a5"
Sep 17 17:34:26.004944 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.004925    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgp7g"
Sep 17 17:34:26.005701 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.005685    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 17:34:26.005788 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.005774    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 17:34:26.006190 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.006172    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 17:34:26.007238 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.007220    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-d8qj5\""
Sep 17 17:34:26.007334 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.007254    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 17:34:26.007446 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.007429    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 17:34:26.007540 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.007475    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 17:34:26.007591 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.007537    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 17:34:26.008207 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.008186    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 17:34:26.008373 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.008294    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 17:34:26.008501 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.008421    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7ws8d\""
Sep 17 17:34:26.009977 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.009959    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ef20182-de84-4cba-9eb4-1fcfdfbd1abe-system-cni-dir\") pod \"multus-d972p\" (UID: \"9ef20182-de84-4cba-9eb4-1fcfdfbd1abe\") " pod="openshift-multus/multus-d972p"
Sep 17 17:34:26.010062 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.009986    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/751c4ff3-f343-4280-b970-3583c2a4c28b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9tfrw\" (UID: \"751c4ff3-f343-4280-b970-3583c2a4c28b\") " pod="openshift-multus/multus-additional-cni-plugins-9tfrw"
Sep 17 17:34:26.010062 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.010002    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/51e093e4-5ec3-4086-985f-3022f731690b-etc-sysconfig\") pod \"tuned-pcc9f\" (UID: \"51e093e4-5ec3-4086-985f-3022f731690b\") " pod="openshift-cluster-node-tuning-operator/tuned-pcc9f"
Sep 17 17:34:26.010062 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.010021    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/51e093e4-5ec3-4086-985f-3022f731690b-etc-sysctl-d\") pod \"tuned-pcc9f\" (UID: \"51e093e4-5ec3-4086-985f-3022f731690b\") " pod="openshift-cluster-node-tuning-operator/tuned-pcc9f"
Sep 17 17:34:26.010062 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.010042    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/51e093e4-5ec3-4086-985f-3022f731690b-host\") pod \"tuned-pcc9f\" (UID: \"51e093e4-5ec3-4086-985f-3022f731690b\") " pod="openshift-cluster-node-tuning-operator/tuned-pcc9f"
Sep 17 17:34:26.010251 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.010073    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5-host-slash\") pod \"ovnkube-node-rrf79\" (UID: \"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.010251 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.010092    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5-env-overrides\") pod \"ovnkube-node-rrf79\" (UID: \"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.010251 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.010108    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5-ovn-node-metrics-cert\") pod \"ovnkube-node-rrf79\" (UID: \"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.010251 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.010125    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/51e093e4-5ec3-4086-985f-3022f731690b-etc-modprobe-d\") pod \"tuned-pcc9f\" (UID: \"51e093e4-5ec3-4086-985f-3022f731690b\") " pod="openshift-cluster-node-tuning-operator/tuned-pcc9f"
Sep 17 17:34:26.010251 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.010155    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51e093e4-5ec3-4086-985f-3022f731690b-var-lib-kubelet\") pod \"tuned-pcc9f\" (UID: \"51e093e4-5ec3-4086-985f-3022f731690b\") " pod="openshift-cluster-node-tuning-operator/tuned-pcc9f"
Sep 17 17:34:26.010251 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.010176    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6bq4\" (UniqueName: \"kubernetes.io/projected/51e093e4-5ec3-4086-985f-3022f731690b-kube-api-access-c6bq4\") pod \"tuned-pcc9f\" (UID: \"51e093e4-5ec3-4086-985f-3022f731690b\") " pod="openshift-cluster-node-tuning-operator/tuned-pcc9f"
Sep 17 17:34:26.010251 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.010190    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5-node-log\") pod \"ovnkube-node-rrf79\" (UID: \"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.010251 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.010209    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5-ovnkube-script-lib\") pod \"ovnkube-node-rrf79\" (UID: \"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.010251 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.010223    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9ef20182-de84-4cba-9eb4-1fcfdfbd1abe-cni-binary-copy\") pod \"multus-d972p\" (UID: \"9ef20182-de84-4cba-9eb4-1fcfdfbd1abe\") " pod="openshift-multus/multus-d972p"
Sep 17 17:34:26.010251 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.010236    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9ef20182-de84-4cba-9eb4-1fcfdfbd1abe-host-run-netns\") pod \"multus-d972p\" (UID: \"9ef20182-de84-4cba-9eb4-1fcfdfbd1abe\") " pod="openshift-multus/multus-d972p"
Sep 17 17:34:26.010251 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.010254    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9ef20182-de84-4cba-9eb4-1fcfdfbd1abe-hostroot\") pod \"multus-d972p\" (UID: \"9ef20182-de84-4cba-9eb4-1fcfdfbd1abe\") " pod="openshift-multus/multus-d972p"
Sep 17 17:34:26.010795 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.010324    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/856535bd-6884-4e98-8089-852deae228aa-konnectivity-ca\") pod \"konnectivity-agent-jpg4z\" (UID: \"856535bd-6884-4e98-8089-852deae228aa\") " pod="kube-system/konnectivity-agent-jpg4z"
Sep 17 17:34:26.010795 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.010368    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e8bda048-e7cf-4118-82c4-bd1d987e4566-host\") pod \"node-ca-mglss\" (UID: \"e8bda048-e7cf-4118-82c4-bd1d987e4566\") " pod="openshift-image-registry/node-ca-mglss"
Sep 17 17:34:26.010795 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.010411    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9ef20182-de84-4cba-9eb4-1fcfdfbd1abe-host-run-k8s-cni-cncf-io\") pod \"multus-d972p\" (UID: \"9ef20182-de84-4cba-9eb4-1fcfdfbd1abe\") " pod="openshift-multus/multus-d972p"
Sep 17 17:34:26.010795 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.010460    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qpqn\" (UniqueName: \"kubernetes.io/projected/39af04b6-713b-406b-952d-7b2ef2b2b952-kube-api-access-7qpqn\") pod \"network-metrics-daemon-p7vh6\" (UID: \"39af04b6-713b-406b-952d-7b2ef2b2b952\") " pod="openshift-multus/network-metrics-daemon-p7vh6"
Sep 17 17:34:26.010795 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.010509    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/751c4ff3-f343-4280-b970-3583c2a4c28b-cnibin\") pod \"multus-additional-cni-plugins-9tfrw\" (UID: \"751c4ff3-f343-4280-b970-3583c2a4c28b\") " pod="openshift-multus/multus-additional-cni-plugins-9tfrw"
Sep 17 17:34:26.010795 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.010541    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a40b9f88-37de-405c-9702-3a73548b4f63-host-slash\") pod \"iptables-alerter-lt9dx\" (UID: \"a40b9f88-37de-405c-9702-3a73548b4f63\") " pod="openshift-network-operator/iptables-alerter-lt9dx"
Sep 17 17:34:26.010795 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.010569    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjcbl\" (UniqueName: \"kubernetes.io/projected/a40b9f88-37de-405c-9702-3a73548b4f63-kube-api-access-tjcbl\") pod \"iptables-alerter-lt9dx\" (UID: \"a40b9f88-37de-405c-9702-3a73548b4f63\") " pod="openshift-network-operator/iptables-alerter-lt9dx"
Sep 17 17:34:26.010795 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.010625    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5-host-run-netns\") pod \"ovnkube-node-rrf79\" (UID: \"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.010795 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.010657    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5-host-cni-bin\") pod \"ovnkube-node-rrf79\" (UID: \"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.010795 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.010701    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/856535bd-6884-4e98-8089-852deae228aa-agent-certs\") pod \"konnectivity-agent-jpg4z\" (UID: \"856535bd-6884-4e98-8089-852deae228aa\") " pod="kube-system/konnectivity-agent-jpg4z"
Sep 17 17:34:26.010795 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.010734    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/51e093e4-5ec3-4086-985f-3022f731690b-tmp\") pod \"tuned-pcc9f\" (UID: \"51e093e4-5ec3-4086-985f-3022f731690b\") " pod="openshift-cluster-node-tuning-operator/tuned-pcc9f"
Sep 17 17:34:26.010795 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.010766    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/751c4ff3-f343-4280-b970-3583c2a4c28b-system-cni-dir\") pod \"multus-additional-cni-plugins-9tfrw\" (UID: \"751c4ff3-f343-4280-b970-3583c2a4c28b\") " pod="openshift-multus/multus-additional-cni-plugins-9tfrw"
Sep 17 17:34:26.010795 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.010805    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hsct\" (UniqueName: \"kubernetes.io/projected/751c4ff3-f343-4280-b970-3583c2a4c28b-kube-api-access-8hsct\") pod \"multus-additional-cni-plugins-9tfrw\" (UID: \"751c4ff3-f343-4280-b970-3583c2a4c28b\") " pod="openshift-multus/multus-additional-cni-plugins-9tfrw"
Sep 17 17:34:26.011455 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.010869    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pxj5\" (UniqueName: \"kubernetes.io/projected/e8bda048-e7cf-4118-82c4-bd1d987e4566-kube-api-access-5pxj5\") pod \"node-ca-mglss\" (UID: \"e8bda048-e7cf-4118-82c4-bd1d987e4566\") " pod="openshift-image-registry/node-ca-mglss"
Sep 17 17:34:26.011455 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.010898    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5-run-ovn\") pod \"ovnkube-node-rrf79\" (UID: \"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.011455 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.010920    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pr56\" (UniqueName: \"kubernetes.io/projected/beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5-kube-api-access-5pr56\") pod \"ovnkube-node-rrf79\" (UID: \"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.011455 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.010954    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39af04b6-713b-406b-952d-7b2ef2b2b952-metrics-certs\") pod \"network-metrics-daemon-p7vh6\" (UID: \"39af04b6-713b-406b-952d-7b2ef2b2b952\") " pod="openshift-multus/network-metrics-daemon-p7vh6"
Sep 17 17:34:26.011455 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.010980    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rlql\" (UniqueName: \"kubernetes.io/projected/7bcabfd1-e72e-4866-9190-6bcce1f7ce64-kube-api-access-4rlql\") pod \"network-check-target-z8nx5\" (UID: \"7bcabfd1-e72e-4866-9190-6bcce1f7ce64\") " pod="openshift-network-diagnostics/network-check-target-z8nx5"
Sep 17 17:34:26.011455 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.011004    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5-run-systemd\") pod \"ovnkube-node-rrf79\" (UID: \"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.011455 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.011026    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5-run-openvswitch\") pod \"ovnkube-node-rrf79\" (UID: \"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.011455 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.011048    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5-host-run-ovn-kubernetes\") pod \"ovnkube-node-rrf79\" (UID: \"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.011455 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.011062    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5-host-cni-netd\") pod \"ovnkube-node-rrf79\" (UID: \"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.011455 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.011075    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9ef20182-de84-4cba-9eb4-1fcfdfbd1abe-multus-daemon-config\") pod \"multus-d972p\" (UID: \"9ef20182-de84-4cba-9eb4-1fcfdfbd1abe\") " pod="openshift-multus/multus-d972p"
Sep 17 17:34:26.011455 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.011088    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/751c4ff3-f343-4280-b970-3583c2a4c28b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9tfrw\" (UID: \"751c4ff3-f343-4280-b970-3583c2a4c28b\") " pod="openshift-multus/multus-additional-cni-plugins-9tfrw"
Sep 17 17:34:26.011455 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.011107    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/51e093e4-5ec3-4086-985f-3022f731690b-sys\") pod \"tuned-pcc9f\" (UID: \"51e093e4-5ec3-4086-985f-3022f731690b\") " pod="openshift-cluster-node-tuning-operator/tuned-pcc9f"
Sep 17 17:34:26.011455 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.011152    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5-systemd-units\") pod \"ovnkube-node-rrf79\" (UID: \"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.011455 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.011184    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5-log-socket\") pod \"ovnkube-node-rrf79\" (UID: \"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.011455 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.011226    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9ef20182-de84-4cba-9eb4-1fcfdfbd1abe-multus-conf-dir\") pod \"multus-d972p\" (UID: \"9ef20182-de84-4cba-9eb4-1fcfdfbd1abe\") " pod="openshift-multus/multus-d972p"
Sep 17 17:34:26.011455 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.011249    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/51e093e4-5ec3-4086-985f-3022f731690b-lib-modules\") pod \"tuned-pcc9f\" (UID: \"51e093e4-5ec3-4086-985f-3022f731690b\") " pod="openshift-cluster-node-tuning-operator/tuned-pcc9f"
Sep 17 17:34:26.012174 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.011271    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/51e093e4-5ec3-4086-985f-3022f731690b-etc-tuned\") pod \"tuned-pcc9f\" (UID: \"51e093e4-5ec3-4086-985f-3022f731690b\") " pod="openshift-cluster-node-tuning-operator/tuned-pcc9f"
Sep 17 17:34:26.012174 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.011292    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5-etc-openvswitch\") pod \"ovnkube-node-rrf79\" (UID: \"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.012174 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.011329    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/751c4ff3-f343-4280-b970-3583c2a4c28b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9tfrw\" (UID: \"751c4ff3-f343-4280-b970-3583c2a4c28b\") " pod="openshift-multus/multus-additional-cni-plugins-9tfrw"
Sep 17 17:34:26.012174 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.011375    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ef20182-de84-4cba-9eb4-1fcfdfbd1abe-etc-kubernetes\") pod \"multus-d972p\" (UID: \"9ef20182-de84-4cba-9eb4-1fcfdfbd1abe\") " pod="openshift-multus/multus-d972p"
Sep 17 17:34:26.012174 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.011402    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ef20182-de84-4cba-9eb4-1fcfdfbd1abe-host-var-lib-kubelet\") pod \"multus-d972p\" (UID: \"9ef20182-de84-4cba-9eb4-1fcfdfbd1abe\") " pod="openshift-multus/multus-d972p"
Sep 17 17:34:26.012174 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.011425    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7pnj\" (UniqueName: \"kubernetes.io/projected/9ef20182-de84-4cba-9eb4-1fcfdfbd1abe-kube-api-access-f7pnj\") pod \"multus-d972p\" (UID: \"9ef20182-de84-4cba-9eb4-1fcfdfbd1abe\") " pod="openshift-multus/multus-d972p"
Sep 17 17:34:26.012174 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.011450    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/51e093e4-5ec3-4086-985f-3022f731690b-etc-sysctl-conf\") pod \"tuned-pcc9f\" (UID: \"51e093e4-5ec3-4086-985f-3022f731690b\") " pod="openshift-cluster-node-tuning-operator/tuned-pcc9f"
Sep 17 17:34:26.012174 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.011470    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/51e093e4-5ec3-4086-985f-3022f731690b-etc-systemd\") pod \"tuned-pcc9f\" (UID: \"51e093e4-5ec3-4086-985f-3022f731690b\") " pod="openshift-cluster-node-tuning-operator/tuned-pcc9f"
Sep 17 17:34:26.012174 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.011513    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e8bda048-e7cf-4118-82c4-bd1d987e4566-serviceca\") pod \"node-ca-mglss\" (UID: \"e8bda048-e7cf-4118-82c4-bd1d987e4566\") " pod="openshift-image-registry/node-ca-mglss"
Sep 17 17:34:26.012174 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.011538    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a40b9f88-37de-405c-9702-3a73548b4f63-iptables-alerter-script\") pod \"iptables-alerter-lt9dx\" (UID: \"a40b9f88-37de-405c-9702-3a73548b4f63\") " pod="openshift-network-operator/iptables-alerter-lt9dx"
Sep 17 17:34:26.012174 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.011563    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rrf79\" (UID: \"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.012174 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.011588    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51e093e4-5ec3-4086-985f-3022f731690b-etc-kubernetes\") pod \"tuned-pcc9f\" (UID: \"51e093e4-5ec3-4086-985f-3022f731690b\") " pod="openshift-cluster-node-tuning-operator/tuned-pcc9f"
Sep 17 17:34:26.012174 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.011608    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5-host-kubelet\") pod \"ovnkube-node-rrf79\" (UID: \"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.012174 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.011629    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ef20182-de84-4cba-9eb4-1fcfdfbd1abe-multus-cni-dir\") pod \"multus-d972p\" (UID: \"9ef20182-de84-4cba-9eb4-1fcfdfbd1abe\") " pod="openshift-multus/multus-d972p"
Sep 17 17:34:26.012174 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.011653    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9ef20182-de84-4cba-9eb4-1fcfdfbd1abe-host-var-lib-cni-bin\") pod \"multus-d972p\" (UID: \"9ef20182-de84-4cba-9eb4-1fcfdfbd1abe\") " pod="openshift-multus/multus-d972p"
Sep 17 17:34:26.012174 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.011683    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9ef20182-de84-4cba-9eb4-1fcfdfbd1abe-host-var-lib-cni-multus\") pod \"multus-d972p\" (UID: \"9ef20182-de84-4cba-9eb4-1fcfdfbd1abe\") " pod="openshift-multus/multus-d972p"
Sep 17 17:34:26.012821 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.011708    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5-var-lib-openvswitch\") pod \"ovnkube-node-rrf79\" (UID: \"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.012821 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.011729    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9ef20182-de84-4cba-9eb4-1fcfdfbd1abe-cnibin\") pod \"multus-d972p\" (UID: \"9ef20182-de84-4cba-9eb4-1fcfdfbd1abe\") " pod="openshift-multus/multus-d972p"
Sep 17 17:34:26.012821 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.011751    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/51e093e4-5ec3-4086-985f-3022f731690b-run\") pod \"tuned-pcc9f\" (UID: \"51e093e4-5ec3-4086-985f-3022f731690b\") " pod="openshift-cluster-node-tuning-operator/tuned-pcc9f"
Sep 17 17:34:26.012821 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.011775    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/751c4ff3-f343-4280-b970-3583c2a4c28b-cni-binary-copy\") pod \"multus-additional-cni-plugins-9tfrw\" (UID: \"751c4ff3-f343-4280-b970-3583c2a4c28b\") " pod="openshift-multus/multus-additional-cni-plugins-9tfrw"
Sep 17 17:34:26.012821 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.011788    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5-ovnkube-config\") pod \"ovnkube-node-rrf79\" (UID: \"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.012821 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.011806    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9ef20182-de84-4cba-9eb4-1fcfdfbd1abe-multus-socket-dir-parent\") pod \"multus-d972p\" (UID: \"9ef20182-de84-4cba-9eb4-1fcfdfbd1abe\") " pod="openshift-multus/multus-d972p"
Sep 17 17:34:26.012821 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.011837    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9ef20182-de84-4cba-9eb4-1fcfdfbd1abe-host-run-multus-certs\") pod \"multus-d972p\" (UID: \"9ef20182-de84-4cba-9eb4-1fcfdfbd1abe\") " pod="openshift-multus/multus-d972p"
Sep 17 17:34:26.014212 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.014190    2585 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 17:34:26.046814 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.046788    2585 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-26zvs"
Sep 17 17:34:26.057360 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.057336    2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-26zvs"
Sep 17 17:34:26.099715 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.099684    2585 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 17:34:26.104299 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.104246    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-139.ec2.internal" event={"ID":"aba6d40ed12a7528810e76c18b9ddea9","Type":"ContainerStarted","Data":"862fcf55cc5761e4abf49caa25de3c68ad6bcc46864674cc20ea8826dce99700"}
Sep 17 17:34:26.105432 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.105409    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-139.ec2.internal" event={"ID":"359ad2cc4231b18d20008ab45c907b6d","Type":"ContainerStarted","Data":"edd4c47f3bf070aa4d79ee2e6449874679aed0c9198490f1819658be9ae194ed"}
Sep 17 17:34:26.112688 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.112668    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5-host-cni-bin\") pod \"ovnkube-node-rrf79\" (UID: \"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.112801 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.112695    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/856535bd-6884-4e98-8089-852deae228aa-agent-certs\") pod \"konnectivity-agent-jpg4z\" (UID: \"856535bd-6884-4e98-8089-852deae228aa\") " pod="kube-system/konnectivity-agent-jpg4z"
Sep 17 17:34:26.112801 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.112717    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/51e093e4-5ec3-4086-985f-3022f731690b-tmp\") pod \"tuned-pcc9f\" (UID: \"51e093e4-5ec3-4086-985f-3022f731690b\") " pod="openshift-cluster-node-tuning-operator/tuned-pcc9f"
Sep 17 17:34:26.112801 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.112741    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dbb6518b-5a9b-4a33-89a1-71c9e4db70a5-dbus\") pod \"global-pull-secret-syncer-7spmx\" (UID: \"dbb6518b-5a9b-4a33-89a1-71c9e4db70a5\") " pod="kube-system/global-pull-secret-syncer-7spmx"
Sep 17 17:34:26.112801 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.112762    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/751c4ff3-f343-4280-b970-3583c2a4c28b-system-cni-dir\") pod \"multus-additional-cni-plugins-9tfrw\" (UID: \"751c4ff3-f343-4280-b970-3583c2a4c28b\") " pod="openshift-multus/multus-additional-cni-plugins-9tfrw"
Sep 17 17:34:26.112801 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.112787    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hsct\" (UniqueName: \"kubernetes.io/projected/751c4ff3-f343-4280-b970-3583c2a4c28b-kube-api-access-8hsct\") pod \"multus-additional-cni-plugins-9tfrw\" (UID: \"751c4ff3-f343-4280-b970-3583c2a4c28b\") " pod="openshift-multus/multus-additional-cni-plugins-9tfrw"
Sep 17 17:34:26.112972 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.112809    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pxj5\" (UniqueName: \"kubernetes.io/projected/e8bda048-e7cf-4118-82c4-bd1d987e4566-kube-api-access-5pxj5\") pod \"node-ca-mglss\" (UID: \"e8bda048-e7cf-4118-82c4-bd1d987e4566\") " pod="openshift-image-registry/node-ca-mglss"
Sep 17 17:34:26.112972 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.112830    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5-run-ovn\") pod \"ovnkube-node-rrf79\" (UID: \"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.112972 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.112832    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5-host-cni-bin\") pod \"ovnkube-node-rrf79\" (UID: \"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.112972 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.112851    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pr56\" (UniqueName: \"kubernetes.io/projected/beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5-kube-api-access-5pr56\") pod \"ovnkube-node-rrf79\" (UID: \"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.112972 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.112872    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39af04b6-713b-406b-952d-7b2ef2b2b952-metrics-certs\") pod \"network-metrics-daemon-p7vh6\" (UID: \"39af04b6-713b-406b-952d-7b2ef2b2b952\") " pod="openshift-multus/network-metrics-daemon-p7vh6"
Sep 17 17:34:26.112972 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:26.112956    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:26.113137 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.113022    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/751c4ff3-f343-4280-b970-3583c2a4c28b-system-cni-dir\") pod \"multus-additional-cni-plugins-9tfrw\" (UID: \"751c4ff3-f343-4280-b970-3583c2a4c28b\") " pod="openshift-multus/multus-additional-cni-plugins-9tfrw"
Sep 17 17:34:26.113137 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:26.113039    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39af04b6-713b-406b-952d-7b2ef2b2b952-metrics-certs podName:39af04b6-713b-406b-952d-7b2ef2b2b952 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:26.612999325 +0000 UTC m=+3.026789761 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/39af04b6-713b-406b-952d-7b2ef2b2b952-metrics-certs") pod "network-metrics-daemon-p7vh6" (UID: "39af04b6-713b-406b-952d-7b2ef2b2b952") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:26.113137 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.113066    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fced737e-fcb0-4559-b00b-e4813b035223-device-dir\") pod \"aws-ebs-csi-driver-node-cgp7g\" (UID: \"fced737e-fcb0-4559-b00b-e4813b035223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgp7g"
Sep 17 17:34:26.113137 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.113096    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rlql\" (UniqueName: \"kubernetes.io/projected/7bcabfd1-e72e-4866-9190-6bcce1f7ce64-kube-api-access-4rlql\") pod \"network-check-target-z8nx5\" (UID: \"7bcabfd1-e72e-4866-9190-6bcce1f7ce64\") " pod="openshift-network-diagnostics/network-check-target-z8nx5"
Sep 17 17:34:26.113137 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.113113    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5-run-systemd\") pod \"ovnkube-node-rrf79\" (UID: \"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.113285 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.113148    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5-run-systemd\") pod \"ovnkube-node-rrf79\" (UID: \"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.113285 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.113165    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5-run-openvswitch\") pod \"ovnkube-node-rrf79\" (UID: \"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.113285 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.113185    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5-host-run-ovn-kubernetes\") pod \"ovnkube-node-rrf79\" (UID: \"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.113285 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.113199    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5-host-cni-netd\") pod \"ovnkube-node-rrf79\" (UID: \"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.113285 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.113226    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5-run-openvswitch\") pod \"ovnkube-node-rrf79\" (UID: \"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.113285 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.113243    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9ef20182-de84-4cba-9eb4-1fcfdfbd1abe-multus-daemon-config\") pod \"multus-d972p\" (UID: \"9ef20182-de84-4cba-9eb4-1fcfdfbd1abe\") " pod="openshift-multus/multus-d972p"
Sep 17 17:34:26.113285 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.113244    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5-host-run-ovn-kubernetes\") pod \"ovnkube-node-rrf79\" (UID: \"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.113527 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.113284    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5-host-cni-netd\") pod \"ovnkube-node-rrf79\" (UID: \"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.113527 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.113287    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5-run-ovn\") pod \"ovnkube-node-rrf79\" (UID: \"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.113527 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.113300    2585 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 17:34:26.113527 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.113309    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fced737e-fcb0-4559-b00b-e4813b035223-registration-dir\") pod \"aws-ebs-csi-driver-node-cgp7g\" (UID: \"fced737e-fcb0-4559-b00b-e4813b035223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgp7g"
Sep 17 17:34:26.113527 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.113421    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/751c4ff3-f343-4280-b970-3583c2a4c28b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9tfrw\" (UID: \"751c4ff3-f343-4280-b970-3583c2a4c28b\") " pod="openshift-multus/multus-additional-cni-plugins-9tfrw"
Sep 17 17:34:26.113527 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.113438    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/51e093e4-5ec3-4086-985f-3022f731690b-sys\") pod \"tuned-pcc9f\" (UID: \"51e093e4-5ec3-4086-985f-3022f731690b\") " pod="openshift-cluster-node-tuning-operator/tuned-pcc9f"
Sep 17 17:34:26.113527 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.113453    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5-systemd-units\") pod \"ovnkube-node-rrf79\" (UID: \"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.113527 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.113472    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5-log-socket\") pod \"ovnkube-node-rrf79\" (UID: \"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.113527 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.113510    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9ef20182-de84-4cba-9eb4-1fcfdfbd1abe-multus-conf-dir\") pod \"multus-d972p\" (UID: \"9ef20182-de84-4cba-9eb4-1fcfdfbd1abe\") " pod="openshift-multus/multus-d972p"
Sep 17 17:34:26.113907 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.113530    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/51e093e4-5ec3-4086-985f-3022f731690b-sys\") pod \"tuned-pcc9f\" (UID: \"51e093e4-5ec3-4086-985f-3022f731690b\") " pod="openshift-cluster-node-tuning-operator/tuned-pcc9f"
Sep 17 17:34:26.113907 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.113536    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dbb6518b-5a9b-4a33-89a1-71c9e4db70a5-original-pull-secret\") pod \"global-pull-secret-syncer-7spmx\" (UID: \"dbb6518b-5a9b-4a33-89a1-71c9e4db70a5\") " pod="kube-system/global-pull-secret-syncer-7spmx"
Sep 17 17:34:26.113907 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.113561    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/51e093e4-5ec3-4086-985f-3022f731690b-lib-modules\") pod \"tuned-pcc9f\" (UID: \"51e093e4-5ec3-4086-985f-3022f731690b\") " pod="openshift-cluster-node-tuning-operator/tuned-pcc9f"
Sep 17 17:34:26.113907 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.113664    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/51e093e4-5ec3-4086-985f-3022f731690b-etc-tuned\") pod \"tuned-pcc9f\" (UID: \"51e093e4-5ec3-4086-985f-3022f731690b\") " pod="openshift-cluster-node-tuning-operator/tuned-pcc9f"
Sep 17 17:34:26.113907 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.113708    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5-etc-openvswitch\") pod \"ovnkube-node-rrf79\" (UID: \"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.113907 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.113719    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/51e093e4-5ec3-4086-985f-3022f731690b-lib-modules\") pod \"tuned-pcc9f\" (UID: \"51e093e4-5ec3-4086-985f-3022f731690b\") " pod="openshift-cluster-node-tuning-operator/tuned-pcc9f"
Sep 17 17:34:26.113907 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.113770    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/751c4ff3-f343-4280-b970-3583c2a4c28b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9tfrw\" (UID: \"751c4ff3-f343-4280-b970-3583c2a4c28b\") " pod="openshift-multus/multus-additional-cni-plugins-9tfrw"
Sep 17 17:34:26.113907 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.113797    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ef20182-de84-4cba-9eb4-1fcfdfbd1abe-etc-kubernetes\") pod \"multus-d972p\" (UID: \"9ef20182-de84-4cba-9eb4-1fcfdfbd1abe\") " pod="openshift-multus/multus-d972p"
Sep 17 17:34:26.113907 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.113824    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fced737e-fcb0-4559-b00b-e4813b035223-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cgp7g\" (UID: \"fced737e-fcb0-4559-b00b-e4813b035223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgp7g"
Sep 17 17:34:26.113907 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.113829    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9ef20182-de84-4cba-9eb4-1fcfdfbd1abe-multus-daemon-config\") pod \"multus-d972p\" (UID: \"9ef20182-de84-4cba-9eb4-1fcfdfbd1abe\") " pod="openshift-multus/multus-d972p"
Sep 17 17:34:26.113907 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.113849    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ef20182-de84-4cba-9eb4-1fcfdfbd1abe-host-var-lib-kubelet\") pod \"multus-d972p\" (UID: \"9ef20182-de84-4cba-9eb4-1fcfdfbd1abe\") " pod="openshift-multus/multus-d972p"
Sep 17 17:34:26.113907 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.113874    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7pnj\" (UniqueName: \"kubernetes.io/projected/9ef20182-de84-4cba-9eb4-1fcfdfbd1abe-kube-api-access-f7pnj\") pod \"multus-d972p\" (UID: \"9ef20182-de84-4cba-9eb4-1fcfdfbd1abe\") " pod="openshift-multus/multus-d972p"
Sep 17 17:34:26.114364 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.113920    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5-log-socket\") pod \"ovnkube-node-rrf79\" (UID: \"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.114364 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.113954    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5-etc-openvswitch\") pod \"ovnkube-node-rrf79\" (UID: \"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.114364 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.113987    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5-systemd-units\") pod \"ovnkube-node-rrf79\" (UID: \"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.114364 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.114017    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9ef20182-de84-4cba-9eb4-1fcfdfbd1abe-multus-conf-dir\") pod \"multus-d972p\" (UID: \"9ef20182-de84-4cba-9eb4-1fcfdfbd1abe\") " pod="openshift-multus/multus-d972p"
Sep 17 17:34:26.114364 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.114060    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/751c4ff3-f343-4280-b970-3583c2a4c28b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9tfrw\" (UID: \"751c4ff3-f343-4280-b970-3583c2a4c28b\") " pod="openshift-multus/multus-additional-cni-plugins-9tfrw"
Sep 17 17:34:26.114364 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.114081    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ef20182-de84-4cba-9eb4-1fcfdfbd1abe-host-var-lib-kubelet\") pod \"multus-d972p\" (UID: \"9ef20182-de84-4cba-9eb4-1fcfdfbd1abe\") " pod="openshift-multus/multus-d972p"
Sep 17 17:34:26.114364 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.114113    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ef20182-de84-4cba-9eb4-1fcfdfbd1abe-etc-kubernetes\") pod \"multus-d972p\" (UID: \"9ef20182-de84-4cba-9eb4-1fcfdfbd1abe\") " pod="openshift-multus/multus-d972p"
Sep 17 17:34:26.114364 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.114116    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fced737e-fcb0-4559-b00b-e4813b035223-socket-dir\") pod \"aws-ebs-csi-driver-node-cgp7g\" (UID: \"fced737e-fcb0-4559-b00b-e4813b035223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgp7g"
Sep 17 17:34:26.114364 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.114152    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/51e093e4-5ec3-4086-985f-3022f731690b-etc-sysctl-conf\") pod \"tuned-pcc9f\" (UID: \"51e093e4-5ec3-4086-985f-3022f731690b\") " pod="openshift-cluster-node-tuning-operator/tuned-pcc9f"
Sep 17 17:34:26.114364 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.114173    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/751c4ff3-f343-4280-b970-3583c2a4c28b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9tfrw\" (UID: \"751c4ff3-f343-4280-b970-3583c2a4c28b\") " pod="openshift-multus/multus-additional-cni-plugins-9tfrw"
Sep 17 17:34:26.114364 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.114185    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/51e093e4-5ec3-4086-985f-3022f731690b-etc-systemd\") pod \"tuned-pcc9f\" (UID: \"51e093e4-5ec3-4086-985f-3022f731690b\") " pod="openshift-cluster-node-tuning-operator/tuned-pcc9f"
Sep 17 17:34:26.114364 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.114229    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e8bda048-e7cf-4118-82c4-bd1d987e4566-serviceca\") pod \"node-ca-mglss\" (UID: \"e8bda048-e7cf-4118-82c4-bd1d987e4566\") " pod="openshift-image-registry/node-ca-mglss"
Sep 17 17:34:26.114364 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.114249    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a40b9f88-37de-405c-9702-3a73548b4f63-iptables-alerter-script\") pod \"iptables-alerter-lt9dx\" (UID: \"a40b9f88-37de-405c-9702-3a73548b4f63\") " pod="openshift-network-operator/iptables-alerter-lt9dx"
Sep 17 17:34:26.114364 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.114273    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rrf79\" (UID: \"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.114364 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.114279    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/51e093e4-5ec3-4086-985f-3022f731690b-etc-sysctl-conf\") pod \"tuned-pcc9f\" (UID: \"51e093e4-5ec3-4086-985f-3022f731690b\") " pod="openshift-cluster-node-tuning-operator/tuned-pcc9f"
Sep 17 17:34:26.114364 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.114291    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/51e093e4-5ec3-4086-985f-3022f731690b-etc-systemd\") pod \"tuned-pcc9f\" (UID: \"51e093e4-5ec3-4086-985f-3022f731690b\") " pod="openshift-cluster-node-tuning-operator/tuned-pcc9f"
Sep 17 17:34:26.114364 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.114310    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dbb6518b-5a9b-4a33-89a1-71c9e4db70a5-kubelet-config\") pod \"global-pull-secret-syncer-7spmx\" (UID: \"dbb6518b-5a9b-4a33-89a1-71c9e4db70a5\") " pod="kube-system/global-pull-secret-syncer-7spmx"
Sep 17 17:34:26.115044 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.114331    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fced737e-fcb0-4559-b00b-e4813b035223-sys-fs\") pod \"aws-ebs-csi-driver-node-cgp7g\" (UID: \"fced737e-fcb0-4559-b00b-e4813b035223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgp7g"
Sep 17 17:34:26.115044 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.114471    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51e093e4-5ec3-4086-985f-3022f731690b-etc-kubernetes\") pod \"tuned-pcc9f\" (UID: \"51e093e4-5ec3-4086-985f-3022f731690b\") " pod="openshift-cluster-node-tuning-operator/tuned-pcc9f"
Sep 17 17:34:26.115044 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.114510    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5-host-kubelet\") pod \"ovnkube-node-rrf79\" (UID: \"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.115044 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.114531    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ef20182-de84-4cba-9eb4-1fcfdfbd1abe-multus-cni-dir\") pod \"multus-d972p\" (UID: \"9ef20182-de84-4cba-9eb4-1fcfdfbd1abe\") " pod="openshift-multus/multus-d972p"
Sep 17 17:34:26.115044 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.114564    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9ef20182-de84-4cba-9eb4-1fcfdfbd1abe-host-var-lib-cni-bin\") pod \"multus-d972p\" (UID: \"9ef20182-de84-4cba-9eb4-1fcfdfbd1abe\") " pod="openshift-multus/multus-d972p"
Sep 17 17:34:26.115044 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.114583    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9ef20182-de84-4cba-9eb4-1fcfdfbd1abe-host-var-lib-cni-multus\") pod \"multus-d972p\" (UID: \"9ef20182-de84-4cba-9eb4-1fcfdfbd1abe\") " pod="openshift-multus/multus-d972p"
Sep 17 17:34:26.115044 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.114603    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fced737e-fcb0-4559-b00b-e4813b035223-etc-selinux\") pod \"aws-ebs-csi-driver-node-cgp7g\" (UID: \"fced737e-fcb0-4559-b00b-e4813b035223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgp7g"
Sep 17 17:34:26.115044 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.114623    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5-var-lib-openvswitch\") pod \"ovnkube-node-rrf79\" (UID: \"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.115044 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.114643    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9ef20182-de84-4cba-9eb4-1fcfdfbd1abe-cnibin\") pod \"multus-d972p\" (UID: \"9ef20182-de84-4cba-9eb4-1fcfdfbd1abe\") " pod="openshift-multus/multus-d972p"
Sep 17 17:34:26.115044 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.114664    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nndwc\" (UniqueName: \"kubernetes.io/projected/fced737e-fcb0-4559-b00b-e4813b035223-kube-api-access-nndwc\") pod \"aws-ebs-csi-driver-node-cgp7g\" (UID: \"fced737e-fcb0-4559-b00b-e4813b035223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgp7g"
Sep 17 17:34:26.115044 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.114683    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/51e093e4-5ec3-4086-985f-3022f731690b-run\") pod \"tuned-pcc9f\" (UID: \"51e093e4-5ec3-4086-985f-3022f731690b\") " pod="openshift-cluster-node-tuning-operator/tuned-pcc9f"
Sep 17 17:34:26.115044 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.114690    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e8bda048-e7cf-4118-82c4-bd1d987e4566-serviceca\") pod \"node-ca-mglss\" (UID: \"e8bda048-e7cf-4118-82c4-bd1d987e4566\") " pod="openshift-image-registry/node-ca-mglss"
Sep 17 17:34:26.115044 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.114717    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/751c4ff3-f343-4280-b970-3583c2a4c28b-cni-binary-copy\") pod \"multus-additional-cni-plugins-9tfrw\" (UID: \"751c4ff3-f343-4280-b970-3583c2a4c28b\") " pod="openshift-multus/multus-additional-cni-plugins-9tfrw"
Sep 17 17:34:26.115044 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.114727    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9ef20182-de84-4cba-9eb4-1fcfdfbd1abe-host-var-lib-cni-bin\") pod \"multus-d972p\" (UID: \"9ef20182-de84-4cba-9eb4-1fcfdfbd1abe\") " pod="openshift-multus/multus-d972p"
Sep 17 17:34:26.115044 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.114737    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5-ovnkube-config\") pod \"ovnkube-node-rrf79\" (UID: \"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.115044 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.114757    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9ef20182-de84-4cba-9eb4-1fcfdfbd1abe-multus-socket-dir-parent\") pod \"multus-d972p\" (UID: \"9ef20182-de84-4cba-9eb4-1fcfdfbd1abe\") " pod="openshift-multus/multus-d972p"
Sep 17 17:34:26.115044 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.114766    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5-var-lib-openvswitch\") pod \"ovnkube-node-rrf79\" (UID: \"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.115837 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.114779    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9ef20182-de84-4cba-9eb4-1fcfdfbd1abe-host-run-multus-certs\") pod \"multus-d972p\" (UID: \"9ef20182-de84-4cba-9eb4-1fcfdfbd1abe\") " pod="openshift-multus/multus-d972p"
Sep 17 17:34:26.115837 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.114792    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a40b9f88-37de-405c-9702-3a73548b4f63-iptables-alerter-script\") pod \"iptables-alerter-lt9dx\" (UID: \"a40b9f88-37de-405c-9702-3a73548b4f63\") " pod="openshift-network-operator/iptables-alerter-lt9dx"
Sep 17 17:34:26.115837 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.114808    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9ef20182-de84-4cba-9eb4-1fcfdfbd1abe-cnibin\") pod \"multus-d972p\" (UID: \"9ef20182-de84-4cba-9eb4-1fcfdfbd1abe\") " pod="openshift-multus/multus-d972p"
Sep 17 17:34:26.115837 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.114859    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/51e093e4-5ec3-4086-985f-3022f731690b-run\") pod \"tuned-pcc9f\" (UID: \"51e093e4-5ec3-4086-985f-3022f731690b\") " pod="openshift-cluster-node-tuning-operator/tuned-pcc9f"
Sep 17 17:34:26.115837 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.114861    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ef20182-de84-4cba-9eb4-1fcfdfbd1abe-multus-cni-dir\") pod \"multus-d972p\" (UID: \"9ef20182-de84-4cba-9eb4-1fcfdfbd1abe\") " pod="openshift-multus/multus-d972p"
Sep 17 17:34:26.115837 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.114799    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ef20182-de84-4cba-9eb4-1fcfdfbd1abe-system-cni-dir\") pod \"multus-d972p\" (UID: \"9ef20182-de84-4cba-9eb4-1fcfdfbd1abe\") " pod="openshift-multus/multus-d972p"
Sep 17 17:34:26.115837 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.114896    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/751c4ff3-f343-4280-b970-3583c2a4c28b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9tfrw\" (UID: \"751c4ff3-f343-4280-b970-3583c2a4c28b\") " pod="openshift-multus/multus-additional-cni-plugins-9tfrw"
Sep 17 17:34:26.115837 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.114919    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/51e093e4-5ec3-4086-985f-3022f731690b-etc-sysconfig\") pod \"tuned-pcc9f\" (UID: \"51e093e4-5ec3-4086-985f-3022f731690b\") " pod="openshift-cluster-node-tuning-operator/tuned-pcc9f"
Sep 17 17:34:26.115837 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.114932    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ef20182-de84-4cba-9eb4-1fcfdfbd1abe-system-cni-dir\") pod \"multus-d972p\" (UID: \"9ef20182-de84-4cba-9eb4-1fcfdfbd1abe\") " pod="openshift-multus/multus-d972p"
Sep 17 17:34:26.115837 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.114954    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/51e093e4-5ec3-4086-985f-3022f731690b-etc-sysctl-d\") pod \"tuned-pcc9f\" (UID: \"51e093e4-5ec3-4086-985f-3022f731690b\") " pod="openshift-cluster-node-tuning-operator/tuned-pcc9f"
Sep 17 17:34:26.115837 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.114990    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9ef20182-de84-4cba-9eb4-1fcfdfbd1abe-host-var-lib-cni-multus\") pod \"multus-d972p\" (UID: \"9ef20182-de84-4cba-9eb4-1fcfdfbd1abe\") " pod="openshift-multus/multus-d972p"
Sep 17 17:34:26.115837 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.115056    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/51e093e4-5ec3-4086-985f-3022f731690b-host\") pod \"tuned-pcc9f\" (UID: \"51e093e4-5ec3-4086-985f-3022f731690b\") " pod="openshift-cluster-node-tuning-operator/tuned-pcc9f"
Sep 17 17:34:26.115837 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.115082    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5-host-slash\") pod \"ovnkube-node-rrf79\" (UID: \"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.115837 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.115106    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5-env-overrides\") pod \"ovnkube-node-rrf79\" (UID: \"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.115837 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.115130    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5-ovn-node-metrics-cert\") pod \"ovnkube-node-rrf79\" (UID: \"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.115837 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.115154    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/51e093e4-5ec3-4086-985f-3022f731690b-etc-modprobe-d\") pod \"tuned-pcc9f\" (UID: \"51e093e4-5ec3-4086-985f-3022f731690b\") " pod="openshift-cluster-node-tuning-operator/tuned-pcc9f"
Sep 17 17:34:26.115837 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.115178    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51e093e4-5ec3-4086-985f-3022f731690b-var-lib-kubelet\") pod \"tuned-pcc9f\" (UID: \"51e093e4-5ec3-4086-985f-3022f731690b\") " pod="openshift-cluster-node-tuning-operator/tuned-pcc9f"
Sep 17 17:34:26.115837 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.115202    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6bq4\" (UniqueName: \"kubernetes.io/projected/51e093e4-5ec3-4086-985f-3022f731690b-kube-api-access-c6bq4\") pod \"tuned-pcc9f\" (UID: \"51e093e4-5ec3-4086-985f-3022f731690b\") " pod="openshift-cluster-node-tuning-operator/tuned-pcc9f"
Sep 17 17:34:26.116672 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.115224    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5-node-log\") pod \"ovnkube-node-rrf79\" (UID: \"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.116672 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.115247    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5-ovnkube-script-lib\") pod \"ovnkube-node-rrf79\" (UID: \"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.116672 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.115269    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9ef20182-de84-4cba-9eb4-1fcfdfbd1abe-cni-binary-copy\") pod \"multus-d972p\" (UID: \"9ef20182-de84-4cba-9eb4-1fcfdfbd1abe\") " pod="openshift-multus/multus-d972p"
Sep 17 17:34:26.116672 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.115291    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9ef20182-de84-4cba-9eb4-1fcfdfbd1abe-host-run-netns\") pod \"multus-d972p\" (UID: \"9ef20182-de84-4cba-9eb4-1fcfdfbd1abe\") " pod="openshift-multus/multus-d972p"
Sep 17 17:34:26.116672 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.115350    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5-ovnkube-config\") pod \"ovnkube-node-rrf79\" (UID: \"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.116672 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.115399    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/751c4ff3-f343-4280-b970-3583c2a4c28b-cni-binary-copy\") pod \"multus-additional-cni-plugins-9tfrw\" (UID: \"751c4ff3-f343-4280-b970-3583c2a4c28b\") " pod="openshift-multus/multus-additional-cni-plugins-9tfrw"
Sep 17 17:34:26.116672 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.115411    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5-node-log\") pod \"ovnkube-node-rrf79\" (UID: \"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.116672 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.115402    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9ef20182-de84-4cba-9eb4-1fcfdfbd1abe-hostroot\") pod \"multus-d972p\" (UID: \"9ef20182-de84-4cba-9eb4-1fcfdfbd1abe\") " pod="openshift-multus/multus-d972p"
Sep 17 17:34:26.116672 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.115432    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9ef20182-de84-4cba-9eb4-1fcfdfbd1abe-hostroot\") pod \"multus-d972p\" (UID: \"9ef20182-de84-4cba-9eb4-1fcfdfbd1abe\") " pod="openshift-multus/multus-d972p"
Sep 17 17:34:26.116672 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.115460    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9ef20182-de84-4cba-9eb4-1fcfdfbd1abe-multus-socket-dir-parent\") pod \"multus-d972p\" (UID: \"9ef20182-de84-4cba-9eb4-1fcfdfbd1abe\") " pod="openshift-multus/multus-d972p"
Sep 17 17:34:26.116672 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.115469    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/856535bd-6884-4e98-8089-852deae228aa-konnectivity-ca\") pod \"konnectivity-agent-jpg4z\" (UID: \"856535bd-6884-4e98-8089-852deae228aa\") " pod="kube-system/konnectivity-agent-jpg4z"
Sep 17 17:34:26.116672 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.115465    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9ef20182-de84-4cba-9eb4-1fcfdfbd1abe-host-run-multus-certs\") pod \"multus-d972p\" (UID: \"9ef20182-de84-4cba-9eb4-1fcfdfbd1abe\") " pod="openshift-multus/multus-d972p"
Sep 17 17:34:26.116672 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.115512    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9ef20182-de84-4cba-9eb4-1fcfdfbd1abe-host-run-netns\") pod \"multus-d972p\" (UID: \"9ef20182-de84-4cba-9eb4-1fcfdfbd1abe\") " pod="openshift-multus/multus-d972p"
Sep 17 17:34:26.116672 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.115522    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e8bda048-e7cf-4118-82c4-bd1d987e4566-host\") pod \"node-ca-mglss\" (UID: \"e8bda048-e7cf-4118-82c4-bd1d987e4566\") " pod="openshift-image-registry/node-ca-mglss"
Sep 17 17:34:26.116672 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.116074    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5-env-overrides\") pod \"ovnkube-node-rrf79\" (UID: \"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.116672 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.116081    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9ef20182-de84-4cba-9eb4-1fcfdfbd1abe-host-run-k8s-cni-cncf-io\") pod \"multus-d972p\" (UID: \"9ef20182-de84-4cba-9eb4-1fcfdfbd1abe\") " pod="openshift-multus/multus-d972p"
Sep 17 17:34:26.116672 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.116084    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rrf79\" (UID: \"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.116672 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.115582    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/51e093e4-5ec3-4086-985f-3022f731690b-etc-sysctl-d\") pod \"tuned-pcc9f\" (UID: \"51e093e4-5ec3-4086-985f-3022f731690b\") " pod="openshift-cluster-node-tuning-operator/tuned-pcc9f"
Sep 17 17:34:26.118320 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.115852    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/51e093e4-5ec3-4086-985f-3022f731690b-etc-sysconfig\") pod \"tuned-pcc9f\" (UID: \"51e093e4-5ec3-4086-985f-3022f731690b\") " pod="openshift-cluster-node-tuning-operator/tuned-pcc9f"
Sep 17 17:34:26.118320 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.116032    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/751c4ff3-f343-4280-b970-3583c2a4c28b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9tfrw\" (UID: \"751c4ff3-f343-4280-b970-3583c2a4c28b\") " pod="openshift-multus/multus-additional-cni-plugins-9tfrw"
Sep 17 17:34:26.118320 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.116052    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/856535bd-6884-4e98-8089-852deae228aa-konnectivity-ca\") pod \"konnectivity-agent-jpg4z\" (UID: \"856535bd-6884-4e98-8089-852deae228aa\") " pod="kube-system/konnectivity-agent-jpg4z"
Sep 17 17:34:26.118320 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.116111    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qpqn\" (UniqueName: \"kubernetes.io/projected/39af04b6-713b-406b-952d-7b2ef2b2b952-kube-api-access-7qpqn\") pod \"network-metrics-daemon-p7vh6\" (UID: \"39af04b6-713b-406b-952d-7b2ef2b2b952\") " pod="openshift-multus/network-metrics-daemon-p7vh6"
Sep 17 17:34:26.118320 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.116125    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5-ovnkube-script-lib\") pod \"ovnkube-node-rrf79\" (UID: \"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.118320 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.115650    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5-host-slash\") pod \"ovnkube-node-rrf79\" (UID: \"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.118320 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.115548    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e8bda048-e7cf-4118-82c4-bd1d987e4566-host\") pod \"node-ca-mglss\" (UID: \"e8bda048-e7cf-4118-82c4-bd1d987e4566\") " pod="openshift-image-registry/node-ca-mglss"
Sep 17 17:34:26.118320 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.116146    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51e093e4-5ec3-4086-985f-3022f731690b-etc-kubernetes\") pod \"tuned-pcc9f\" (UID: \"51e093e4-5ec3-4086-985f-3022f731690b\") " pod="openshift-cluster-node-tuning-operator/tuned-pcc9f"
Sep 17 17:34:26.118320 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.116178    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5-host-kubelet\") pod \"ovnkube-node-rrf79\" (UID: \"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.118320 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.116191    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/751c4ff3-f343-4280-b970-3583c2a4c28b-cnibin\") pod \"multus-additional-cni-plugins-9tfrw\" (UID: \"751c4ff3-f343-4280-b970-3583c2a4c28b\") " pod="openshift-multus/multus-additional-cni-plugins-9tfrw"
Sep 17 17:34:26.118320 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.116218    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/51e093e4-5ec3-4086-985f-3022f731690b-host\") pod \"tuned-pcc9f\" (UID: \"51e093e4-5ec3-4086-985f-3022f731690b\") " pod="openshift-cluster-node-tuning-operator/tuned-pcc9f"
Sep 17 17:34:26.118320 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.116225    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/51e093e4-5ec3-4086-985f-3022f731690b-etc-modprobe-d\") pod \"tuned-pcc9f\" (UID: \"51e093e4-5ec3-4086-985f-3022f731690b\") " pod="openshift-cluster-node-tuning-operator/tuned-pcc9f"
Sep 17 17:34:26.118320 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.116219    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a40b9f88-37de-405c-9702-3a73548b4f63-host-slash\") pod \"iptables-alerter-lt9dx\" (UID: \"a40b9f88-37de-405c-9702-3a73548b4f63\") " pod="openshift-network-operator/iptables-alerter-lt9dx"
Sep 17 17:34:26.118320 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.116259    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51e093e4-5ec3-4086-985f-3022f731690b-var-lib-kubelet\") pod \"tuned-pcc9f\" (UID: \"51e093e4-5ec3-4086-985f-3022f731690b\") " pod="openshift-cluster-node-tuning-operator/tuned-pcc9f"
Sep 17 17:34:26.118320 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.116266    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9ef20182-de84-4cba-9eb4-1fcfdfbd1abe-host-run-k8s-cni-cncf-io\") pod \"multus-d972p\" (UID: \"9ef20182-de84-4cba-9eb4-1fcfdfbd1abe\") " pod="openshift-multus/multus-d972p"
Sep 17 17:34:26.118320 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.116300    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a40b9f88-37de-405c-9702-3a73548b4f63-host-slash\") pod \"iptables-alerter-lt9dx\" (UID: \"a40b9f88-37de-405c-9702-3a73548b4f63\") " pod="openshift-network-operator/iptables-alerter-lt9dx"
Sep 17 17:34:26.118320 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.116306    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/751c4ff3-f343-4280-b970-3583c2a4c28b-cnibin\") pod \"multus-additional-cni-plugins-9tfrw\" (UID: \"751c4ff3-f343-4280-b970-3583c2a4c28b\") " pod="openshift-multus/multus-additional-cni-plugins-9tfrw"
Sep 17 17:34:26.118886 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.116330    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjcbl\" (UniqueName: \"kubernetes.io/projected/a40b9f88-37de-405c-9702-3a73548b4f63-kube-api-access-tjcbl\") pod \"iptables-alerter-lt9dx\" (UID: \"a40b9f88-37de-405c-9702-3a73548b4f63\") " pod="openshift-network-operator/iptables-alerter-lt9dx"
Sep 17 17:34:26.118886 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.116360    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5-host-run-netns\") pod \"ovnkube-node-rrf79\" (UID: \"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.118886 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.116483    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9ef20182-de84-4cba-9eb4-1fcfdfbd1abe-cni-binary-copy\") pod \"multus-d972p\" (UID: \"9ef20182-de84-4cba-9eb4-1fcfdfbd1abe\") " pod="openshift-multus/multus-d972p"
Sep 17 17:34:26.118886 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.116483    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5-host-run-netns\") pod \"ovnkube-node-rrf79\" (UID: \"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.118886 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.117827    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/51e093e4-5ec3-4086-985f-3022f731690b-tmp\") pod \"tuned-pcc9f\" (UID: \"51e093e4-5ec3-4086-985f-3022f731690b\") " pod="openshift-cluster-node-tuning-operator/tuned-pcc9f"
Sep 17 17:34:26.118886 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.118062    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/51e093e4-5ec3-4086-985f-3022f731690b-etc-tuned\") pod \"tuned-pcc9f\" (UID: \"51e093e4-5ec3-4086-985f-3022f731690b\") " pod="openshift-cluster-node-tuning-operator/tuned-pcc9f"
Sep 17 17:34:26.118886 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.118510    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5-ovn-node-metrics-cert\") pod \"ovnkube-node-rrf79\" (UID: \"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.118886 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.118586    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/856535bd-6884-4e98-8089-852deae228aa-agent-certs\") pod \"konnectivity-agent-jpg4z\" (UID: \"856535bd-6884-4e98-8089-852deae228aa\") " pod="kube-system/konnectivity-agent-jpg4z"
Sep 17 17:34:26.124789 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.124761    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hsct\" (UniqueName: \"kubernetes.io/projected/751c4ff3-f343-4280-b970-3583c2a4c28b-kube-api-access-8hsct\") pod \"multus-additional-cni-plugins-9tfrw\" (UID: \"751c4ff3-f343-4280-b970-3583c2a4c28b\") " pod="openshift-multus/multus-additional-cni-plugins-9tfrw"
Sep 17 17:34:26.125066 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:26.125039    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:34:26.125066 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:26.125056    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:34:26.125066 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:26.125067    2585 projected.go:196] Error preparing data for projected volume kube-api-access-4rlql for pod openshift-network-diagnostics/network-check-target-z8nx5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:26.125231 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:26.125133    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7bcabfd1-e72e-4866-9190-6bcce1f7ce64-kube-api-access-4rlql podName:7bcabfd1-e72e-4866-9190-6bcce1f7ce64 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:26.625106014 +0000 UTC m=+3.038896508 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4rlql" (UniqueName: "kubernetes.io/projected/7bcabfd1-e72e-4866-9190-6bcce1f7ce64-kube-api-access-4rlql") pod "network-check-target-z8nx5" (UID: "7bcabfd1-e72e-4866-9190-6bcce1f7ce64") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:26.125300 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.125283    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7pnj\" (UniqueName: \"kubernetes.io/projected/9ef20182-de84-4cba-9eb4-1fcfdfbd1abe-kube-api-access-f7pnj\") pod \"multus-d972p\" (UID: \"9ef20182-de84-4cba-9eb4-1fcfdfbd1abe\") " pod="openshift-multus/multus-d972p"
Sep 17 17:34:26.126838 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.126812    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pxj5\" (UniqueName: \"kubernetes.io/projected/e8bda048-e7cf-4118-82c4-bd1d987e4566-kube-api-access-5pxj5\") pod \"node-ca-mglss\" (UID: \"e8bda048-e7cf-4118-82c4-bd1d987e4566\") " pod="openshift-image-registry/node-ca-mglss"
Sep 17 17:34:26.127142 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.127121    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjcbl\" (UniqueName: \"kubernetes.io/projected/a40b9f88-37de-405c-9702-3a73548b4f63-kube-api-access-tjcbl\") pod \"iptables-alerter-lt9dx\" (UID: \"a40b9f88-37de-405c-9702-3a73548b4f63\") " pod="openshift-network-operator/iptables-alerter-lt9dx"
Sep 17 17:34:26.127753 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.127733    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6bq4\" (UniqueName: \"kubernetes.io/projected/51e093e4-5ec3-4086-985f-3022f731690b-kube-api-access-c6bq4\") pod \"tuned-pcc9f\" (UID: \"51e093e4-5ec3-4086-985f-3022f731690b\") " pod="openshift-cluster-node-tuning-operator/tuned-pcc9f"
Sep 17 17:34:26.128036 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.128014    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pr56\" (UniqueName: \"kubernetes.io/projected/beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5-kube-api-access-5pr56\") pod \"ovnkube-node-rrf79\" (UID: \"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5\") " pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.129029 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.129008    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qpqn\" (UniqueName: \"kubernetes.io/projected/39af04b6-713b-406b-952d-7b2ef2b2b952-kube-api-access-7qpqn\") pod \"network-metrics-daemon-p7vh6\" (UID: \"39af04b6-713b-406b-952d-7b2ef2b2b952\") " pod="openshift-multus/network-metrics-daemon-p7vh6"
Sep 17 17:34:26.217330 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.217294    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fced737e-fcb0-4559-b00b-e4813b035223-socket-dir\") pod \"aws-ebs-csi-driver-node-cgp7g\" (UID: \"fced737e-fcb0-4559-b00b-e4813b035223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgp7g"
Sep 17 17:34:26.217330 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.217341    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dbb6518b-5a9b-4a33-89a1-71c9e4db70a5-kubelet-config\") pod \"global-pull-secret-syncer-7spmx\" (UID: \"dbb6518b-5a9b-4a33-89a1-71c9e4db70a5\") " pod="kube-system/global-pull-secret-syncer-7spmx"
Sep 17 17:34:26.217579 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.217366    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fced737e-fcb0-4559-b00b-e4813b035223-sys-fs\") pod \"aws-ebs-csi-driver-node-cgp7g\" (UID: \"fced737e-fcb0-4559-b00b-e4813b035223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgp7g"
Sep 17 17:34:26.217579 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.217429    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fced737e-fcb0-4559-b00b-e4813b035223-sys-fs\") pod \"aws-ebs-csi-driver-node-cgp7g\" (UID: \"fced737e-fcb0-4559-b00b-e4813b035223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgp7g"
Sep 17 17:34:26.217579 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.217444    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dbb6518b-5a9b-4a33-89a1-71c9e4db70a5-kubelet-config\") pod \"global-pull-secret-syncer-7spmx\" (UID: \"dbb6518b-5a9b-4a33-89a1-71c9e4db70a5\") " pod="kube-system/global-pull-secret-syncer-7spmx"
Sep 17 17:34:26.217579 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.217484    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fced737e-fcb0-4559-b00b-e4813b035223-socket-dir\") pod \"aws-ebs-csi-driver-node-cgp7g\" (UID: \"fced737e-fcb0-4559-b00b-e4813b035223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgp7g"
Sep 17 17:34:26.217579 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.217487    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fced737e-fcb0-4559-b00b-e4813b035223-etc-selinux\") pod \"aws-ebs-csi-driver-node-cgp7g\" (UID: \"fced737e-fcb0-4559-b00b-e4813b035223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgp7g"
Sep 17 17:34:26.217579 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.217538    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nndwc\" (UniqueName: \"kubernetes.io/projected/fced737e-fcb0-4559-b00b-e4813b035223-kube-api-access-nndwc\") pod \"aws-ebs-csi-driver-node-cgp7g\" (UID: \"fced737e-fcb0-4559-b00b-e4813b035223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgp7g"
Sep 17 17:34:26.217579 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.217577    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dbb6518b-5a9b-4a33-89a1-71c9e4db70a5-dbus\") pod \"global-pull-secret-syncer-7spmx\" (UID: \"dbb6518b-5a9b-4a33-89a1-71c9e4db70a5\") " pod="kube-system/global-pull-secret-syncer-7spmx"
Sep 17 17:34:26.217894 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.217600    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fced737e-fcb0-4559-b00b-e4813b035223-etc-selinux\") pod \"aws-ebs-csi-driver-node-cgp7g\" (UID: \"fced737e-fcb0-4559-b00b-e4813b035223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgp7g"
Sep 17 17:34:26.217894 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.217649    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fced737e-fcb0-4559-b00b-e4813b035223-device-dir\") pod \"aws-ebs-csi-driver-node-cgp7g\" (UID: \"fced737e-fcb0-4559-b00b-e4813b035223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgp7g"
Sep 17 17:34:26.217894 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.217688    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fced737e-fcb0-4559-b00b-e4813b035223-registration-dir\") pod \"aws-ebs-csi-driver-node-cgp7g\" (UID: \"fced737e-fcb0-4559-b00b-e4813b035223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgp7g"
Sep 17 17:34:26.217894 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.217716    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dbb6518b-5a9b-4a33-89a1-71c9e4db70a5-original-pull-secret\") pod \"global-pull-secret-syncer-7spmx\" (UID: \"dbb6518b-5a9b-4a33-89a1-71c9e4db70a5\") " pod="kube-system/global-pull-secret-syncer-7spmx"
Sep 17 17:34:26.217894 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.217741    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fced737e-fcb0-4559-b00b-e4813b035223-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cgp7g\" (UID: \"fced737e-fcb0-4559-b00b-e4813b035223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgp7g"
Sep 17 17:34:26.217894 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.217775    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dbb6518b-5a9b-4a33-89a1-71c9e4db70a5-dbus\") pod \"global-pull-secret-syncer-7spmx\" (UID: \"dbb6518b-5a9b-4a33-89a1-71c9e4db70a5\") " pod="kube-system/global-pull-secret-syncer-7spmx"
Sep 17 17:34:26.217894 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.217797    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fced737e-fcb0-4559-b00b-e4813b035223-device-dir\") pod \"aws-ebs-csi-driver-node-cgp7g\" (UID: \"fced737e-fcb0-4559-b00b-e4813b035223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgp7g"
Sep 17 17:34:26.217894 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.217807    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fced737e-fcb0-4559-b00b-e4813b035223-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cgp7g\" (UID: \"fced737e-fcb0-4559-b00b-e4813b035223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgp7g"
Sep 17 17:34:26.217894 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.217838    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fced737e-fcb0-4559-b00b-e4813b035223-registration-dir\") pod \"aws-ebs-csi-driver-node-cgp7g\" (UID: \"fced737e-fcb0-4559-b00b-e4813b035223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgp7g"
Sep 17 17:34:26.217894 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:26.217878    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:26.218236 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:26.217922    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbb6518b-5a9b-4a33-89a1-71c9e4db70a5-original-pull-secret podName:dbb6518b-5a9b-4a33-89a1-71c9e4db70a5 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:26.717908867 +0000 UTC m=+3.131699287 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dbb6518b-5a9b-4a33-89a1-71c9e4db70a5-original-pull-secret") pod "global-pull-secret-syncer-7spmx" (UID: "dbb6518b-5a9b-4a33-89a1-71c9e4db70a5") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:26.229672 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.229646    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nndwc\" (UniqueName: \"kubernetes.io/projected/fced737e-fcb0-4559-b00b-e4813b035223-kube-api-access-nndwc\") pod \"aws-ebs-csi-driver-node-cgp7g\" (UID: \"fced737e-fcb0-4559-b00b-e4813b035223\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgp7g"
Sep 17 17:34:26.291638 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.291556    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pcc9f"
Sep 17 17:34:26.297789 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.297769    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9tfrw"
Sep 17 17:34:26.306594 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.306576    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-d972p"
Sep 17 17:34:26.312456 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.312440    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jpg4z"
Sep 17 17:34:26.318154 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.318132    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mglss"
Sep 17 17:34:26.325789 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.325771    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lt9dx"
Sep 17 17:34:26.331704 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.331685    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:26.336502 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.336465    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgp7g"
Sep 17 17:34:26.620709 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.620634    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39af04b6-713b-406b-952d-7b2ef2b2b952-metrics-certs\") pod \"network-metrics-daemon-p7vh6\" (UID: \"39af04b6-713b-406b-952d-7b2ef2b2b952\") " pod="openshift-multus/network-metrics-daemon-p7vh6"
Sep 17 17:34:26.620853 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:26.620781    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:26.620853 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:26.620844    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39af04b6-713b-406b-952d-7b2ef2b2b952-metrics-certs podName:39af04b6-713b-406b-952d-7b2ef2b2b952 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:27.620828357 +0000 UTC m=+4.034618774 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/39af04b6-713b-406b-952d-7b2ef2b2b952-metrics-certs") pod "network-metrics-daemon-p7vh6" (UID: "39af04b6-713b-406b-952d-7b2ef2b2b952") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:26.721181 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.721153    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rlql\" (UniqueName: \"kubernetes.io/projected/7bcabfd1-e72e-4866-9190-6bcce1f7ce64-kube-api-access-4rlql\") pod \"network-check-target-z8nx5\" (UID: \"7bcabfd1-e72e-4866-9190-6bcce1f7ce64\") " pod="openshift-network-diagnostics/network-check-target-z8nx5"
Sep 17 17:34:26.721350 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:26.721187    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dbb6518b-5a9b-4a33-89a1-71c9e4db70a5-original-pull-secret\") pod \"global-pull-secret-syncer-7spmx\" (UID: \"dbb6518b-5a9b-4a33-89a1-71c9e4db70a5\") " pod="kube-system/global-pull-secret-syncer-7spmx"
Sep 17 17:34:26.721350 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:26.721329    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:34:26.721350 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:26.721348    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:34:26.721479 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:26.721354    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:26.721479 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:26.721411    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbb6518b-5a9b-4a33-89a1-71c9e4db70a5-original-pull-secret podName:dbb6518b-5a9b-4a33-89a1-71c9e4db70a5 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:27.721394053 +0000 UTC m=+4.135184473 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dbb6518b-5a9b-4a33-89a1-71c9e4db70a5-original-pull-secret") pod "global-pull-secret-syncer-7spmx" (UID: "dbb6518b-5a9b-4a33-89a1-71c9e4db70a5") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:26.721479 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:26.721357    2585 projected.go:196] Error preparing data for projected volume kube-api-access-4rlql for pod openshift-network-diagnostics/network-check-target-z8nx5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:26.721628 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:26.721536    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7bcabfd1-e72e-4866-9190-6bcce1f7ce64-kube-api-access-4rlql podName:7bcabfd1-e72e-4866-9190-6bcce1f7ce64 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:27.721516955 +0000 UTC m=+4.135307376 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4rlql" (UniqueName: "kubernetes.io/projected/7bcabfd1-e72e-4866-9190-6bcce1f7ce64-kube-api-access-4rlql") pod "network-check-target-z8nx5" (UID: "7bcabfd1-e72e-4866-9190-6bcce1f7ce64") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:26.936367 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:26.936344    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfced737e_fcb0_4559_b00b_e4813b035223.slice/crio-6b4f6fca299c05c3c9a5e1181eb20404fdaac909496119c6c0ec9e622ba97443 WatchSource:0}: Error finding container 6b4f6fca299c05c3c9a5e1181eb20404fdaac909496119c6c0ec9e622ba97443: Status 404 returned error can't find the container with id 6b4f6fca299c05c3c9a5e1181eb20404fdaac909496119c6c0ec9e622ba97443
Sep 17 17:34:26.936985 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:26.936961    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51e093e4_5ec3_4086_985f_3022f731690b.slice/crio-0289b0698bb6598713663356f558324faae29330ca413b658c3f13bae7ba9968 WatchSource:0}: Error finding container 0289b0698bb6598713663356f558324faae29330ca413b658c3f13bae7ba9968: Status 404 returned error can't find the container with id 0289b0698bb6598713663356f558324faae29330ca413b658c3f13bae7ba9968
Sep 17 17:34:26.938549 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:26.938365    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod751c4ff3_f343_4280_b970_3583c2a4c28b.slice/crio-018b3beb320a70e1dd6d6242423ff702b7aba85979804f08a2f25d6b66e50c9b WatchSource:0}: Error finding container 018b3beb320a70e1dd6d6242423ff702b7aba85979804f08a2f25d6b66e50c9b: Status 404 returned error can't find the container with id 018b3beb320a70e1dd6d6242423ff702b7aba85979804f08a2f25d6b66e50c9b
Sep 17 17:34:26.941135 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:26.941110    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ef20182_de84_4cba_9eb4_1fcfdfbd1abe.slice/crio-b8824bf795a5e9ab92693bf4fdb58054099a7060b440c0766a2fdf8d00a25a88 WatchSource:0}: Error finding container b8824bf795a5e9ab92693bf4fdb58054099a7060b440c0766a2fdf8d00a25a88: Status 404 returned error can't find the container with id b8824bf795a5e9ab92693bf4fdb58054099a7060b440c0766a2fdf8d00a25a88
Sep 17 17:34:26.942806 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:26.942685    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod856535bd_6884_4e98_8089_852deae228aa.slice/crio-34b29b6108232b0a6515dc742df8f6a8cae1ec446a4940bf997c3b8272bb6c31 WatchSource:0}: Error finding container 34b29b6108232b0a6515dc742df8f6a8cae1ec446a4940bf997c3b8272bb6c31: Status 404 returned error can't find the container with id 34b29b6108232b0a6515dc742df8f6a8cae1ec446a4940bf997c3b8272bb6c31
Sep 17 17:34:26.944334 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:26.944215    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8bda048_e7cf_4118_82c4_bd1d987e4566.slice/crio-113cba662440f1864245a47d36e13d1625f62cb9d6052e46adb0975a8aa075ae WatchSource:0}: Error finding container 113cba662440f1864245a47d36e13d1625f62cb9d6052e46adb0975a8aa075ae: Status 404 returned error can't find the container with id 113cba662440f1864245a47d36e13d1625f62cb9d6052e46adb0975a8aa075ae
Sep 17 17:34:26.945546 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:26.945216    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeff8ae2_1a86_49b3_a9bc_d9d2e1b99bf5.slice/crio-61323a7d66f579f8c4f4841a369e6d8cb01a4b8b414b1091d6fe7a1bfb2c8b9c WatchSource:0}: Error finding container 61323a7d66f579f8c4f4841a369e6d8cb01a4b8b414b1091d6fe7a1bfb2c8b9c: Status 404 returned error can't find the container with id 61323a7d66f579f8c4f4841a369e6d8cb01a4b8b414b1091d6fe7a1bfb2c8b9c
Sep 17 17:34:27.061192 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:27.061146    2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:29:26 +0000 UTC" deadline="2028-02-19 06:37:42.35694989 +0000 UTC"
Sep 17 17:34:27.061659 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:27.061199    2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12469h3m15.295765015s"
Sep 17 17:34:27.108538 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:27.108507    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-139.ec2.internal" event={"ID":"aba6d40ed12a7528810e76c18b9ddea9","Type":"ContainerStarted","Data":"42c2cc0b73dce1eae09d770b481d0d528f2f8dcf9332b0ce26a3ef374e9ad067"}
Sep 17 17:34:27.109662 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:27.109630    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lt9dx" event={"ID":"a40b9f88-37de-405c-9702-3a73548b4f63","Type":"ContainerStarted","Data":"f4fd4c4ad3e6f7786c13ead568b399bbb425655d89c58007085e0ee11e2318b5"}
Sep 17 17:34:27.110856 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:27.110832    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rrf79" event={"ID":"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5","Type":"ContainerStarted","Data":"61323a7d66f579f8c4f4841a369e6d8cb01a4b8b414b1091d6fe7a1bfb2c8b9c"}
Sep 17 17:34:27.112387 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:27.112367    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-d972p" event={"ID":"9ef20182-de84-4cba-9eb4-1fcfdfbd1abe","Type":"ContainerStarted","Data":"b8824bf795a5e9ab92693bf4fdb58054099a7060b440c0766a2fdf8d00a25a88"}
Sep 17 17:34:27.113472 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:27.113449    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9tfrw" event={"ID":"751c4ff3-f343-4280-b970-3583c2a4c28b","Type":"ContainerStarted","Data":"018b3beb320a70e1dd6d6242423ff702b7aba85979804f08a2f25d6b66e50c9b"}
Sep 17 17:34:27.114595 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:27.114563    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgp7g" event={"ID":"fced737e-fcb0-4559-b00b-e4813b035223","Type":"ContainerStarted","Data":"6b4f6fca299c05c3c9a5e1181eb20404fdaac909496119c6c0ec9e622ba97443"}
Sep 17 17:34:27.115756 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:27.115733    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mglss" event={"ID":"e8bda048-e7cf-4118-82c4-bd1d987e4566","Type":"ContainerStarted","Data":"113cba662440f1864245a47d36e13d1625f62cb9d6052e46adb0975a8aa075ae"}
Sep 17 17:34:27.116773 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:27.116752    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jpg4z" event={"ID":"856535bd-6884-4e98-8089-852deae228aa","Type":"ContainerStarted","Data":"34b29b6108232b0a6515dc742df8f6a8cae1ec446a4940bf997c3b8272bb6c31"}
Sep 17 17:34:27.118024 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:27.118004    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pcc9f" event={"ID":"51e093e4-5ec3-4086-985f-3022f731690b","Type":"ContainerStarted","Data":"0289b0698bb6598713663356f558324faae29330ca413b658c3f13bae7ba9968"}
Sep 17 17:34:27.125858 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:27.125803    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-139.ec2.internal" podStartSLOduration=2.125790865 podStartE2EDuration="2.125790865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:27.125283888 +0000 UTC m=+3.539074326" watchObservedRunningTime="2026-09-17 17:34:27.125790865 +0000 UTC m=+3.539581302"
Sep 17 17:34:27.627671 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:27.627052    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39af04b6-713b-406b-952d-7b2ef2b2b952-metrics-certs\") pod \"network-metrics-daemon-p7vh6\" (UID: \"39af04b6-713b-406b-952d-7b2ef2b2b952\") " pod="openshift-multus/network-metrics-daemon-p7vh6"
Sep 17 17:34:27.627671 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:27.627251    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:27.627671 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:27.627322    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39af04b6-713b-406b-952d-7b2ef2b2b952-metrics-certs podName:39af04b6-713b-406b-952d-7b2ef2b2b952 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:29.627294685 +0000 UTC m=+6.041085116 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/39af04b6-713b-406b-952d-7b2ef2b2b952-metrics-certs") pod "network-metrics-daemon-p7vh6" (UID: "39af04b6-713b-406b-952d-7b2ef2b2b952") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:27.728043 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:27.728009    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rlql\" (UniqueName: \"kubernetes.io/projected/7bcabfd1-e72e-4866-9190-6bcce1f7ce64-kube-api-access-4rlql\") pod \"network-check-target-z8nx5\" (UID: \"7bcabfd1-e72e-4866-9190-6bcce1f7ce64\") " pod="openshift-network-diagnostics/network-check-target-z8nx5"
Sep 17 17:34:27.728195 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:27.728060    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dbb6518b-5a9b-4a33-89a1-71c9e4db70a5-original-pull-secret\") pod \"global-pull-secret-syncer-7spmx\" (UID: \"dbb6518b-5a9b-4a33-89a1-71c9e4db70a5\") " pod="kube-system/global-pull-secret-syncer-7spmx"
Sep 17 17:34:27.728265 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:27.728223    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:27.728319 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:27.728298    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbb6518b-5a9b-4a33-89a1-71c9e4db70a5-original-pull-secret podName:dbb6518b-5a9b-4a33-89a1-71c9e4db70a5 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:29.72828149 +0000 UTC m=+6.142071907 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dbb6518b-5a9b-4a33-89a1-71c9e4db70a5-original-pull-secret") pod "global-pull-secret-syncer-7spmx" (UID: "dbb6518b-5a9b-4a33-89a1-71c9e4db70a5") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:27.728728 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:27.728708    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:34:27.728827 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:27.728734    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:34:27.728827 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:27.728746    2585 projected.go:196] Error preparing data for projected volume kube-api-access-4rlql for pod openshift-network-diagnostics/network-check-target-z8nx5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:27.728827 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:27.728790    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7bcabfd1-e72e-4866-9190-6bcce1f7ce64-kube-api-access-4rlql podName:7bcabfd1-e72e-4866-9190-6bcce1f7ce64 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:29.728773364 +0000 UTC m=+6.142563793 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4rlql" (UniqueName: "kubernetes.io/projected/7bcabfd1-e72e-4866-9190-6bcce1f7ce64-kube-api-access-4rlql") pod "network-check-target-z8nx5" (UID: "7bcabfd1-e72e-4866-9190-6bcce1f7ce64") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:28.061947 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:28.061906    2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:29:26 +0000 UTC" deadline="2028-05-17 10:23:57.107365593 +0000 UTC"
Sep 17 17:34:28.061947 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:28.061945    2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14584h49m29.045425321s"
Sep 17 17:34:28.102504 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:28.101638    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z8nx5"
Sep 17 17:34:28.102504 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:28.101753    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z8nx5" podUID="7bcabfd1-e72e-4866-9190-6bcce1f7ce64"
Sep 17 17:34:28.102504 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:28.101833    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p7vh6"
Sep 17 17:34:28.102504 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:28.101911    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p7vh6" podUID="39af04b6-713b-406b-952d-7b2ef2b2b952"
Sep 17 17:34:28.102504 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:28.101977    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7spmx"
Sep 17 17:34:28.102504 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:28.102048    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7spmx" podUID="dbb6518b-5a9b-4a33-89a1-71c9e4db70a5"
Sep 17 17:34:28.125142 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:28.125112    2585 generic.go:356] "Generic (PLEG): container finished" podID="359ad2cc4231b18d20008ab45c907b6d" containerID="8cab2bcbfb0f71d9c4ba7fe249e4b84ee0f3b7a6b12a2cf011d17c7dc25393e8" exitCode=0
Sep 17 17:34:28.125271 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:28.125204    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-139.ec2.internal" event={"ID":"359ad2cc4231b18d20008ab45c907b6d","Type":"ContainerDied","Data":"8cab2bcbfb0f71d9c4ba7fe249e4b84ee0f3b7a6b12a2cf011d17c7dc25393e8"}
Sep 17 17:34:29.140770 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:29.140738    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-139.ec2.internal" event={"ID":"359ad2cc4231b18d20008ab45c907b6d","Type":"ContainerStarted","Data":"b05dd5b37b57d42c9184334126a1c98bdd3065195dd780db2003369c3ade0e8b"}
Sep 17 17:34:29.164651 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:29.164603    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-139.ec2.internal" podStartSLOduration=4.164583412 podStartE2EDuration="4.164583412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:29.164561777 +0000 UTC m=+5.578352216" watchObservedRunningTime="2026-09-17 17:34:29.164583412 +0000 UTC m=+5.578373876"
Sep 17 17:34:29.643046 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:29.643012    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39af04b6-713b-406b-952d-7b2ef2b2b952-metrics-certs\") pod \"network-metrics-daemon-p7vh6\" (UID: \"39af04b6-713b-406b-952d-7b2ef2b2b952\") " pod="openshift-multus/network-metrics-daemon-p7vh6"
Sep 17 17:34:29.643239 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:29.643221    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:29.643308 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:29.643285    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39af04b6-713b-406b-952d-7b2ef2b2b952-metrics-certs podName:39af04b6-713b-406b-952d-7b2ef2b2b952 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:33.643268269 +0000 UTC m=+10.057058690 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/39af04b6-713b-406b-952d-7b2ef2b2b952-metrics-certs") pod "network-metrics-daemon-p7vh6" (UID: "39af04b6-713b-406b-952d-7b2ef2b2b952") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:29.743823 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:29.743783    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rlql\" (UniqueName: \"kubernetes.io/projected/7bcabfd1-e72e-4866-9190-6bcce1f7ce64-kube-api-access-4rlql\") pod \"network-check-target-z8nx5\" (UID: \"7bcabfd1-e72e-4866-9190-6bcce1f7ce64\") " pod="openshift-network-diagnostics/network-check-target-z8nx5"
Sep 17 17:34:29.743952 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:29.743860    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dbb6518b-5a9b-4a33-89a1-71c9e4db70a5-original-pull-secret\") pod \"global-pull-secret-syncer-7spmx\" (UID: \"dbb6518b-5a9b-4a33-89a1-71c9e4db70a5\") " pod="kube-system/global-pull-secret-syncer-7spmx"
Sep 17 17:34:29.744033 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:29.744017    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:29.744120 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:29.744089    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbb6518b-5a9b-4a33-89a1-71c9e4db70a5-original-pull-secret podName:dbb6518b-5a9b-4a33-89a1-71c9e4db70a5 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:33.74407058 +0000 UTC m=+10.157860998 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dbb6518b-5a9b-4a33-89a1-71c9e4db70a5-original-pull-secret") pod "global-pull-secret-syncer-7spmx" (UID: "dbb6518b-5a9b-4a33-89a1-71c9e4db70a5") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:29.744188 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:29.744142    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:34:29.744188 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:29.744163    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:34:29.744188 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:29.744172    2585 projected.go:196] Error preparing data for projected volume kube-api-access-4rlql for pod openshift-network-diagnostics/network-check-target-z8nx5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:29.744321 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:29.744217    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7bcabfd1-e72e-4866-9190-6bcce1f7ce64-kube-api-access-4rlql podName:7bcabfd1-e72e-4866-9190-6bcce1f7ce64 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:33.744202302 +0000 UTC m=+10.157992718 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4rlql" (UniqueName: "kubernetes.io/projected/7bcabfd1-e72e-4866-9190-6bcce1f7ce64-kube-api-access-4rlql") pod "network-check-target-z8nx5" (UID: "7bcabfd1-e72e-4866-9190-6bcce1f7ce64") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:30.100783 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:30.100751    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z8nx5"
Sep 17 17:34:30.100959 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:30.100750    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p7vh6"
Sep 17 17:34:30.100959 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:30.100879    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z8nx5" podUID="7bcabfd1-e72e-4866-9190-6bcce1f7ce64"
Sep 17 17:34:30.100959 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:30.100903    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7spmx"
Sep 17 17:34:30.101117 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:30.101016    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p7vh6" podUID="39af04b6-713b-406b-952d-7b2ef2b2b952"
Sep 17 17:34:30.101117 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:30.101100    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7spmx" podUID="dbb6518b-5a9b-4a33-89a1-71c9e4db70a5"
Sep 17 17:34:32.100573 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:32.100183    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p7vh6"
Sep 17 17:34:32.100573 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:32.100284    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p7vh6" podUID="39af04b6-713b-406b-952d-7b2ef2b2b952"
Sep 17 17:34:32.100573 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:32.100570    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7spmx"
Sep 17 17:34:32.101112 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:32.100686    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7spmx" podUID="dbb6518b-5a9b-4a33-89a1-71c9e4db70a5"
Sep 17 17:34:32.101112 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:32.100740    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z8nx5"
Sep 17 17:34:32.101112 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:32.100820    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z8nx5" podUID="7bcabfd1-e72e-4866-9190-6bcce1f7ce64"
Sep 17 17:34:33.678774 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:33.678525    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39af04b6-713b-406b-952d-7b2ef2b2b952-metrics-certs\") pod \"network-metrics-daemon-p7vh6\" (UID: \"39af04b6-713b-406b-952d-7b2ef2b2b952\") " pod="openshift-multus/network-metrics-daemon-p7vh6"
Sep 17 17:34:33.679215 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:33.678658    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:33.679215 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:33.678897    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39af04b6-713b-406b-952d-7b2ef2b2b952-metrics-certs podName:39af04b6-713b-406b-952d-7b2ef2b2b952 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:41.67887428 +0000 UTC m=+18.092664697 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/39af04b6-713b-406b-952d-7b2ef2b2b952-metrics-certs") pod "network-metrics-daemon-p7vh6" (UID: "39af04b6-713b-406b-952d-7b2ef2b2b952") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:33.779913 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:33.779874    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dbb6518b-5a9b-4a33-89a1-71c9e4db70a5-original-pull-secret\") pod \"global-pull-secret-syncer-7spmx\" (UID: \"dbb6518b-5a9b-4a33-89a1-71c9e4db70a5\") " pod="kube-system/global-pull-secret-syncer-7spmx"
Sep 17 17:34:33.780092 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:33.779976    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rlql\" (UniqueName: \"kubernetes.io/projected/7bcabfd1-e72e-4866-9190-6bcce1f7ce64-kube-api-access-4rlql\") pod \"network-check-target-z8nx5\" (UID: \"7bcabfd1-e72e-4866-9190-6bcce1f7ce64\") " pod="openshift-network-diagnostics/network-check-target-z8nx5"
Sep 17 17:34:33.780154 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:33.780117    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:34:33.780154 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:33.780133    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:34:33.780154 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:33.780142    2585 projected.go:196] Error preparing data for projected volume kube-api-access-4rlql for pod openshift-network-diagnostics/network-check-target-z8nx5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:33.780294 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:33.780191    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7bcabfd1-e72e-4866-9190-6bcce1f7ce64-kube-api-access-4rlql podName:7bcabfd1-e72e-4866-9190-6bcce1f7ce64 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:41.780173287 +0000 UTC m=+18.193963714 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4rlql" (UniqueName: "kubernetes.io/projected/7bcabfd1-e72e-4866-9190-6bcce1f7ce64-kube-api-access-4rlql") pod "network-check-target-z8nx5" (UID: "7bcabfd1-e72e-4866-9190-6bcce1f7ce64") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:33.780591 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:33.780573    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:33.780670 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:33.780618    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbb6518b-5a9b-4a33-89a1-71c9e4db70a5-original-pull-secret podName:dbb6518b-5a9b-4a33-89a1-71c9e4db70a5 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:41.780608135 +0000 UTC m=+18.194398556 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dbb6518b-5a9b-4a33-89a1-71c9e4db70a5-original-pull-secret") pod "global-pull-secret-syncer-7spmx" (UID: "dbb6518b-5a9b-4a33-89a1-71c9e4db70a5") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:34.100519 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:34.100027    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z8nx5"
Sep 17 17:34:34.100519 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:34.100138    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z8nx5" podUID="7bcabfd1-e72e-4866-9190-6bcce1f7ce64"
Sep 17 17:34:34.100519 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:34.100216    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p7vh6"
Sep 17 17:34:34.100519 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:34.100331    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p7vh6" podUID="39af04b6-713b-406b-952d-7b2ef2b2b952"
Sep 17 17:34:34.100519 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:34.100377    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7spmx"
Sep 17 17:34:34.100519 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:34.100442    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7spmx" podUID="dbb6518b-5a9b-4a33-89a1-71c9e4db70a5"
Sep 17 17:34:36.099754 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:36.099720    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p7vh6"
Sep 17 17:34:36.100131 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:36.099721    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7spmx"
Sep 17 17:34:36.100131 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:36.099860    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p7vh6" podUID="39af04b6-713b-406b-952d-7b2ef2b2b952"
Sep 17 17:34:36.100131 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:36.099721    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z8nx5"
Sep 17 17:34:36.100131 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:36.099934    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7spmx" podUID="dbb6518b-5a9b-4a33-89a1-71c9e4db70a5"
Sep 17 17:34:36.100131 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:36.099998    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z8nx5" podUID="7bcabfd1-e72e-4866-9190-6bcce1f7ce64"
Sep 17 17:34:38.099915 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:38.099874    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p7vh6"
Sep 17 17:34:38.099915 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:38.099905    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7spmx"
Sep 17 17:34:38.100368 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:38.099874    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z8nx5"
Sep 17 17:34:38.100368 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:38.100011    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p7vh6" podUID="39af04b6-713b-406b-952d-7b2ef2b2b952"
Sep 17 17:34:38.100368 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:38.100092    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z8nx5" podUID="7bcabfd1-e72e-4866-9190-6bcce1f7ce64"
Sep 17 17:34:38.100368 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:38.100169    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7spmx" podUID="dbb6518b-5a9b-4a33-89a1-71c9e4db70a5"
Sep 17 17:34:40.099788 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:40.099759    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z8nx5"
Sep 17 17:34:40.100161 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:40.099764    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p7vh6"
Sep 17 17:34:40.100161 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:40.099876    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z8nx5" podUID="7bcabfd1-e72e-4866-9190-6bcce1f7ce64"
Sep 17 17:34:40.100161 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:40.099977    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p7vh6" podUID="39af04b6-713b-406b-952d-7b2ef2b2b952"
Sep 17 17:34:40.100161 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:40.100007    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7spmx"
Sep 17 17:34:40.100161 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:40.100055    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7spmx" podUID="dbb6518b-5a9b-4a33-89a1-71c9e4db70a5"
Sep 17 17:34:41.730471 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:41.730432    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39af04b6-713b-406b-952d-7b2ef2b2b952-metrics-certs\") pod \"network-metrics-daemon-p7vh6\" (UID: \"39af04b6-713b-406b-952d-7b2ef2b2b952\") " pod="openshift-multus/network-metrics-daemon-p7vh6"
Sep 17 17:34:41.730981 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:41.730604    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:41.730981 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:41.730690    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39af04b6-713b-406b-952d-7b2ef2b2b952-metrics-certs podName:39af04b6-713b-406b-952d-7b2ef2b2b952 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:57.730668173 +0000 UTC m=+34.144458611 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/39af04b6-713b-406b-952d-7b2ef2b2b952-metrics-certs") pod "network-metrics-daemon-p7vh6" (UID: "39af04b6-713b-406b-952d-7b2ef2b2b952") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:41.831485 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:41.831443    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rlql\" (UniqueName: \"kubernetes.io/projected/7bcabfd1-e72e-4866-9190-6bcce1f7ce64-kube-api-access-4rlql\") pod \"network-check-target-z8nx5\" (UID: \"7bcabfd1-e72e-4866-9190-6bcce1f7ce64\") " pod="openshift-network-diagnostics/network-check-target-z8nx5"
Sep 17 17:34:41.831649 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:41.831514    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dbb6518b-5a9b-4a33-89a1-71c9e4db70a5-original-pull-secret\") pod \"global-pull-secret-syncer-7spmx\" (UID: \"dbb6518b-5a9b-4a33-89a1-71c9e4db70a5\") " pod="kube-system/global-pull-secret-syncer-7spmx"
Sep 17 17:34:41.831649 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:41.831581    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:34:41.831649 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:41.831605    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:34:41.831649 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:41.831617    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:41.831798 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:41.831677    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbb6518b-5a9b-4a33-89a1-71c9e4db70a5-original-pull-secret podName:dbb6518b-5a9b-4a33-89a1-71c9e4db70a5 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:57.831657897 +0000 UTC m=+34.245448319 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dbb6518b-5a9b-4a33-89a1-71c9e4db70a5-original-pull-secret") pod "global-pull-secret-syncer-7spmx" (UID: "dbb6518b-5a9b-4a33-89a1-71c9e4db70a5") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:41.831798 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:41.831620    2585 projected.go:196] Error preparing data for projected volume kube-api-access-4rlql for pod openshift-network-diagnostics/network-check-target-z8nx5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:41.831798 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:41.831744    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7bcabfd1-e72e-4866-9190-6bcce1f7ce64-kube-api-access-4rlql podName:7bcabfd1-e72e-4866-9190-6bcce1f7ce64 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:57.831732502 +0000 UTC m=+34.245522922 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4rlql" (UniqueName: "kubernetes.io/projected/7bcabfd1-e72e-4866-9190-6bcce1f7ce64-kube-api-access-4rlql") pod "network-check-target-z8nx5" (UID: "7bcabfd1-e72e-4866-9190-6bcce1f7ce64") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:42.099333 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:42.099294    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p7vh6"
Sep 17 17:34:42.099333 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:42.099303    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z8nx5"
Sep 17 17:34:42.099606 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:42.099430    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p7vh6" podUID="39af04b6-713b-406b-952d-7b2ef2b2b952"
Sep 17 17:34:42.099606 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:42.099483    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7spmx"
Sep 17 17:34:42.099606 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:42.099484    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z8nx5" podUID="7bcabfd1-e72e-4866-9190-6bcce1f7ce64"
Sep 17 17:34:42.099606 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:42.099564    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7spmx" podUID="dbb6518b-5a9b-4a33-89a1-71c9e4db70a5"
Sep 17 17:34:44.099440 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:44.099411    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7spmx"
Sep 17 17:34:44.099787 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:44.099548    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7spmx" podUID="dbb6518b-5a9b-4a33-89a1-71c9e4db70a5"
Sep 17 17:34:44.099857 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:44.099799    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p7vh6"
Sep 17 17:34:44.100088 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:44.099903    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p7vh6" podUID="39af04b6-713b-406b-952d-7b2ef2b2b952"
Sep 17 17:34:44.100948 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:44.100927    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z8nx5"
Sep 17 17:34:44.101034 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:44.101016    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z8nx5" podUID="7bcabfd1-e72e-4866-9190-6bcce1f7ce64"
Sep 17 17:34:45.173761 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:45.173527    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jpg4z" event={"ID":"856535bd-6884-4e98-8089-852deae228aa","Type":"ContainerStarted","Data":"bb86afa88ac54da360d096702a8f03537bba6c02b630ede181746d1d75ae7cc2"}
Sep 17 17:34:45.175296 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:45.175263    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pcc9f" event={"ID":"51e093e4-5ec3-4086-985f-3022f731690b","Type":"ContainerStarted","Data":"cce67d18e9741484c4c6ce51ba630eed5110a0e0ebbb650c4f86b8dbc6934903"}
Sep 17 17:34:45.178448 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:45.178419    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rrf79_beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5/ovn-acl-logging/0.log"
Sep 17 17:34:45.178871 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:45.178849    2585 generic.go:356] "Generic (PLEG): container finished" podID="beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5" containerID="69e56fbec4e05e149e8f66bedc85f4365850b88de14c2dabd9c1c72a207fbac6" exitCode=1
Sep 17 17:34:45.178956 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:45.178922    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rrf79" event={"ID":"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5","Type":"ContainerStarted","Data":"65e1cac17c474360402b1a15ae27a73c54cab44fec19636b7987561561a444c7"}
Sep 17 17:34:45.179011 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:45.178979    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rrf79" event={"ID":"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5","Type":"ContainerStarted","Data":"f0b8f2858a78eedb5dece6e40a8ba4e0e5603c9fd839245e466af4ae7500a6c8"}
Sep 17 17:34:45.179011 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:45.179010    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rrf79" event={"ID":"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5","Type":"ContainerStarted","Data":"a36b9b15adf677074c48d2912c12cba1aaab3065d6f832989e0c8794b54c5c4d"}
Sep 17 17:34:45.179119 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:45.179050    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rrf79" event={"ID":"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5","Type":"ContainerStarted","Data":"147fe8e6f0d3184f9c7d7587738cf83578d3be66d3cd299a9c41131a8d051ecb"}
Sep 17 17:34:45.179119 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:45.179083    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rrf79" event={"ID":"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5","Type":"ContainerDied","Data":"69e56fbec4e05e149e8f66bedc85f4365850b88de14c2dabd9c1c72a207fbac6"}
Sep 17 17:34:45.179191 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:45.179129    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rrf79" event={"ID":"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5","Type":"ContainerStarted","Data":"2ae1bc2031ae9cc8cab108d26a4b00ff0818879a0beba42f358cca8746cddabc"}
Sep 17 17:34:45.180527 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:45.180504    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-d972p" event={"ID":"9ef20182-de84-4cba-9eb4-1fcfdfbd1abe","Type":"ContainerStarted","Data":"32c2d397e8b20c5ef2c89b8345a8966863c824e5d748e391d184dfd114f766e6"}
Sep 17 17:34:45.182129 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:45.182107    2585 generic.go:356] "Generic (PLEG): container finished" podID="751c4ff3-f343-4280-b970-3583c2a4c28b" containerID="3d41f9d0589870a6926ea76529631e3b798550da2bad72729025b47c3d1aa2b3" exitCode=0
Sep 17 17:34:45.182222 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:45.182183    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9tfrw" event={"ID":"751c4ff3-f343-4280-b970-3583c2a4c28b","Type":"ContainerDied","Data":"3d41f9d0589870a6926ea76529631e3b798550da2bad72729025b47c3d1aa2b3"}
Sep 17 17:34:45.183851 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:45.183826    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgp7g" event={"ID":"fced737e-fcb0-4559-b00b-e4813b035223","Type":"ContainerStarted","Data":"6094ca19999f11db0b1377b6c7e696ba53e2fdb9dc83e6d85d686b4da9f17736"}
Sep 17 17:34:45.185429 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:45.185409    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mglss" event={"ID":"e8bda048-e7cf-4118-82c4-bd1d987e4566","Type":"ContainerStarted","Data":"dec2571247a394cbc98224b74858f863c97c3ae237ac09d804d71d49eb486df5"}
Sep 17 17:34:45.193664 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:45.193617    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-jpg4z" podStartSLOduration=4.112830802 podStartE2EDuration="21.193605534s" totalImagesPullingTime="17.080774732s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:45.193217185 +0000 UTC m=+21.607007644" watchObservedRunningTime="2026-09-17 17:34:45.193605534 +0000 UTC m=+21.607395971"
Sep 17 17:34:45.212472 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:45.212428    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mglss" podStartSLOduration=4.116248827 podStartE2EDuration="21.212412998s" totalImagesPullingTime="17.096164171s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:45.212095723 +0000 UTC m=+21.625886161" watchObservedRunningTime="2026-09-17 17:34:45.212412998 +0000 UTC m=+21.626203442"
Sep 17 17:34:45.244878 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:45.244857    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-jpg4z"
Sep 17 17:34:45.245355 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:45.245338    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-jpg4z"
Sep 17 17:34:45.266086 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:45.266032    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-pcc9f" podStartSLOduration=4.179246346 podStartE2EDuration="21.266015671s" totalImagesPullingTime="17.086769325s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:45.265707391 +0000 UTC m=+21.679497830" watchObservedRunningTime="2026-09-17 17:34:45.266015671 +0000 UTC m=+21.679806110"
Sep 17 17:34:45.289180 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:45.289144    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-d972p" podStartSLOduration=3.520697193 podStartE2EDuration="21.289132703s" totalImagesPullingTime="17.76843551s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:45.288011194 +0000 UTC m=+21.701801632" watchObservedRunningTime="2026-09-17 17:34:45.289132703 +0000 UTC m=+21.702923140"
Sep 17 17:34:45.755815 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:45.755782    2585 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 17:34:46.068548 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:46.068351    2585 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T17:34:45.755810617Z","UUID":"ac050c8a-762a-4d53-9b40-c557fdd509ee","Handler":null,"Name":"","Endpoint":""}
Sep 17 17:34:46.070120 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:46.070092    2585 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 17:34:46.070120 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:46.070125    2585 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 17:34:46.103203 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:46.103175    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p7vh6"
Sep 17 17:34:46.103405 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:46.103295    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z8nx5"
Sep 17 17:34:46.103405 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:46.103306    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7spmx"
Sep 17 17:34:46.103405 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:46.103302    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p7vh6" podUID="39af04b6-713b-406b-952d-7b2ef2b2b952"
Sep 17 17:34:46.103532 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:46.103406    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z8nx5" podUID="7bcabfd1-e72e-4866-9190-6bcce1f7ce64"
Sep 17 17:34:46.103532 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:46.103506    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7spmx" podUID="dbb6518b-5a9b-4a33-89a1-71c9e4db70a5"
Sep 17 17:34:46.191152 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:46.191115    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lt9dx" event={"ID":"a40b9f88-37de-405c-9702-3a73548b4f63","Type":"ContainerStarted","Data":"bd636bff9d933a6a86738844bc8b9f767c15a8114e4c4198cffe87a6d5a43665"}
Sep 17 17:34:46.194835 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:46.193971    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgp7g" event={"ID":"fced737e-fcb0-4559-b00b-e4813b035223","Type":"ContainerStarted","Data":"38fbf489be276a81f83eff1d21880a8f8776c2fcd681d8d2dc412dcb2106d012"}
Sep 17 17:34:46.194835 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:46.194383    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-jpg4z"
Sep 17 17:34:46.195012 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:46.194890    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jpg4z"
Sep 17 17:34:46.217171 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:46.217130    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-lt9dx" podStartSLOduration=5.126179668 podStartE2EDuration="22.217113699s" totalImagesPullingTime="17.090934031s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:46.216643827 +0000 UTC m=+22.630434266" watchObservedRunningTime="2026-09-17 17:34:46.217113699 +0000 UTC m=+22.630904138"
Sep 17 17:34:46.789124 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:46.789098    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-kgbmb"]
Sep 17 17:34:46.805121 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:46.805099    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kgbmb"
Sep 17 17:34:46.808152 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:46.808134    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qqsk8\""
Sep 17 17:34:46.808510 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:46.808474    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 17:34:46.808960 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:46.808940    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 17:34:46.863974 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:46.863947    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6c27507b-19a0-4c12-a379-10474f260f84-hosts-file\") pod \"node-resolver-kgbmb\" (UID: \"6c27507b-19a0-4c12-a379-10474f260f84\") " pod="openshift-dns/node-resolver-kgbmb"
Sep 17 17:34:46.864062 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:46.863994    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9fqw\" (UniqueName: \"kubernetes.io/projected/6c27507b-19a0-4c12-a379-10474f260f84-kube-api-access-g9fqw\") pod \"node-resolver-kgbmb\" (UID: \"6c27507b-19a0-4c12-a379-10474f260f84\") " pod="openshift-dns/node-resolver-kgbmb"
Sep 17 17:34:46.864135 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:46.864111    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c27507b-19a0-4c12-a379-10474f260f84-tmp-dir\") pod \"node-resolver-kgbmb\" (UID: \"6c27507b-19a0-4c12-a379-10474f260f84\") " pod="openshift-dns/node-resolver-kgbmb"
Sep 17 17:34:46.964508 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:46.964463    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c27507b-19a0-4c12-a379-10474f260f84-tmp-dir\") pod \"node-resolver-kgbmb\" (UID: \"6c27507b-19a0-4c12-a379-10474f260f84\") " pod="openshift-dns/node-resolver-kgbmb"
Sep 17 17:34:46.964613 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:46.964549    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6c27507b-19a0-4c12-a379-10474f260f84-hosts-file\") pod \"node-resolver-kgbmb\" (UID: \"6c27507b-19a0-4c12-a379-10474f260f84\") " pod="openshift-dns/node-resolver-kgbmb"
Sep 17 17:34:46.964613 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:46.964577    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9fqw\" (UniqueName: \"kubernetes.io/projected/6c27507b-19a0-4c12-a379-10474f260f84-kube-api-access-g9fqw\") pod \"node-resolver-kgbmb\" (UID: \"6c27507b-19a0-4c12-a379-10474f260f84\") " pod="openshift-dns/node-resolver-kgbmb"
Sep 17 17:34:46.965128 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:46.965106    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c27507b-19a0-4c12-a379-10474f260f84-tmp-dir\") pod \"node-resolver-kgbmb\" (UID: \"6c27507b-19a0-4c12-a379-10474f260f84\") " pod="openshift-dns/node-resolver-kgbmb"
Sep 17 17:34:46.965206 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:46.965185    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6c27507b-19a0-4c12-a379-10474f260f84-hosts-file\") pod \"node-resolver-kgbmb\" (UID: \"6c27507b-19a0-4c12-a379-10474f260f84\") " pod="openshift-dns/node-resolver-kgbmb"
Sep 17 17:34:46.977947 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:46.977923    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9fqw\" (UniqueName: \"kubernetes.io/projected/6c27507b-19a0-4c12-a379-10474f260f84-kube-api-access-g9fqw\") pod \"node-resolver-kgbmb\" (UID: \"6c27507b-19a0-4c12-a379-10474f260f84\") " pod="openshift-dns/node-resolver-kgbmb"
Sep 17 17:34:47.115399 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:47.115365    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kgbmb"
Sep 17 17:34:47.148981 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:47.148956    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c27507b_19a0_4c12_a379_10474f260f84.slice/crio-ae67d2356ba9dd294111a96b32abaac2c4579e885761c1f7d1f667948bb62453 WatchSource:0}: Error finding container ae67d2356ba9dd294111a96b32abaac2c4579e885761c1f7d1f667948bb62453: Status 404 returned error can't find the container with id ae67d2356ba9dd294111a96b32abaac2c4579e885761c1f7d1f667948bb62453
Sep 17 17:34:47.199609 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:47.199580    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rrf79_beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5/ovn-acl-logging/0.log"
Sep 17 17:34:47.200054 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:47.200024    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rrf79" event={"ID":"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5","Type":"ContainerStarted","Data":"0c6f8bfb5dcf9aa070e6ee3d5815757acf6aa0741233331e81f370043339a842"}
Sep 17 17:34:47.202827 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:47.202778    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgp7g" event={"ID":"fced737e-fcb0-4559-b00b-e4813b035223","Type":"ContainerStarted","Data":"a183462242d4b66c4f0273681b2e05e8ac13879e86a1a2b5874a602fab8056c5"}
Sep 17 17:34:47.204592 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:47.204547    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kgbmb" event={"ID":"6c27507b-19a0-4c12-a379-10474f260f84","Type":"ContainerStarted","Data":"ae67d2356ba9dd294111a96b32abaac2c4579e885761c1f7d1f667948bb62453"}
Sep 17 17:34:48.102573 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:48.102546    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z8nx5"
Sep 17 17:34:48.102573 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:48.102569    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p7vh6"
Sep 17 17:34:48.102870 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:48.102547    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7spmx"
Sep 17 17:34:48.102870 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:48.102652    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z8nx5" podUID="7bcabfd1-e72e-4866-9190-6bcce1f7ce64"
Sep 17 17:34:48.102870 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:48.102735    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7spmx" podUID="dbb6518b-5a9b-4a33-89a1-71c9e4db70a5"
Sep 17 17:34:48.102870 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:48.102806    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p7vh6" podUID="39af04b6-713b-406b-952d-7b2ef2b2b952"
Sep 17 17:34:48.208676 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:48.208643    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kgbmb" event={"ID":"6c27507b-19a0-4c12-a379-10474f260f84","Type":"ContainerStarted","Data":"ed1d679eccecf4f7449089f4beae6655b9bb4e328577f29d6dfa414d5e5aa856"}
Sep 17 17:34:48.232591 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:48.232551    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cgp7g" podStartSLOduration=4.978637217 podStartE2EDuration="24.232537887s" totalImagesPullingTime="19.25390067s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:47.228399702 +0000 UTC m=+23.642190220" watchObservedRunningTime="2026-09-17 17:34:48.232537887 +0000 UTC m=+24.646328328"
Sep 17 17:34:50.099176 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:50.098995    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p7vh6"
Sep 17 17:34:50.099749 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:50.098995    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7spmx"
Sep 17 17:34:50.099749 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:50.099259    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p7vh6" podUID="39af04b6-713b-406b-952d-7b2ef2b2b952"
Sep 17 17:34:50.099749 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:50.099023    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z8nx5"
Sep 17 17:34:50.099749 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:50.099320    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7spmx" podUID="dbb6518b-5a9b-4a33-89a1-71c9e4db70a5"
Sep 17 17:34:50.099749 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:50.099415    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z8nx5" podUID="7bcabfd1-e72e-4866-9190-6bcce1f7ce64"
Sep 17 17:34:50.214645 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:50.214616    2585 generic.go:356] "Generic (PLEG): container finished" podID="751c4ff3-f343-4280-b970-3583c2a4c28b" containerID="141fafbf6ede48f4db954b7a19ee9a934620159f6c53ef569cea67f3c93ac7b2" exitCode=0
Sep 17 17:34:50.214877 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:50.214698    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9tfrw" event={"ID":"751c4ff3-f343-4280-b970-3583c2a4c28b","Type":"ContainerDied","Data":"141fafbf6ede48f4db954b7a19ee9a934620159f6c53ef569cea67f3c93ac7b2"}
Sep 17 17:34:50.218396 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:50.218381    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rrf79_beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5/ovn-acl-logging/0.log"
Sep 17 17:34:50.218734 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:50.218714    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rrf79" event={"ID":"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5","Type":"ContainerStarted","Data":"ce3cdf821fd3c2e6eaab67e2dffb2e388546599264ff0429ce3b5b182bf41a9d"}
Sep 17 17:34:50.218946 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:50.218928    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:50.219123 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:50.219099    2585 scope.go:117] "RemoveContainer" containerID="69e56fbec4e05e149e8f66bedc85f4365850b88de14c2dabd9c1c72a207fbac6"
Sep 17 17:34:50.234481 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:50.234457    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:50.253522 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:50.253465    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-kgbmb" podStartSLOduration=4.253454098 podStartE2EDuration="4.253454098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:48.233322193 +0000 UTC m=+24.647112633" watchObservedRunningTime="2026-09-17 17:34:50.253454098 +0000 UTC m=+26.667244581"
Sep 17 17:34:51.194798 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:51.194540    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-p7vh6"]
Sep 17 17:34:51.195324 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:51.194834    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p7vh6"
Sep 17 17:34:51.195324 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:51.194921    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p7vh6" podUID="39af04b6-713b-406b-952d-7b2ef2b2b952"
Sep 17 17:34:51.199314 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:51.199261    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-z8nx5"]
Sep 17 17:34:51.199434 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:51.199373    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z8nx5"
Sep 17 17:34:51.199474 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:51.199448    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z8nx5" podUID="7bcabfd1-e72e-4866-9190-6bcce1f7ce64"
Sep 17 17:34:51.206523 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:51.206499    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7spmx"]
Sep 17 17:34:51.206620 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:51.206583    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7spmx"
Sep 17 17:34:51.206701 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:51.206657    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7spmx" podUID="dbb6518b-5a9b-4a33-89a1-71c9e4db70a5"
Sep 17 17:34:51.225316 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:51.225296    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rrf79_beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5/ovn-acl-logging/0.log"
Sep 17 17:34:51.225746 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:51.225719    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rrf79" event={"ID":"beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5","Type":"ContainerStarted","Data":"3fb18a9b31c317b1f1547067a0e490511668b7358c0b2e9dd92a2c6f8391d439"}
Sep 17 17:34:51.225861 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:51.225803    2585 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 17:34:51.226306 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:51.226290    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:51.228342 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:51.228314    2585 generic.go:356] "Generic (PLEG): container finished" podID="751c4ff3-f343-4280-b970-3583c2a4c28b" containerID="c5a83146ea1d61ea83732df2accec28069c478aee1dd5ed2527f5e00511c96a2" exitCode=0
Sep 17 17:34:51.228435 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:51.228358    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9tfrw" event={"ID":"751c4ff3-f343-4280-b970-3583c2a4c28b","Type":"ContainerDied","Data":"c5a83146ea1d61ea83732df2accec28069c478aee1dd5ed2527f5e00511c96a2"}
Sep 17 17:34:51.244148 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:51.244129    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:51.273830 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:51.273794    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-rrf79" podStartSLOduration=10.148303752 podStartE2EDuration="27.273781162s" totalImagesPullingTime="17.12547741s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:51.272288084 +0000 UTC m=+27.686078521" watchObservedRunningTime="2026-09-17 17:34:51.273781162 +0000 UTC m=+27.687571599"
Sep 17 17:34:52.233405 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:52.233317    2585 generic.go:356] "Generic (PLEG): container finished" podID="751c4ff3-f343-4280-b970-3583c2a4c28b" containerID="62742d01af336c16e65e7697f8a531b148421bd180688125fe882af4d9a2eaba" exitCode=0
Sep 17 17:34:52.233815 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:52.233402    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9tfrw" event={"ID":"751c4ff3-f343-4280-b970-3583c2a4c28b","Type":"ContainerDied","Data":"62742d01af336c16e65e7697f8a531b148421bd180688125fe882af4d9a2eaba"}
Sep 17 17:34:52.233815 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:52.233684    2585 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 17:34:52.562205 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:52.562178    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:34:53.100000 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:53.099967    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z8nx5"
Sep 17 17:34:53.100151 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:53.099966    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p7vh6"
Sep 17 17:34:53.100151 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:53.100082    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z8nx5" podUID="7bcabfd1-e72e-4866-9190-6bcce1f7ce64"
Sep 17 17:34:53.100151 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:53.099966    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7spmx"
Sep 17 17:34:53.100151 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:53.100144    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p7vh6" podUID="39af04b6-713b-406b-952d-7b2ef2b2b952"
Sep 17 17:34:53.100346 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:53.100245    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7spmx" podUID="dbb6518b-5a9b-4a33-89a1-71c9e4db70a5"
Sep 17 17:34:54.160028 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:54.159936    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:54.250423 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:54.250383    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-rrf79" podUID="beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5" containerName="ovnkube-controller" probeResult="failure" output=""
Sep 17 17:34:54.266702 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:54.266675    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:55.099517 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:55.099463    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7spmx"
Sep 17 17:34:55.099680 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:55.099465    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p7vh6"
Sep 17 17:34:55.099680 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:55.099596    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7spmx" podUID="dbb6518b-5a9b-4a33-89a1-71c9e4db70a5"
Sep 17 17:34:55.099680 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:55.099606    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z8nx5"
Sep 17 17:34:55.099839 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:55.099677    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p7vh6" podUID="39af04b6-713b-406b-952d-7b2ef2b2b952"
Sep 17 17:34:55.099839 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:55.099728    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z8nx5" podUID="7bcabfd1-e72e-4866-9190-6bcce1f7ce64"
Sep 17 17:34:56.342723 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:56.342689    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:56.443884 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:56.443850    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:56.626672 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:56.626605    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:56.695295 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:56.695255    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:57.099909 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:57.099881    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z8nx5"
Sep 17 17:34:57.100128 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:57.099978    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p7vh6"
Sep 17 17:34:57.100128 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:57.099987    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z8nx5" podUID="7bcabfd1-e72e-4866-9190-6bcce1f7ce64"
Sep 17 17:34:57.100128 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:57.100065    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7spmx"
Sep 17 17:34:57.100128 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:57.100067    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p7vh6" podUID="39af04b6-713b-406b-952d-7b2ef2b2b952"
Sep 17 17:34:57.100254 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:57.100140    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7spmx" podUID="dbb6518b-5a9b-4a33-89a1-71c9e4db70a5"
Sep 17 17:34:57.248034 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:57.248006    2585 generic.go:356] "Generic (PLEG): container finished" podID="751c4ff3-f343-4280-b970-3583c2a4c28b" containerID="0b9e24c69a47066a8426ab262b42b2a671b6f898e51b6b698ec68b6ab76007e4" exitCode=0
Sep 17 17:34:57.248171 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:57.248061    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9tfrw" event={"ID":"751c4ff3-f343-4280-b970-3583c2a4c28b","Type":"ContainerDied","Data":"0b9e24c69a47066a8426ab262b42b2a671b6f898e51b6b698ec68b6ab76007e4"}
Sep 17 17:34:57.421809 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:57.421789    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-139.ec2.internal" event="NodeReady"
Sep 17 17:34:57.422152 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:57.421946    2585 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 17:34:57.477986 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:57.477958    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-t7fvf"]
Sep 17 17:34:57.483878 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:57.483861    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jvtw5"]
Sep 17 17:34:57.483987 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:57.483973    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-t7fvf"
Sep 17 17:34:57.486711 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:57.486682    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 17:34:57.487961 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:57.486917    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 17:34:57.487961 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:57.487157    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bqqq4\""
Sep 17 17:34:57.491965 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:57.491943    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jvtw5"
Sep 17 17:34:57.494249 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:57.494232    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-mjhsr\""
Sep 17 17:34:57.494349 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:57.494328    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 17:34:57.494527 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:57.494511    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 17:34:57.494597 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:57.494555    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 17:34:57.500278 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:57.500258    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-t7fvf"]
Sep 17 17:34:57.500798 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:57.500782    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jvtw5"]
Sep 17 17:34:57.648268 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:57.648193    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3973eda4-9544-4956-980a-4c11ba7700c1-config-volume\") pod \"dns-default-t7fvf\" (UID: \"3973eda4-9544-4956-980a-4c11ba7700c1\") " pod="openshift-dns/dns-default-t7fvf"
Sep 17 17:34:57.648268 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:57.648231    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgfjq\" (UniqueName: \"kubernetes.io/projected/98e53603-643c-4852-b805-5e07c9016190-kube-api-access-fgfjq\") pod \"ingress-canary-jvtw5\" (UID: \"98e53603-643c-4852-b805-5e07c9016190\") " pod="openshift-ingress-canary/ingress-canary-jvtw5"
Sep 17 17:34:57.648268 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:57.648260    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwgjz\" (UniqueName: \"kubernetes.io/projected/3973eda4-9544-4956-980a-4c11ba7700c1-kube-api-access-gwgjz\") pod \"dns-default-t7fvf\" (UID: \"3973eda4-9544-4956-980a-4c11ba7700c1\") " pod="openshift-dns/dns-default-t7fvf"
Sep 17 17:34:57.648462 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:57.648315    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3973eda4-9544-4956-980a-4c11ba7700c1-metrics-tls\") pod \"dns-default-t7fvf\" (UID: \"3973eda4-9544-4956-980a-4c11ba7700c1\") " pod="openshift-dns/dns-default-t7fvf"
Sep 17 17:34:57.648462 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:57.648335    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/98e53603-643c-4852-b805-5e07c9016190-cert\") pod \"ingress-canary-jvtw5\" (UID: \"98e53603-643c-4852-b805-5e07c9016190\") " pod="openshift-ingress-canary/ingress-canary-jvtw5"
Sep 17 17:34:57.648462 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:57.648348    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3973eda4-9544-4956-980a-4c11ba7700c1-tmp-dir\") pod \"dns-default-t7fvf\" (UID: \"3973eda4-9544-4956-980a-4c11ba7700c1\") " pod="openshift-dns/dns-default-t7fvf"
Sep 17 17:34:57.749533 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:57.749486    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3973eda4-9544-4956-980a-4c11ba7700c1-config-volume\") pod \"dns-default-t7fvf\" (UID: \"3973eda4-9544-4956-980a-4c11ba7700c1\") " pod="openshift-dns/dns-default-t7fvf"
Sep 17 17:34:57.749610 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:57.749538    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgfjq\" (UniqueName: \"kubernetes.io/projected/98e53603-643c-4852-b805-5e07c9016190-kube-api-access-fgfjq\") pod \"ingress-canary-jvtw5\" (UID: \"98e53603-643c-4852-b805-5e07c9016190\") " pod="openshift-ingress-canary/ingress-canary-jvtw5"
Sep 17 17:34:57.749610 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:57.749562    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39af04b6-713b-406b-952d-7b2ef2b2b952-metrics-certs\") pod \"network-metrics-daemon-p7vh6\" (UID: \"39af04b6-713b-406b-952d-7b2ef2b2b952\") " pod="openshift-multus/network-metrics-daemon-p7vh6"
Sep 17 17:34:57.749610 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:57.749581    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwgjz\" (UniqueName: \"kubernetes.io/projected/3973eda4-9544-4956-980a-4c11ba7700c1-kube-api-access-gwgjz\") pod \"dns-default-t7fvf\" (UID: \"3973eda4-9544-4956-980a-4c11ba7700c1\") " pod="openshift-dns/dns-default-t7fvf"
Sep 17 17:34:57.749740 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:57.749630    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3973eda4-9544-4956-980a-4c11ba7700c1-metrics-tls\") pod \"dns-default-t7fvf\" (UID: \"3973eda4-9544-4956-980a-4c11ba7700c1\") " pod="openshift-dns/dns-default-t7fvf"
Sep 17 17:34:57.749740 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:57.749662    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/98e53603-643c-4852-b805-5e07c9016190-cert\") pod \"ingress-canary-jvtw5\" (UID: \"98e53603-643c-4852-b805-5e07c9016190\") " pod="openshift-ingress-canary/ingress-canary-jvtw5"
Sep 17 17:34:57.749740 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:57.749683    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3973eda4-9544-4956-980a-4c11ba7700c1-tmp-dir\") pod \"dns-default-t7fvf\" (UID: \"3973eda4-9544-4956-980a-4c11ba7700c1\") " pod="openshift-dns/dns-default-t7fvf"
Sep 17 17:34:57.749740 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:57.749688    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:57.749973 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:57.749771    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39af04b6-713b-406b-952d-7b2ef2b2b952-metrics-certs podName:39af04b6-713b-406b-952d-7b2ef2b2b952 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:29.749751164 +0000 UTC m=+66.163541603 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/39af04b6-713b-406b-952d-7b2ef2b2b952-metrics-certs") pod "network-metrics-daemon-p7vh6" (UID: "39af04b6-713b-406b-952d-7b2ef2b2b952") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 17:34:57.749973 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:57.749912    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:34:57.750081 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:57.749971    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3973eda4-9544-4956-980a-4c11ba7700c1-metrics-tls podName:3973eda4-9544-4956-980a-4c11ba7700c1 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:58.249954978 +0000 UTC m=+34.663745394 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3973eda4-9544-4956-980a-4c11ba7700c1-metrics-tls") pod "dns-default-t7fvf" (UID: "3973eda4-9544-4956-980a-4c11ba7700c1") : secret "dns-default-metrics-tls" not found
Sep 17 17:34:57.750081 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:57.750041    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3973eda4-9544-4956-980a-4c11ba7700c1-tmp-dir\") pod \"dns-default-t7fvf\" (UID: \"3973eda4-9544-4956-980a-4c11ba7700c1\") " pod="openshift-dns/dns-default-t7fvf"
Sep 17 17:34:57.750081 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:57.750058    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3973eda4-9544-4956-980a-4c11ba7700c1-config-volume\") pod \"dns-default-t7fvf\" (UID: \"3973eda4-9544-4956-980a-4c11ba7700c1\") " pod="openshift-dns/dns-default-t7fvf"
Sep 17 17:34:57.753720 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:57.753698    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/98e53603-643c-4852-b805-5e07c9016190-cert\") pod \"ingress-canary-jvtw5\" (UID: \"98e53603-643c-4852-b805-5e07c9016190\") " pod="openshift-ingress-canary/ingress-canary-jvtw5"
Sep 17 17:34:57.758826 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:57.758806    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwgjz\" (UniqueName: \"kubernetes.io/projected/3973eda4-9544-4956-980a-4c11ba7700c1-kube-api-access-gwgjz\") pod \"dns-default-t7fvf\" (UID: \"3973eda4-9544-4956-980a-4c11ba7700c1\") " pod="openshift-dns/dns-default-t7fvf"
Sep 17 17:34:57.759172 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:57.759158    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgfjq\" (UniqueName: \"kubernetes.io/projected/98e53603-643c-4852-b805-5e07c9016190-kube-api-access-fgfjq\") pod \"ingress-canary-jvtw5\" (UID: \"98e53603-643c-4852-b805-5e07c9016190\") " pod="openshift-ingress-canary/ingress-canary-jvtw5"
Sep 17 17:34:57.800887 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:57.800848    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jvtw5"
Sep 17 17:34:57.850682 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:57.850656    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rlql\" (UniqueName: \"kubernetes.io/projected/7bcabfd1-e72e-4866-9190-6bcce1f7ce64-kube-api-access-4rlql\") pod \"network-check-target-z8nx5\" (UID: \"7bcabfd1-e72e-4866-9190-6bcce1f7ce64\") " pod="openshift-network-diagnostics/network-check-target-z8nx5"
Sep 17 17:34:57.850790 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:57.850700    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dbb6518b-5a9b-4a33-89a1-71c9e4db70a5-original-pull-secret\") pod \"global-pull-secret-syncer-7spmx\" (UID: \"dbb6518b-5a9b-4a33-89a1-71c9e4db70a5\") " pod="kube-system/global-pull-secret-syncer-7spmx"
Sep 17 17:34:57.850847 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:57.850818    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 17:34:57.850847 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:57.850836    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 17:34:57.850847 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:57.850846    2585 projected.go:196] Error preparing data for projected volume kube-api-access-4rlql for pod openshift-network-diagnostics/network-check-target-z8nx5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:57.850972 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:57.850849    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:57.850972 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:57.850905    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbb6518b-5a9b-4a33-89a1-71c9e4db70a5-original-pull-secret podName:dbb6518b-5a9b-4a33-89a1-71c9e4db70a5 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:29.850886023 +0000 UTC m=+66.264676453 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dbb6518b-5a9b-4a33-89a1-71c9e4db70a5-original-pull-secret") pod "global-pull-secret-syncer-7spmx" (UID: "dbb6518b-5a9b-4a33-89a1-71c9e4db70a5") : object "kube-system"/"original-pull-secret" not registered
Sep 17 17:34:57.850972 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:57.850923    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7bcabfd1-e72e-4866-9190-6bcce1f7ce64-kube-api-access-4rlql podName:7bcabfd1-e72e-4866-9190-6bcce1f7ce64 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:29.850914069 +0000 UTC m=+66.264704489 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-4rlql" (UniqueName: "kubernetes.io/projected/7bcabfd1-e72e-4866-9190-6bcce1f7ce64-kube-api-access-4rlql") pod "network-check-target-z8nx5" (UID: "7bcabfd1-e72e-4866-9190-6bcce1f7ce64") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 17:34:58.136664 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:58.136632    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jvtw5"]
Sep 17 17:34:58.141676 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:34:58.141650    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98e53603_643c_4852_b805_5e07c9016190.slice/crio-3568ae888abd4ff665e51ae8cc3b9feb8988a073b4053d4727f4d7885ac0ce37 WatchSource:0}: Error finding container 3568ae888abd4ff665e51ae8cc3b9feb8988a073b4053d4727f4d7885ac0ce37: Status 404 returned error can't find the container with id 3568ae888abd4ff665e51ae8cc3b9feb8988a073b4053d4727f4d7885ac0ce37
Sep 17 17:34:58.253045 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:58.253016    2585 generic.go:356] "Generic (PLEG): container finished" podID="751c4ff3-f343-4280-b970-3583c2a4c28b" containerID="a82fe73df7091cd1c57b2906c319d0fc8b821c2668125122b8958070caf89a89" exitCode=0
Sep 17 17:34:58.253201 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:58.253096    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9tfrw" event={"ID":"751c4ff3-f343-4280-b970-3583c2a4c28b","Type":"ContainerDied","Data":"a82fe73df7091cd1c57b2906c319d0fc8b821c2668125122b8958070caf89a89"}
Sep 17 17:34:58.253676 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:58.253658    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3973eda4-9544-4956-980a-4c11ba7700c1-metrics-tls\") pod \"dns-default-t7fvf\" (UID: \"3973eda4-9544-4956-980a-4c11ba7700c1\") " pod="openshift-dns/dns-default-t7fvf"
Sep 17 17:34:58.253792 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:58.253768    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:34:58.253914 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:58.253826    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3973eda4-9544-4956-980a-4c11ba7700c1-metrics-tls podName:3973eda4-9544-4956-980a-4c11ba7700c1 nodeName:}" failed. No retries permitted until 2026-09-17 17:34:59.253804912 +0000 UTC m=+35.667595328 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3973eda4-9544-4956-980a-4c11ba7700c1-metrics-tls") pod "dns-default-t7fvf" (UID: "3973eda4-9544-4956-980a-4c11ba7700c1") : secret "dns-default-metrics-tls" not found
Sep 17 17:34:58.254254 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:58.254236    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jvtw5" event={"ID":"98e53603-643c-4852-b805-5e07c9016190","Type":"ContainerStarted","Data":"3568ae888abd4ff665e51ae8cc3b9feb8988a073b4053d4727f4d7885ac0ce37"}
Sep 17 17:34:59.099255 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:59.099219    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7spmx"
Sep 17 17:34:59.100117 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:59.099229    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p7vh6"
Sep 17 17:34:59.100117 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:59.099404    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z8nx5"
Sep 17 17:34:59.105001 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:59.104977    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 17:34:59.105001 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:59.104989    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 17:34:59.105769 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:59.105746    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 17:34:59.105868 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:59.105794    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-g24m8\""
Sep 17 17:34:59.106095 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:59.106075    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 17:34:59.106153 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:59.106117    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2vz4h\""
Sep 17 17:34:59.261121 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:59.261092    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3973eda4-9544-4956-980a-4c11ba7700c1-metrics-tls\") pod \"dns-default-t7fvf\" (UID: \"3973eda4-9544-4956-980a-4c11ba7700c1\") " pod="openshift-dns/dns-default-t7fvf"
Sep 17 17:34:59.261291 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:59.261228    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9tfrw" event={"ID":"751c4ff3-f343-4280-b970-3583c2a4c28b","Type":"ContainerStarted","Data":"81d23cbb734e7ce5201d2dd6571dcd2a8185852dd90cefd20f04c6dfa49d428d"}
Sep 17 17:34:59.261342 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:59.261323    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:34:59.261390 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:34:59.261378    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3973eda4-9544-4956-980a-4c11ba7700c1-metrics-tls podName:3973eda4-9544-4956-980a-4c11ba7700c1 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:01.261361763 +0000 UTC m=+37.675152203 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3973eda4-9544-4956-980a-4c11ba7700c1-metrics-tls") pod "dns-default-t7fvf" (UID: "3973eda4-9544-4956-980a-4c11ba7700c1") : secret "dns-default-metrics-tls" not found
Sep 17 17:34:59.452015 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:59.451945    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:59.552807 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:59.552620    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:34:59.653274 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:34:59.653234    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:01.267541 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:01.267506    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jvtw5" event={"ID":"98e53603-643c-4852-b805-5e07c9016190","Type":"ContainerStarted","Data":"3f76084c8c25c2b402613f7d03a7a93a341c1502e24f2d7ecfc7640fbc069ebb"}
Sep 17 17:35:01.276310 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:01.276288    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3973eda4-9544-4956-980a-4c11ba7700c1-metrics-tls\") pod \"dns-default-t7fvf\" (UID: \"3973eda4-9544-4956-980a-4c11ba7700c1\") " pod="openshift-dns/dns-default-t7fvf"
Sep 17 17:35:01.276430 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:35:01.276414    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:35:01.276480 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:35:01.276472    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3973eda4-9544-4956-980a-4c11ba7700c1-metrics-tls podName:3973eda4-9544-4956-980a-4c11ba7700c1 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:05.27645747 +0000 UTC m=+41.690247891 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3973eda4-9544-4956-980a-4c11ba7700c1-metrics-tls") pod "dns-default-t7fvf" (UID: "3973eda4-9544-4956-980a-4c11ba7700c1") : secret "dns-default-metrics-tls" not found
Sep 17 17:35:01.290804 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:01.290765    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9tfrw" podStartSLOduration=10.757158824 podStartE2EDuration="37.290751289s" totalImagesPullingTime="26.533592465s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:24 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:34:59.295961462 +0000 UTC m=+35.709751901" watchObservedRunningTime="2026-09-17 17:35:01.290751289 +0000 UTC m=+37.704541727"
Sep 17 17:35:01.291231 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:01.291208    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jvtw5" podStartSLOduration=2.001220505 podStartE2EDuration="4.291201975s" totalImagesPullingTime="2.28998147s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:01.289790924 +0000 UTC m=+37.703581374" watchObservedRunningTime="2026-09-17 17:35:01.291201975 +0000 UTC m=+37.704992412"
Sep 17 17:35:02.267562 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:02.267527    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:02.373757 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:02.373718    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:05.104837 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:05.104806    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:05.204538 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:05.204505    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:05.303990 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:05.303957    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3973eda4-9544-4956-980a-4c11ba7700c1-metrics-tls\") pod \"dns-default-t7fvf\" (UID: \"3973eda4-9544-4956-980a-4c11ba7700c1\") " pod="openshift-dns/dns-default-t7fvf"
Sep 17 17:35:05.304159 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:35:05.304074    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:35:05.304159 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:35:05.304138    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3973eda4-9544-4956-980a-4c11ba7700c1-metrics-tls podName:3973eda4-9544-4956-980a-4c11ba7700c1 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:13.304118988 +0000 UTC m=+49.717909418 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3973eda4-9544-4956-980a-4c11ba7700c1-metrics-tls") pod "dns-default-t7fvf" (UID: "3973eda4-9544-4956-980a-4c11ba7700c1") : secret "dns-default-metrics-tls" not found
Sep 17 17:35:05.306268 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:05.306243    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:06.125281 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:06.125246    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:06.219297 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:06.219260    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:07.109112 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:07.109070    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:13.356367 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:13.356335    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3973eda4-9544-4956-980a-4c11ba7700c1-metrics-tls\") pod \"dns-default-t7fvf\" (UID: \"3973eda4-9544-4956-980a-4c11ba7700c1\") " pod="openshift-dns/dns-default-t7fvf"
Sep 17 17:35:13.356779 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:35:13.356443    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:35:13.356779 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:35:13.356509    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3973eda4-9544-4956-980a-4c11ba7700c1-metrics-tls podName:3973eda4-9544-4956-980a-4c11ba7700c1 nodeName:}" failed. No retries permitted until 2026-09-17 17:35:29.356478571 +0000 UTC m=+65.770268987 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3973eda4-9544-4956-980a-4c11ba7700c1-metrics-tls") pod "dns-default-t7fvf" (UID: "3973eda4-9544-4956-980a-4c11ba7700c1") : secret "dns-default-metrics-tls" not found
Sep 17 17:35:13.669032 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:13.668951    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7474c64b66-nmmvh"]
Sep 17 17:35:13.697983 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:13.697959    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cc6c86fb4-4nnvr"]
Sep 17 17:35:13.698111 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:13.698073    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7474c64b66-nmmvh"
Sep 17 17:35:13.704679 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:13.704662    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 17:35:13.704756 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:13.704695    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 17:35:13.705228 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:13.705215    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 17 17:35:13.705266 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:13.705218    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 17:35:13.713423 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:13.713403    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cc6c86fb4-4nnvr"]
Sep 17 17:35:13.713423 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:13.713423    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7474c64b66-nmmvh"]
Sep 17 17:35:13.713553 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:13.713523    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cc6c86fb4-4nnvr"
Sep 17 17:35:13.715987 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:13.715972    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-xfmz5\""
Sep 17 17:35:13.717045 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:13.717032    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 17 17:35:13.733197 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:13.733178    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57d467ccf7-rw9lz"]
Sep 17 17:35:13.748390 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:13.748374    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57d467ccf7-rw9lz"
Sep 17 17:35:13.753698 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:13.753678    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 17 17:35:13.753874 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:13.753802    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 17 17:35:13.753986 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:13.753814    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 17 17:35:13.754268 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:13.754123    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 17 17:35:13.754370 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:13.754325    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57d467ccf7-rw9lz"]
Sep 17 17:35:13.859527 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:13.859477    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f2ee4712-52c9-434d-b7f3-247871387e09-ca\") pod \"cluster-proxy-proxy-agent-57d467ccf7-rw9lz\" (UID: \"f2ee4712-52c9-434d-b7f3-247871387e09\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57d467ccf7-rw9lz"
Sep 17 17:35:13.859679 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:13.859538    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f2ee4712-52c9-434d-b7f3-247871387e09-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-57d467ccf7-rw9lz\" (UID: \"f2ee4712-52c9-434d-b7f3-247871387e09\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57d467ccf7-rw9lz"
Sep 17 17:35:13.859679 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:13.859564    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52zkw\" (UniqueName: \"kubernetes.io/projected/f2ee4712-52c9-434d-b7f3-247871387e09-kube-api-access-52zkw\") pod \"cluster-proxy-proxy-agent-57d467ccf7-rw9lz\" (UID: \"f2ee4712-52c9-434d-b7f3-247871387e09\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57d467ccf7-rw9lz"
Sep 17 17:35:13.859679 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:13.859582    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/44e90de9-f4fe-46f6-89ee-bc3c36e6cba6-tmp\") pod \"klusterlet-addon-workmgr-7474c64b66-nmmvh\" (UID: \"44e90de9-f4fe-46f6-89ee-bc3c36e6cba6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7474c64b66-nmmvh"
Sep 17 17:35:13.859679 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:13.859609    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/66e734c2-24ae-4d10-a666-e190f555d036-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-cc6c86fb4-4nnvr\" (UID: \"66e734c2-24ae-4d10-a666-e190f555d036\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cc6c86fb4-4nnvr"
Sep 17 17:35:13.859679 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:13.859631    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf24h\" (UniqueName: \"kubernetes.io/projected/66e734c2-24ae-4d10-a666-e190f555d036-kube-api-access-rf24h\") pod \"managed-serviceaccount-addon-agent-cc6c86fb4-4nnvr\" (UID: \"66e734c2-24ae-4d10-a666-e190f555d036\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cc6c86fb4-4nnvr"
Sep 17 17:35:13.859679 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:13.859650    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2ee4712-52c9-434d-b7f3-247871387e09-additional-service-ca\") pod \"cluster-proxy-proxy-agent-57d467ccf7-rw9lz\" (UID: \"f2ee4712-52c9-434d-b7f3-247871387e09\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57d467ccf7-rw9lz"
Sep 17 17:35:13.859679 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:13.859665    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdbp2\" (UniqueName: \"kubernetes.io/projected/44e90de9-f4fe-46f6-89ee-bc3c36e6cba6-kube-api-access-vdbp2\") pod \"klusterlet-addon-workmgr-7474c64b66-nmmvh\" (UID: \"44e90de9-f4fe-46f6-89ee-bc3c36e6cba6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7474c64b66-nmmvh"
Sep 17 17:35:13.859919 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:13.859698    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/44e90de9-f4fe-46f6-89ee-bc3c36e6cba6-klusterlet-config\") pod \"klusterlet-addon-workmgr-7474c64b66-nmmvh\" (UID: \"44e90de9-f4fe-46f6-89ee-bc3c36e6cba6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7474c64b66-nmmvh"
Sep 17 17:35:13.859919 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:13.859739    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f2ee4712-52c9-434d-b7f3-247871387e09-hub\") pod \"cluster-proxy-proxy-agent-57d467ccf7-rw9lz\" (UID: \"f2ee4712-52c9-434d-b7f3-247871387e09\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57d467ccf7-rw9lz"
Sep 17 17:35:13.859919 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:13.859762    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f2ee4712-52c9-434d-b7f3-247871387e09-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-57d467ccf7-rw9lz\" (UID: \"f2ee4712-52c9-434d-b7f3-247871387e09\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57d467ccf7-rw9lz"
Sep 17 17:35:13.960305 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:13.960223    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/44e90de9-f4fe-46f6-89ee-bc3c36e6cba6-klusterlet-config\") pod \"klusterlet-addon-workmgr-7474c64b66-nmmvh\" (UID: \"44e90de9-f4fe-46f6-89ee-bc3c36e6cba6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7474c64b66-nmmvh"
Sep 17 17:35:13.960305 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:13.960265    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f2ee4712-52c9-434d-b7f3-247871387e09-hub\") pod \"cluster-proxy-proxy-agent-57d467ccf7-rw9lz\" (UID: \"f2ee4712-52c9-434d-b7f3-247871387e09\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57d467ccf7-rw9lz"
Sep 17 17:35:13.960305 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:13.960283    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f2ee4712-52c9-434d-b7f3-247871387e09-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-57d467ccf7-rw9lz\" (UID: \"f2ee4712-52c9-434d-b7f3-247871387e09\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57d467ccf7-rw9lz"
Sep 17 17:35:13.960535 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:13.960367    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f2ee4712-52c9-434d-b7f3-247871387e09-ca\") pod \"cluster-proxy-proxy-agent-57d467ccf7-rw9lz\" (UID: \"f2ee4712-52c9-434d-b7f3-247871387e09\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57d467ccf7-rw9lz"
Sep 17 17:35:13.960535 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:13.960428    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f2ee4712-52c9-434d-b7f3-247871387e09-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-57d467ccf7-rw9lz\" (UID: \"f2ee4712-52c9-434d-b7f3-247871387e09\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57d467ccf7-rw9lz"
Sep 17 17:35:13.960535 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:13.960453    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52zkw\" (UniqueName: \"kubernetes.io/projected/f2ee4712-52c9-434d-b7f3-247871387e09-kube-api-access-52zkw\") pod \"cluster-proxy-proxy-agent-57d467ccf7-rw9lz\" (UID: \"f2ee4712-52c9-434d-b7f3-247871387e09\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57d467ccf7-rw9lz"
Sep 17 17:35:13.960535 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:13.960475    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/44e90de9-f4fe-46f6-89ee-bc3c36e6cba6-tmp\") pod \"klusterlet-addon-workmgr-7474c64b66-nmmvh\" (UID: \"44e90de9-f4fe-46f6-89ee-bc3c36e6cba6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7474c64b66-nmmvh"
Sep 17 17:35:13.960733 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:13.960538    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/66e734c2-24ae-4d10-a666-e190f555d036-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-cc6c86fb4-4nnvr\" (UID: \"66e734c2-24ae-4d10-a666-e190f555d036\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cc6c86fb4-4nnvr"
Sep 17 17:35:13.960733 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:13.960564    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf24h\" (UniqueName: \"kubernetes.io/projected/66e734c2-24ae-4d10-a666-e190f555d036-kube-api-access-rf24h\") pod \"managed-serviceaccount-addon-agent-cc6c86fb4-4nnvr\" (UID: \"66e734c2-24ae-4d10-a666-e190f555d036\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cc6c86fb4-4nnvr"
Sep 17 17:35:13.960733 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:13.960589    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2ee4712-52c9-434d-b7f3-247871387e09-additional-service-ca\") pod \"cluster-proxy-proxy-agent-57d467ccf7-rw9lz\" (UID: \"f2ee4712-52c9-434d-b7f3-247871387e09\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57d467ccf7-rw9lz"
Sep 17 17:35:13.960733 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:13.960610    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdbp2\" (UniqueName: \"kubernetes.io/projected/44e90de9-f4fe-46f6-89ee-bc3c36e6cba6-kube-api-access-vdbp2\") pod \"klusterlet-addon-workmgr-7474c64b66-nmmvh\" (UID: \"44e90de9-f4fe-46f6-89ee-bc3c36e6cba6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7474c64b66-nmmvh"
Sep 17 17:35:13.960986 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:13.960873    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/44e90de9-f4fe-46f6-89ee-bc3c36e6cba6-tmp\") pod \"klusterlet-addon-workmgr-7474c64b66-nmmvh\" (UID: \"44e90de9-f4fe-46f6-89ee-bc3c36e6cba6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7474c64b66-nmmvh"
Sep 17 17:35:13.961590 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:13.961569    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2ee4712-52c9-434d-b7f3-247871387e09-additional-service-ca\") pod \"cluster-proxy-proxy-agent-57d467ccf7-rw9lz\" (UID: \"f2ee4712-52c9-434d-b7f3-247871387e09\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57d467ccf7-rw9lz"
Sep 17 17:35:13.964121 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:13.964074    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f2ee4712-52c9-434d-b7f3-247871387e09-hub\") pod \"cluster-proxy-proxy-agent-57d467ccf7-rw9lz\" (UID: \"f2ee4712-52c9-434d-b7f3-247871387e09\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57d467ccf7-rw9lz"
Sep 17 17:35:13.964121 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:13.964094    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f2ee4712-52c9-434d-b7f3-247871387e09-ca\") pod \"cluster-proxy-proxy-agent-57d467ccf7-rw9lz\" (UID: \"f2ee4712-52c9-434d-b7f3-247871387e09\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57d467ccf7-rw9lz"
Sep 17 17:35:13.964291 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:13.964125    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/44e90de9-f4fe-46f6-89ee-bc3c36e6cba6-klusterlet-config\") pod \"klusterlet-addon-workmgr-7474c64b66-nmmvh\" (UID: \"44e90de9-f4fe-46f6-89ee-bc3c36e6cba6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7474c64b66-nmmvh"
Sep 17 17:35:13.964291 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:13.964226    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f2ee4712-52c9-434d-b7f3-247871387e09-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-57d467ccf7-rw9lz\" (UID: \"f2ee4712-52c9-434d-b7f3-247871387e09\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57d467ccf7-rw9lz"
Sep 17 17:35:13.964291 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:13.964272    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/66e734c2-24ae-4d10-a666-e190f555d036-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-cc6c86fb4-4nnvr\" (UID: \"66e734c2-24ae-4d10-a666-e190f555d036\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cc6c86fb4-4nnvr"
Sep 17 17:35:13.964394 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:13.964319    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f2ee4712-52c9-434d-b7f3-247871387e09-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-57d467ccf7-rw9lz\" (UID: \"f2ee4712-52c9-434d-b7f3-247871387e09\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57d467ccf7-rw9lz"
Sep 17 17:35:13.973073 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:13.973052    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf24h\" (UniqueName: \"kubernetes.io/projected/66e734c2-24ae-4d10-a666-e190f555d036-kube-api-access-rf24h\") pod \"managed-serviceaccount-addon-agent-cc6c86fb4-4nnvr\" (UID: \"66e734c2-24ae-4d10-a666-e190f555d036\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cc6c86fb4-4nnvr"
Sep 17 17:35:13.973159 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:13.973108    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdbp2\" (UniqueName: \"kubernetes.io/projected/44e90de9-f4fe-46f6-89ee-bc3c36e6cba6-kube-api-access-vdbp2\") pod \"klusterlet-addon-workmgr-7474c64b66-nmmvh\" (UID: \"44e90de9-f4fe-46f6-89ee-bc3c36e6cba6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7474c64b66-nmmvh"
Sep 17 17:35:13.975592 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:13.975577    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52zkw\" (UniqueName: \"kubernetes.io/projected/f2ee4712-52c9-434d-b7f3-247871387e09-kube-api-access-52zkw\") pod \"cluster-proxy-proxy-agent-57d467ccf7-rw9lz\" (UID: \"f2ee4712-52c9-434d-b7f3-247871387e09\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57d467ccf7-rw9lz"
Sep 17 17:35:14.015020 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:14.014997    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7474c64b66-nmmvh"
Sep 17 17:35:14.034368 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:14.034349    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cc6c86fb4-4nnvr"
Sep 17 17:35:14.057593 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:14.057568    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57d467ccf7-rw9lz"
Sep 17 17:35:14.319946 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:14.319917    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7474c64b66-nmmvh"]
Sep 17 17:35:14.322507 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:35:14.322461    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44e90de9_f4fe_46f6_89ee_bc3c36e6cba6.slice/crio-ae0e50a7cc58b7f852a343ff7759a46fd7b78c00ffc6ba66ce865c9838031f05 WatchSource:0}: Error finding container ae0e50a7cc58b7f852a343ff7759a46fd7b78c00ffc6ba66ce865c9838031f05: Status 404 returned error can't find the container with id ae0e50a7cc58b7f852a343ff7759a46fd7b78c00ffc6ba66ce865c9838031f05
Sep 17 17:35:14.353421 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:14.353396    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cc6c86fb4-4nnvr"]
Sep 17 17:35:14.355155 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:35:14.355132    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66e734c2_24ae_4d10_a666_e190f555d036.slice/crio-61ef9732a9ce92bd4494331314feaf9701d81c841199d26348682a028eed328f WatchSource:0}: Error finding container 61ef9732a9ce92bd4494331314feaf9701d81c841199d26348682a028eed328f: Status 404 returned error can't find the container with id 61ef9732a9ce92bd4494331314feaf9701d81c841199d26348682a028eed328f
Sep 17 17:35:14.379572 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:14.379445    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57d467ccf7-rw9lz"]
Sep 17 17:35:14.381164 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:35:14.381143    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2ee4712_52c9_434d_b7f3_247871387e09.slice/crio-17717ecbefb54a76cd9fbfefe3130357b08de275e2e005c9e9565d218464c5e1 WatchSource:0}: Error finding container 17717ecbefb54a76cd9fbfefe3130357b08de275e2e005c9e9565d218464c5e1: Status 404 returned error can't find the container with id 17717ecbefb54a76cd9fbfefe3130357b08de275e2e005c9e9565d218464c5e1
Sep 17 17:35:15.308441 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:15.308354    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57d467ccf7-rw9lz" event={"ID":"f2ee4712-52c9-434d-b7f3-247871387e09","Type":"ContainerStarted","Data":"17717ecbefb54a76cd9fbfefe3130357b08de275e2e005c9e9565d218464c5e1"}
Sep 17 17:35:15.310329 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:15.310277    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cc6c86fb4-4nnvr" event={"ID":"66e734c2-24ae-4d10-a666-e190f555d036","Type":"ContainerStarted","Data":"61ef9732a9ce92bd4494331314feaf9701d81c841199d26348682a028eed328f"}
Sep 17 17:35:15.313029 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:15.312959    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7474c64b66-nmmvh" event={"ID":"44e90de9-f4fe-46f6-89ee-bc3c36e6cba6","Type":"ContainerStarted","Data":"ae0e50a7cc58b7f852a343ff7759a46fd7b78c00ffc6ba66ce865c9838031f05"}
Sep 17 17:35:16.124713 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:16.124683    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:16.225354 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:16.225201    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:21.156849 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:21.156803    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:21.252537 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:21.252507    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:21.331223 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:21.331195    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7474c64b66-nmmvh" event={"ID":"44e90de9-f4fe-46f6-89ee-bc3c36e6cba6","Type":"ContainerStarted","Data":"b97043ce64813be5f0c3ee46479802e9b355ca491424c4fbe00a405f0e0da33a"}
Sep 17 17:35:21.331407 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:21.331377    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7474c64b66-nmmvh"
Sep 17 17:35:21.333164 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:21.333146    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7474c64b66-nmmvh"
Sep 17 17:35:21.333612 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:21.333583    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57d467ccf7-rw9lz" event={"ID":"f2ee4712-52c9-434d-b7f3-247871387e09","Type":"ContainerStarted","Data":"3a487ce198a0c9886b3db83fbe6a987b0855f9eaa12abdb2b1f4b2659d103b64"}
Sep 17 17:35:21.333682 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:21.333615    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57d467ccf7-rw9lz" event={"ID":"f2ee4712-52c9-434d-b7f3-247871387e09","Type":"ContainerStarted","Data":"c7d8689ced8b146c8a964e5fc067d1973db045abd719c5b6fc7caa8128997b92"}
Sep 17 17:35:21.333682 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:21.333633    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57d467ccf7-rw9lz" event={"ID":"f2ee4712-52c9-434d-b7f3-247871387e09","Type":"ContainerStarted","Data":"ae178e3e880d03df9116031b18609e53968f86b5367104408588de4d60c1aaf2"}
Sep 17 17:35:21.335003 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:21.334982    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cc6c86fb4-4nnvr" event={"ID":"66e734c2-24ae-4d10-a666-e190f555d036","Type":"ContainerStarted","Data":"7471931714fb1ff841e7fac7ef578058e8c4df00424467171890f9a1ad4665b1"}
Sep 17 17:35:21.355400 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:21.355356    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7474c64b66-nmmvh" podStartSLOduration=1.76638272 podStartE2EDuration="8.355344366s" totalImagesPullingTime="6.588961646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:35:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:21.354684245 +0000 UTC m=+57.768474683" watchObservedRunningTime="2026-09-17 17:35:21.355344366 +0000 UTC m=+57.769134846"
Sep 17 17:35:21.378742 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:21.378707    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cc6c86fb4-4nnvr" podStartSLOduration=1.837273471 podStartE2EDuration="8.378695924s" totalImagesPullingTime="6.541422453s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:35:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:21.378575983 +0000 UTC m=+57.792366420" watchObservedRunningTime="2026-09-17 17:35:21.378695924 +0000 UTC m=+57.792486361"
Sep 17 17:35:21.428361 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:21.428283    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57d467ccf7-rw9lz" podStartSLOduration=1.912536206 podStartE2EDuration="8.428271698s" totalImagesPullingTime="6.515735492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:35:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:21.427874989 +0000 UTC m=+57.841665426" watchObservedRunningTime="2026-09-17 17:35:21.428271698 +0000 UTC m=+57.842062134"
Sep 17 17:35:22.295003 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:22.294967    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:22.384054 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:22.384021    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:24.247693 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:24.247664    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rrf79"
Sep 17 17:35:24.418461 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:24.418429    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:24.484506 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:24.484468    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:27.035155 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:27.035128    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:27.117790 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:27.117759    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:29.379040 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:29.379004    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3973eda4-9544-4956-980a-4c11ba7700c1-metrics-tls\") pod \"dns-default-t7fvf\" (UID: \"3973eda4-9544-4956-980a-4c11ba7700c1\") " pod="openshift-dns/dns-default-t7fvf"
Sep 17 17:35:29.379402 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:35:29.379105    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:35:29.379402 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:35:29.379160    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3973eda4-9544-4956-980a-4c11ba7700c1-metrics-tls podName:3973eda4-9544-4956-980a-4c11ba7700c1 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:01.379146682 +0000 UTC m=+97.792937098 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3973eda4-9544-4956-980a-4c11ba7700c1-metrics-tls") pod "dns-default-t7fvf" (UID: "3973eda4-9544-4956-980a-4c11ba7700c1") : secret "dns-default-metrics-tls" not found
Sep 17 17:35:29.781495 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:29.781467    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39af04b6-713b-406b-952d-7b2ef2b2b952-metrics-certs\") pod \"network-metrics-daemon-p7vh6\" (UID: \"39af04b6-713b-406b-952d-7b2ef2b2b952\") " pod="openshift-multus/network-metrics-daemon-p7vh6"
Sep 17 17:35:29.784557 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:29.784540    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 17:35:29.792362 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:35:29.792346    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 17:35:29.792404 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:35:29.792396    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39af04b6-713b-406b-952d-7b2ef2b2b952-metrics-certs podName:39af04b6-713b-406b-952d-7b2ef2b2b952 nodeName:}" failed. No retries permitted until 2026-09-17 17:36:33.792382117 +0000 UTC m=+130.206172533 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/39af04b6-713b-406b-952d-7b2ef2b2b952-metrics-certs") pod "network-metrics-daemon-p7vh6" (UID: "39af04b6-713b-406b-952d-7b2ef2b2b952") : secret "metrics-daemon-secret" not found
Sep 17 17:35:29.882053 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:29.882020    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rlql\" (UniqueName: \"kubernetes.io/projected/7bcabfd1-e72e-4866-9190-6bcce1f7ce64-kube-api-access-4rlql\") pod \"network-check-target-z8nx5\" (UID: \"7bcabfd1-e72e-4866-9190-6bcce1f7ce64\") " pod="openshift-network-diagnostics/network-check-target-z8nx5"
Sep 17 17:35:29.882053 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:29.882057    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dbb6518b-5a9b-4a33-89a1-71c9e4db70a5-original-pull-secret\") pod \"global-pull-secret-syncer-7spmx\" (UID: \"dbb6518b-5a9b-4a33-89a1-71c9e4db70a5\") " pod="kube-system/global-pull-secret-syncer-7spmx"
Sep 17 17:35:29.886924 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:29.886904    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 17:35:29.887281 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:29.887267    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 17:35:29.894189 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:29.894175    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dbb6518b-5a9b-4a33-89a1-71c9e4db70a5-original-pull-secret\") pod \"global-pull-secret-syncer-7spmx\" (UID: \"dbb6518b-5a9b-4a33-89a1-71c9e4db70a5\") " pod="kube-system/global-pull-secret-syncer-7spmx"
Sep 17 17:35:29.894637 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:29.894623    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 17:35:29.904356 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:29.904336    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rlql\" (UniqueName: \"kubernetes.io/projected/7bcabfd1-e72e-4866-9190-6bcce1f7ce64-kube-api-access-4rlql\") pod \"network-check-target-z8nx5\" (UID: \"7bcabfd1-e72e-4866-9190-6bcce1f7ce64\") " pod="openshift-network-diagnostics/network-check-target-z8nx5"
Sep 17 17:35:30.012691 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:30.012670    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7spmx"
Sep 17 17:35:30.027456 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:30.027418    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2vz4h\""
Sep 17 17:35:30.034845 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:30.034803    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z8nx5"
Sep 17 17:35:30.294512 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:30.294432    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7spmx"]
Sep 17 17:35:30.297299 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:35:30.297280    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbb6518b_5a9b_4a33_89a1_71c9e4db70a5.slice/crio-c9684e44f8c86bd4cbed09f2e60af508424bb5a695e660ac245adbb4ce868d1f WatchSource:0}: Error finding container c9684e44f8c86bd4cbed09f2e60af508424bb5a695e660ac245adbb4ce868d1f: Status 404 returned error can't find the container with id c9684e44f8c86bd4cbed09f2e60af508424bb5a695e660ac245adbb4ce868d1f
Sep 17 17:35:30.310527 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:30.310506    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-z8nx5"]
Sep 17 17:35:30.313368 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:35:30.313341    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bcabfd1_e72e_4866_9190_6bcce1f7ce64.slice/crio-a527679e941118e97f64a0e46dd462ba4d50fa4537d3b54667917d3cdaf84451 WatchSource:0}: Error finding container a527679e941118e97f64a0e46dd462ba4d50fa4537d3b54667917d3cdaf84451: Status 404 returned error can't find the container with id a527679e941118e97f64a0e46dd462ba4d50fa4537d3b54667917d3cdaf84451
Sep 17 17:35:30.361535 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:30.361505    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7spmx" event={"ID":"dbb6518b-5a9b-4a33-89a1-71c9e4db70a5","Type":"ContainerStarted","Data":"c9684e44f8c86bd4cbed09f2e60af508424bb5a695e660ac245adbb4ce868d1f"}
Sep 17 17:35:30.362510 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:30.362468    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-z8nx5" event={"ID":"7bcabfd1-e72e-4866-9190-6bcce1f7ce64","Type":"ContainerStarted","Data":"a527679e941118e97f64a0e46dd462ba4d50fa4537d3b54667917d3cdaf84451"}
Sep 17 17:35:35.381318 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:35.381277    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7spmx" event={"ID":"dbb6518b-5a9b-4a33-89a1-71c9e4db70a5","Type":"ContainerStarted","Data":"8ce8f1ebba3b7d74a8fa8db1c968fa62ac1fb4ab53d0adc267f6a7a7581ba0e4"}
Sep 17 17:35:35.382647 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:35.382621    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-z8nx5" event={"ID":"7bcabfd1-e72e-4866-9190-6bcce1f7ce64","Type":"ContainerStarted","Data":"3b715521e62af8d8b0ff61f4b0e8c1e782d660a63a021bd5cdf416cbdc4d5eca"}
Sep 17 17:35:35.382756 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:35.382671    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-z8nx5"
Sep 17 17:35:35.405018 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:35.404890    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-7spmx" podStartSLOduration=67.213892221 podStartE2EDuration="1m11.404874396s" totalImagesPullingTime="4.190982175s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:34:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:35.404515634 +0000 UTC m=+71.818306073" watchObservedRunningTime="2026-09-17 17:35:35.404874396 +0000 UTC m=+71.818664833"
Sep 17 17:35:35.429697 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:35.429651    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-z8nx5" podStartSLOduration=67.258706919 podStartE2EDuration="1m11.429636809s" totalImagesPullingTime="4.17092989s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:35:35.429319585 +0000 UTC m=+71.843110022" watchObservedRunningTime="2026-09-17 17:35:35.429636809 +0000 UTC m=+71.843427247"
Sep 17 17:35:43.171826 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:43.171789    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:43.263385 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:43.263349    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:35:43.347389 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:35:43.347353    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:01.399082 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:36:01.399048    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3973eda4-9544-4956-980a-4c11ba7700c1-metrics-tls\") pod \"dns-default-t7fvf\" (UID: \"3973eda4-9544-4956-980a-4c11ba7700c1\") " pod="openshift-dns/dns-default-t7fvf"
Sep 17 17:36:01.399470 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:36:01.399158    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:36:01.399470 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:36:01.399212    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3973eda4-9544-4956-980a-4c11ba7700c1-metrics-tls podName:3973eda4-9544-4956-980a-4c11ba7700c1 nodeName:}" failed. No retries permitted until 2026-09-17 17:37:05.399197341 +0000 UTC m=+161.812987760 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3973eda4-9544-4956-980a-4c11ba7700c1-metrics-tls") pod "dns-default-t7fvf" (UID: "3973eda4-9544-4956-980a-4c11ba7700c1") : secret "dns-default-metrics-tls" not found
Sep 17 17:36:06.388160 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:36:06.388123    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-z8nx5"
Sep 17 17:36:21.772913 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:36:21.772876    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:23.698035 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:36:23.697996    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:33.819296 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:36:33.819262    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39af04b6-713b-406b-952d-7b2ef2b2b952-metrics-certs\") pod \"network-metrics-daemon-p7vh6\" (UID: \"39af04b6-713b-406b-952d-7b2ef2b2b952\") " pod="openshift-multus/network-metrics-daemon-p7vh6"
Sep 17 17:36:33.819767 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:36:33.819385    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 17:36:33.819767 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:36:33.819465    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39af04b6-713b-406b-952d-7b2ef2b2b952-metrics-certs podName:39af04b6-713b-406b-952d-7b2ef2b2b952 nodeName:}" failed. No retries permitted until 2026-09-17 17:38:35.819424512 +0000 UTC m=+252.233214941 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/39af04b6-713b-406b-952d-7b2ef2b2b952-metrics-certs") pod "network-metrics-daemon-p7vh6" (UID: "39af04b6-713b-406b-952d-7b2ef2b2b952") : secret "metrics-daemon-secret" not found
Sep 17 17:36:56.317755 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:36:56.317720    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:56.417274 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:36:56.417243    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:58.466362 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:36:58.466331    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:58.551784 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:36:58.551752    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:36:59.940745 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:36:59.940705    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:00.006239 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:00.006206    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:00.070828 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:00.070799    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:00.493281 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:37:00.493246    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-t7fvf" podUID="3973eda4-9544-4956-980a-4c11ba7700c1"
Sep 17 17:37:00.624692 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:00.624653    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-t7fvf"
Sep 17 17:37:00.955203 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:00.955122    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hjpz9"]
Sep 17 17:37:00.958274 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:00.958259    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hjpz9"
Sep 17 17:37:00.960781 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:00.960754    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 17:37:00.960942 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:00.960922    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-d9vw5\""
Sep 17 17:37:00.961013 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:00.960968    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 17:37:00.961930 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:00.961908    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 17:37:00.962385 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:00.962370    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 17:37:00.969417 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:00.969398    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hjpz9"]
Sep 17 17:37:01.006433 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:01.006401    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7c0b81ec-0283-421e-ae34-032edea2eeb8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hjpz9\" (UID: \"7c0b81ec-0283-421e-ae34-032edea2eeb8\") " pod="openshift-insights/insights-runtime-extractor-hjpz9"
Sep 17 17:37:01.006601 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:01.006456    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7c0b81ec-0283-421e-ae34-032edea2eeb8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hjpz9\" (UID: \"7c0b81ec-0283-421e-ae34-032edea2eeb8\") " pod="openshift-insights/insights-runtime-extractor-hjpz9"
Sep 17 17:37:01.006601 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:01.006517    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctjgz\" (UniqueName: \"kubernetes.io/projected/7c0b81ec-0283-421e-ae34-032edea2eeb8-kube-api-access-ctjgz\") pod \"insights-runtime-extractor-hjpz9\" (UID: \"7c0b81ec-0283-421e-ae34-032edea2eeb8\") " pod="openshift-insights/insights-runtime-extractor-hjpz9"
Sep 17 17:37:01.006601 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:01.006573    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7c0b81ec-0283-421e-ae34-032edea2eeb8-data-volume\") pod \"insights-runtime-extractor-hjpz9\" (UID: \"7c0b81ec-0283-421e-ae34-032edea2eeb8\") " pod="openshift-insights/insights-runtime-extractor-hjpz9"
Sep 17 17:37:01.006601 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:01.006599    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7c0b81ec-0283-421e-ae34-032edea2eeb8-crio-socket\") pod \"insights-runtime-extractor-hjpz9\" (UID: \"7c0b81ec-0283-421e-ae34-032edea2eeb8\") " pod="openshift-insights/insights-runtime-extractor-hjpz9"
Sep 17 17:37:01.050437 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:01.050407    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:01.107355 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:01.107327    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctjgz\" (UniqueName: \"kubernetes.io/projected/7c0b81ec-0283-421e-ae34-032edea2eeb8-kube-api-access-ctjgz\") pod \"insights-runtime-extractor-hjpz9\" (UID: \"7c0b81ec-0283-421e-ae34-032edea2eeb8\") " pod="openshift-insights/insights-runtime-extractor-hjpz9"
Sep 17 17:37:01.107355 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:01.107355    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7c0b81ec-0283-421e-ae34-032edea2eeb8-data-volume\") pod \"insights-runtime-extractor-hjpz9\" (UID: \"7c0b81ec-0283-421e-ae34-032edea2eeb8\") " pod="openshift-insights/insights-runtime-extractor-hjpz9"
Sep 17 17:37:01.107591 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:01.107375    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7c0b81ec-0283-421e-ae34-032edea2eeb8-crio-socket\") pod \"insights-runtime-extractor-hjpz9\" (UID: \"7c0b81ec-0283-421e-ae34-032edea2eeb8\") " pod="openshift-insights/insights-runtime-extractor-hjpz9"
Sep 17 17:37:01.107591 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:01.107409    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7c0b81ec-0283-421e-ae34-032edea2eeb8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hjpz9\" (UID: \"7c0b81ec-0283-421e-ae34-032edea2eeb8\") " pod="openshift-insights/insights-runtime-extractor-hjpz9"
Sep 17 17:37:01.107591 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:01.107449    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7c0b81ec-0283-421e-ae34-032edea2eeb8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hjpz9\" (UID: \"7c0b81ec-0283-421e-ae34-032edea2eeb8\") " pod="openshift-insights/insights-runtime-extractor-hjpz9"
Sep 17 17:37:01.107591 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:01.107479    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7c0b81ec-0283-421e-ae34-032edea2eeb8-crio-socket\") pod \"insights-runtime-extractor-hjpz9\" (UID: \"7c0b81ec-0283-421e-ae34-032edea2eeb8\") " pod="openshift-insights/insights-runtime-extractor-hjpz9"
Sep 17 17:37:01.107591 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:37:01.107558    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:37:01.107591 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:37:01.107578    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:37:01.107826 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:37:01.107633    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7c0b81ec-0283-421e-ae34-032edea2eeb8-kube-rbac-proxy-cm podName:7c0b81ec-0283-421e-ae34-032edea2eeb8 nodeName:}" failed. No retries permitted until 2026-09-17 17:37:01.607614121 +0000 UTC m=+158.021404554 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7c0b81ec-0283-421e-ae34-032edea2eeb8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hjpz9" (UID: "7c0b81ec-0283-421e-ae34-032edea2eeb8") : configmap "kube-rbac-proxy" not found
Sep 17 17:37:01.107826 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:37:01.107649    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c0b81ec-0283-421e-ae34-032edea2eeb8-insights-runtime-extractor-tls podName:7c0b81ec-0283-421e-ae34-032edea2eeb8 nodeName:}" failed. No retries permitted until 2026-09-17 17:37:01.607640487 +0000 UTC m=+158.021430903 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7c0b81ec-0283-421e-ae34-032edea2eeb8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hjpz9" (UID: "7c0b81ec-0283-421e-ae34-032edea2eeb8") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:37:01.107826 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:01.107665    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7c0b81ec-0283-421e-ae34-032edea2eeb8-data-volume\") pod \"insights-runtime-extractor-hjpz9\" (UID: \"7c0b81ec-0283-421e-ae34-032edea2eeb8\") " pod="openshift-insights/insights-runtime-extractor-hjpz9"
Sep 17 17:37:01.120933 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:01.120912    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctjgz\" (UniqueName: \"kubernetes.io/projected/7c0b81ec-0283-421e-ae34-032edea2eeb8-kube-api-access-ctjgz\") pod \"insights-runtime-extractor-hjpz9\" (UID: \"7c0b81ec-0283-421e-ae34-032edea2eeb8\") " pod="openshift-insights/insights-runtime-extractor-hjpz9"
Sep 17 17:37:01.218685 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:01.218612    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:01.371069 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:01.371040    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:01.441306 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:01.441273    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:01.611388 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:01.611358    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7c0b81ec-0283-421e-ae34-032edea2eeb8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hjpz9\" (UID: \"7c0b81ec-0283-421e-ae34-032edea2eeb8\") " pod="openshift-insights/insights-runtime-extractor-hjpz9"
Sep 17 17:37:01.611552 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:01.611400    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7c0b81ec-0283-421e-ae34-032edea2eeb8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hjpz9\" (UID: \"7c0b81ec-0283-421e-ae34-032edea2eeb8\") " pod="openshift-insights/insights-runtime-extractor-hjpz9"
Sep 17 17:37:01.611552 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:37:01.611479    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:37:01.611552 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:37:01.611524    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:37:01.611552 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:37:01.611552    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7c0b81ec-0283-421e-ae34-032edea2eeb8-kube-rbac-proxy-cm podName:7c0b81ec-0283-421e-ae34-032edea2eeb8 nodeName:}" failed. No retries permitted until 2026-09-17 17:37:02.611538599 +0000 UTC m=+159.025329015 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7c0b81ec-0283-421e-ae34-032edea2eeb8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hjpz9" (UID: "7c0b81ec-0283-421e-ae34-032edea2eeb8") : configmap "kube-rbac-proxy" not found
Sep 17 17:37:01.611690 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:37:01.611579    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c0b81ec-0283-421e-ae34-032edea2eeb8-insights-runtime-extractor-tls podName:7c0b81ec-0283-421e-ae34-032edea2eeb8 nodeName:}" failed. No retries permitted until 2026-09-17 17:37:02.61156499 +0000 UTC m=+159.025355406 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7c0b81ec-0283-421e-ae34-032edea2eeb8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hjpz9" (UID: "7c0b81ec-0283-421e-ae34-032edea2eeb8") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:37:01.649438 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:01.649410    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:01.752640 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:01.752615    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:02.118906 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:37:02.118872    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-p7vh6" podUID="39af04b6-713b-406b-952d-7b2ef2b2b952"
Sep 17 17:37:02.374257 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:02.374169    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:02.478341 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:02.478306    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:02.619830 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:02.619796    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7c0b81ec-0283-421e-ae34-032edea2eeb8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hjpz9\" (UID: \"7c0b81ec-0283-421e-ae34-032edea2eeb8\") " pod="openshift-insights/insights-runtime-extractor-hjpz9"
Sep 17 17:37:02.620010 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:02.619841    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7c0b81ec-0283-421e-ae34-032edea2eeb8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hjpz9\" (UID: \"7c0b81ec-0283-421e-ae34-032edea2eeb8\") " pod="openshift-insights/insights-runtime-extractor-hjpz9"
Sep 17 17:37:02.620010 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:37:02.619947    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:37:02.620010 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:37:02.619970    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:37:02.620169 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:37:02.620013    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7c0b81ec-0283-421e-ae34-032edea2eeb8-kube-rbac-proxy-cm podName:7c0b81ec-0283-421e-ae34-032edea2eeb8 nodeName:}" failed. No retries permitted until 2026-09-17 17:37:04.619996518 +0000 UTC m=+161.033786947 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7c0b81ec-0283-421e-ae34-032edea2eeb8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hjpz9" (UID: "7c0b81ec-0283-421e-ae34-032edea2eeb8") : configmap "kube-rbac-proxy" not found
Sep 17 17:37:02.620169 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:37:02.620029    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c0b81ec-0283-421e-ae34-032edea2eeb8-insights-runtime-extractor-tls podName:7c0b81ec-0283-421e-ae34-032edea2eeb8 nodeName:}" failed. No retries permitted until 2026-09-17 17:37:04.620022243 +0000 UTC m=+161.033812659 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7c0b81ec-0283-421e-ae34-032edea2eeb8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hjpz9" (UID: "7c0b81ec-0283-421e-ae34-032edea2eeb8") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:37:03.354888 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:03.354856    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:03.472889 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:03.472860    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:03.518928 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:03.518906    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kgbmb_6c27507b-19a0-4c12-a379-10474f260f84/dns-node-resolver/0.log"
Sep 17 17:37:04.633878 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:04.633842    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7c0b81ec-0283-421e-ae34-032edea2eeb8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hjpz9\" (UID: \"7c0b81ec-0283-421e-ae34-032edea2eeb8\") " pod="openshift-insights/insights-runtime-extractor-hjpz9"
Sep 17 17:37:04.634311 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:04.633924    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7c0b81ec-0283-421e-ae34-032edea2eeb8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hjpz9\" (UID: \"7c0b81ec-0283-421e-ae34-032edea2eeb8\") " pod="openshift-insights/insights-runtime-extractor-hjpz9"
Sep 17 17:37:04.634311 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:37:04.633959    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:37:04.634311 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:37:04.634028    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:37:04.634311 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:37:04.634030    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7c0b81ec-0283-421e-ae34-032edea2eeb8-kube-rbac-proxy-cm podName:7c0b81ec-0283-421e-ae34-032edea2eeb8 nodeName:}" failed. No retries permitted until 2026-09-17 17:37:08.634009621 +0000 UTC m=+165.047800053 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7c0b81ec-0283-421e-ae34-032edea2eeb8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hjpz9" (UID: "7c0b81ec-0283-421e-ae34-032edea2eeb8") : configmap "kube-rbac-proxy" not found
Sep 17 17:37:04.634311 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:37:04.634085    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c0b81ec-0283-421e-ae34-032edea2eeb8-insights-runtime-extractor-tls podName:7c0b81ec-0283-421e-ae34-032edea2eeb8 nodeName:}" failed. No retries permitted until 2026-09-17 17:37:08.63407237 +0000 UTC m=+165.047862786 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7c0b81ec-0283-421e-ae34-032edea2eeb8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hjpz9" (UID: "7c0b81ec-0283-421e-ae34-032edea2eeb8") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:37:04.668943 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:04.668918    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:04.917782 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:04.917717    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mglss_e8bda048-e7cf-4118-82c4-bd1d987e4566/node-ca/0.log"
Sep 17 17:37:05.439874 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:05.439841    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3973eda4-9544-4956-980a-4c11ba7700c1-metrics-tls\") pod \"dns-default-t7fvf\" (UID: \"3973eda4-9544-4956-980a-4c11ba7700c1\") " pod="openshift-dns/dns-default-t7fvf"
Sep 17 17:37:05.440046 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:37:05.439987    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 17:37:05.440090 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:37:05.440056    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3973eda4-9544-4956-980a-4c11ba7700c1-metrics-tls podName:3973eda4-9544-4956-980a-4c11ba7700c1 nodeName:}" failed. No retries permitted until 2026-09-17 17:39:07.440041511 +0000 UTC m=+283.853831927 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3973eda4-9544-4956-980a-4c11ba7700c1-metrics-tls") pod "dns-default-t7fvf" (UID: "3973eda4-9544-4956-980a-4c11ba7700c1") : secret "dns-default-metrics-tls" not found
Sep 17 17:37:05.716334 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:05.716258    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jvtw5_98e53603-643c-4852-b805-5e07c9016190/serve-healthcheck-canary/0.log"
Sep 17 17:37:07.813598 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:07.813566    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:07.921859 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:07.921830    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:08.017042 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:08.017009    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:08.169184 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:08.169103    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:08.295993 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:08.295961    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:08.400444 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:08.400417    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:08.523052 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:08.523024    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:08.660682 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:08.660658    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7c0b81ec-0283-421e-ae34-032edea2eeb8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hjpz9\" (UID: \"7c0b81ec-0283-421e-ae34-032edea2eeb8\") " pod="openshift-insights/insights-runtime-extractor-hjpz9"
Sep 17 17:37:08.660847 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:08.660723    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7c0b81ec-0283-421e-ae34-032edea2eeb8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hjpz9\" (UID: \"7c0b81ec-0283-421e-ae34-032edea2eeb8\") " pod="openshift-insights/insights-runtime-extractor-hjpz9"
Sep 17 17:37:08.660847 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:37:08.660787    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 17:37:08.660847 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:37:08.660814    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:37:08.660947 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:37:08.660851    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7c0b81ec-0283-421e-ae34-032edea2eeb8-kube-rbac-proxy-cm podName:7c0b81ec-0283-421e-ae34-032edea2eeb8 nodeName:}" failed. No retries permitted until 2026-09-17 17:37:16.660832342 +0000 UTC m=+173.074622777 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7c0b81ec-0283-421e-ae34-032edea2eeb8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hjpz9" (UID: "7c0b81ec-0283-421e-ae34-032edea2eeb8") : configmap "kube-rbac-proxy" not found
Sep 17 17:37:08.660947 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:37:08.660865    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c0b81ec-0283-421e-ae34-032edea2eeb8-insights-runtime-extractor-tls podName:7c0b81ec-0283-421e-ae34-032edea2eeb8 nodeName:}" failed. No retries permitted until 2026-09-17 17:37:16.660859344 +0000 UTC m=+173.074649760 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7c0b81ec-0283-421e-ae34-032edea2eeb8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hjpz9" (UID: "7c0b81ec-0283-421e-ae34-032edea2eeb8") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:37:09.704060 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:09.704028    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:09.978383 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:09.978308    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:10.128965 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:10.128934    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:10.332340 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:10.332304    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:10.533435 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:10.533400    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:10.692181 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:10.692102    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:11.533455 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:11.533420    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:11.625694 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:11.625667    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:11.733218 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:11.733191    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:11.967797 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:11.967665    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:12.332368 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:12.332338    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:12.961322 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:12.961296    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:13.255096 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:13.255064    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:13.358730 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:13.358699    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:13.540302 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:13.540217    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:13.901877 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:13.901796    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:14.101032 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:14.101000    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p7vh6"
Sep 17 17:37:14.381456 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:14.381423    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:14.514721 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:14.514694    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:14.903930 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:14.903893    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:15.303266 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:15.303238    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:15.519999 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:15.519961    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:15.906017 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:15.905982    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:16.706908 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:16.706870    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:16.719244 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:16.719218    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7c0b81ec-0283-421e-ae34-032edea2eeb8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hjpz9\" (UID: \"7c0b81ec-0283-421e-ae34-032edea2eeb8\") " pod="openshift-insights/insights-runtime-extractor-hjpz9"
Sep 17 17:37:16.719344 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:16.719262    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7c0b81ec-0283-421e-ae34-032edea2eeb8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hjpz9\" (UID: \"7c0b81ec-0283-421e-ae34-032edea2eeb8\") " pod="openshift-insights/insights-runtime-extractor-hjpz9"
Sep 17 17:37:16.719381 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:37:16.719371    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 17:37:16.719461 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:37:16.719450    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c0b81ec-0283-421e-ae34-032edea2eeb8-insights-runtime-extractor-tls podName:7c0b81ec-0283-421e-ae34-032edea2eeb8 nodeName:}" failed. No retries permitted until 2026-09-17 17:37:32.719431284 +0000 UTC m=+189.133221705 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7c0b81ec-0283-421e-ae34-032edea2eeb8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hjpz9" (UID: "7c0b81ec-0283-421e-ae34-032edea2eeb8") : secret "insights-runtime-extractor-tls" not found
Sep 17 17:37:16.719898 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:16.719881    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7c0b81ec-0283-421e-ae34-032edea2eeb8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hjpz9\" (UID: \"7c0b81ec-0283-421e-ae34-032edea2eeb8\") " pod="openshift-insights/insights-runtime-extractor-hjpz9"
Sep 17 17:37:16.925186 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:16.925109    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:17.023549 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:17.023516    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:17.344983 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:17.344902    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:17.522854 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:17.522827    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:17.926725 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:17.926690    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:18.330210 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:18.330172    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:18.565055 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:18.565026    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:21.331739 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:21.331650    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7474c64b66-nmmvh" podUID="44e90de9-f4fe-46f6-89ee-bc3c36e6cba6" containerName="acm-agent" probeResult="failure" output="Get \"http://10.133.0.8:8000/readyz\": dial tcp 10.133.0.8:8000: connect: connection refused"
Sep 17 17:37:21.685246 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:21.685165    2585 generic.go:356] "Generic (PLEG): container finished" podID="44e90de9-f4fe-46f6-89ee-bc3c36e6cba6" containerID="b97043ce64813be5f0c3ee46479802e9b355ca491424c4fbe00a405f0e0da33a" exitCode=1
Sep 17 17:37:21.685246 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:21.685236    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7474c64b66-nmmvh" event={"ID":"44e90de9-f4fe-46f6-89ee-bc3c36e6cba6","Type":"ContainerDied","Data":"b97043ce64813be5f0c3ee46479802e9b355ca491424c4fbe00a405f0e0da33a"}
Sep 17 17:37:21.685585 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:21.685569    2585 scope.go:117] "RemoveContainer" containerID="b97043ce64813be5f0c3ee46479802e9b355ca491424c4fbe00a405f0e0da33a"
Sep 17 17:37:21.686723 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:21.686703    2585 generic.go:356] "Generic (PLEG): container finished" podID="66e734c2-24ae-4d10-a666-e190f555d036" containerID="7471931714fb1ff841e7fac7ef578058e8c4df00424467171890f9a1ad4665b1" exitCode=255
Sep 17 17:37:21.686820 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:21.686741    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cc6c86fb4-4nnvr" event={"ID":"66e734c2-24ae-4d10-a666-e190f555d036","Type":"ContainerDied","Data":"7471931714fb1ff841e7fac7ef578058e8c4df00424467171890f9a1ad4665b1"}
Sep 17 17:37:21.687009 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:21.686994    2585 scope.go:117] "RemoveContainer" containerID="7471931714fb1ff841e7fac7ef578058e8c4df00424467171890f9a1ad4665b1"
Sep 17 17:37:22.691333 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:22.691298    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-cc6c86fb4-4nnvr" event={"ID":"66e734c2-24ae-4d10-a666-e190f555d036","Type":"ContainerStarted","Data":"e421ddf4bab661bff5de198043789fc0e7cbc8eec8416e8f4f7ce227ea4b2818"}
Sep 17 17:37:22.692962 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:22.692941    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7474c64b66-nmmvh" event={"ID":"44e90de9-f4fe-46f6-89ee-bc3c36e6cba6","Type":"ContainerStarted","Data":"5092bb27f539132aef0fcd1965dc472f9b7eabffa24404588a0954b7e316f932"}
Sep 17 17:37:22.693192 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:22.693176    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7474c64b66-nmmvh"
Sep 17 17:37:22.693787 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:22.693771    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7474c64b66-nmmvh"
Sep 17 17:37:26.647847 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:26.647816    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:26.756565 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:26.756534    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:29.769678 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:29.769643    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:29.894886 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:29.894845    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:31.125713 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:31.125680    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:31.315601 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:31.315565    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:31.635862 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:31.635833    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:32.726566 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:32.726537    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7c0b81ec-0283-421e-ae34-032edea2eeb8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hjpz9\" (UID: \"7c0b81ec-0283-421e-ae34-032edea2eeb8\") " pod="openshift-insights/insights-runtime-extractor-hjpz9"
Sep 17 17:37:32.728795 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:32.728777    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7c0b81ec-0283-421e-ae34-032edea2eeb8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hjpz9\" (UID: \"7c0b81ec-0283-421e-ae34-032edea2eeb8\") " pod="openshift-insights/insights-runtime-extractor-hjpz9"
Sep 17 17:37:32.773059 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:32.773038    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-d9vw5\""
Sep 17 17:37:32.777445 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:32.777429    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hjpz9"
Sep 17 17:37:33.049974 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:33.049942    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hjpz9"]
Sep 17 17:37:33.054178 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:37:33.054150    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c0b81ec_0283_421e_ae34_032edea2eeb8.slice/crio-b2dd57afe0df29c153afc0d39e943fdcc5cd9dfb326496fa8a31ba6b21a8dcff WatchSource:0}: Error finding container b2dd57afe0df29c153afc0d39e943fdcc5cd9dfb326496fa8a31ba6b21a8dcff: Status 404 returned error can't find the container with id b2dd57afe0df29c153afc0d39e943fdcc5cd9dfb326496fa8a31ba6b21a8dcff
Sep 17 17:37:33.723685 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:33.723659    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hjpz9" event={"ID":"7c0b81ec-0283-421e-ae34-032edea2eeb8","Type":"ContainerStarted","Data":"139a0841ab462907b4da40ee5a4f9b2ef13aa92aa2792212f2439857d02eb1a4"}
Sep 17 17:37:33.723815 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:33.723718    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hjpz9" event={"ID":"7c0b81ec-0283-421e-ae34-032edea2eeb8","Type":"ContainerStarted","Data":"b2dd57afe0df29c153afc0d39e943fdcc5cd9dfb326496fa8a31ba6b21a8dcff"}
Sep 17 17:37:34.732912 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:34.732877    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hjpz9" event={"ID":"7c0b81ec-0283-421e-ae34-032edea2eeb8","Type":"ContainerStarted","Data":"4cc95054c01d7ed75fb812f078b5b2a796bada95b82fdcd63127e89ba093de0c"}
Sep 17 17:37:35.370119 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:35.370088    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:35.475521 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:35.475479    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:35.707017 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:35.706928    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:35.737102 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:35.737072    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hjpz9" event={"ID":"7c0b81ec-0283-421e-ae34-032edea2eeb8","Type":"ContainerStarted","Data":"a861e3ea7c08de4d09bc014271a0e2edf6d6b69a2a508bdb12f22161fea90bcf"}
Sep 17 17:37:38.097401 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:38.097369    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:40.502670 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:40.502640    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:40.725569 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:40.725541    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:40.796097 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:40.796002    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hjpz9" podStartSLOduration=38.690508567 podStartE2EDuration="40.795985896s" totalImagesPullingTime="2.105477329s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:37:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:37:35.758573072 +0000 UTC m=+192.172363512" watchObservedRunningTime="2026-09-17 17:37:40.795985896 +0000 UTC m=+197.209776331"
Sep 17 17:37:40.796559 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:40.796545    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-wp9c7"]
Sep 17 17:37:40.799686 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:40.799670    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wp9c7"
Sep 17 17:37:40.802360 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:40.802340    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-sg5vz\""
Sep 17 17:37:40.803236 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:40.803206    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 17:37:40.803236 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:40.803227    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 17:37:40.803396 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:40.803241    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 17:37:40.803396 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:40.803292    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 17:37:40.803396 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:40.803305    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 17:37:40.803539 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:40.803526    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 17:37:40.829928 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:40.829902    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:40.897062 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:40.897035    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e29f50f9-0bcf-4660-8348-914273f06fc5-metrics-client-ca\") pod \"node-exporter-wp9c7\" (UID: \"e29f50f9-0bcf-4660-8348-914273f06fc5\") " pod="openshift-monitoring/node-exporter-wp9c7"
Sep 17 17:37:40.897183 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:40.897084    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e29f50f9-0bcf-4660-8348-914273f06fc5-node-exporter-wtmp\") pod \"node-exporter-wp9c7\" (UID: \"e29f50f9-0bcf-4660-8348-914273f06fc5\") " pod="openshift-monitoring/node-exporter-wp9c7"
Sep 17 17:37:40.897183 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:40.897103    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e29f50f9-0bcf-4660-8348-914273f06fc5-root\") pod \"node-exporter-wp9c7\" (UID: \"e29f50f9-0bcf-4660-8348-914273f06fc5\") " pod="openshift-monitoring/node-exporter-wp9c7"
Sep 17 17:37:40.897183 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:40.897156    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e29f50f9-0bcf-4660-8348-914273f06fc5-node-exporter-textfile\") pod \"node-exporter-wp9c7\" (UID: \"e29f50f9-0bcf-4660-8348-914273f06fc5\") " pod="openshift-monitoring/node-exporter-wp9c7"
Sep 17 17:37:40.897183 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:40.897170    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e29f50f9-0bcf-4660-8348-914273f06fc5-node-exporter-tls\") pod \"node-exporter-wp9c7\" (UID: \"e29f50f9-0bcf-4660-8348-914273f06fc5\") " pod="openshift-monitoring/node-exporter-wp9c7"
Sep 17 17:37:40.897323 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:40.897187    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e29f50f9-0bcf-4660-8348-914273f06fc5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wp9c7\" (UID: \"e29f50f9-0bcf-4660-8348-914273f06fc5\") " pod="openshift-monitoring/node-exporter-wp9c7"
Sep 17 17:37:40.897323 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:40.897249    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e29f50f9-0bcf-4660-8348-914273f06fc5-sys\") pod \"node-exporter-wp9c7\" (UID: \"e29f50f9-0bcf-4660-8348-914273f06fc5\") " pod="openshift-monitoring/node-exporter-wp9c7"
Sep 17 17:37:40.897323 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:40.897281    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhdst\" (UniqueName: \"kubernetes.io/projected/e29f50f9-0bcf-4660-8348-914273f06fc5-kube-api-access-vhdst\") pod \"node-exporter-wp9c7\" (UID: \"e29f50f9-0bcf-4660-8348-914273f06fc5\") " pod="openshift-monitoring/node-exporter-wp9c7"
Sep 17 17:37:40.897323 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:40.897300    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e29f50f9-0bcf-4660-8348-914273f06fc5-node-exporter-accelerators-collector-config\") pod \"node-exporter-wp9c7\" (UID: \"e29f50f9-0bcf-4660-8348-914273f06fc5\") " pod="openshift-monitoring/node-exporter-wp9c7"
Sep 17 17:37:40.997671 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:40.997636    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e29f50f9-0bcf-4660-8348-914273f06fc5-node-exporter-wtmp\") pod \"node-exporter-wp9c7\" (UID: \"e29f50f9-0bcf-4660-8348-914273f06fc5\") " pod="openshift-monitoring/node-exporter-wp9c7"
Sep 17 17:37:40.997671 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:40.997671    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e29f50f9-0bcf-4660-8348-914273f06fc5-root\") pod \"node-exporter-wp9c7\" (UID: \"e29f50f9-0bcf-4660-8348-914273f06fc5\") " pod="openshift-monitoring/node-exporter-wp9c7"
Sep 17 17:37:40.997859 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:40.997695    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e29f50f9-0bcf-4660-8348-914273f06fc5-node-exporter-textfile\") pod \"node-exporter-wp9c7\" (UID: \"e29f50f9-0bcf-4660-8348-914273f06fc5\") " pod="openshift-monitoring/node-exporter-wp9c7"
Sep 17 17:37:40.997859 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:40.997748    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e29f50f9-0bcf-4660-8348-914273f06fc5-node-exporter-tls\") pod \"node-exporter-wp9c7\" (UID: \"e29f50f9-0bcf-4660-8348-914273f06fc5\") " pod="openshift-monitoring/node-exporter-wp9c7"
Sep 17 17:37:40.997859 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:40.997795    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e29f50f9-0bcf-4660-8348-914273f06fc5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wp9c7\" (UID: \"e29f50f9-0bcf-4660-8348-914273f06fc5\") " pod="openshift-monitoring/node-exporter-wp9c7"
Sep 17 17:37:40.997859 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:40.997753    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e29f50f9-0bcf-4660-8348-914273f06fc5-root\") pod \"node-exporter-wp9c7\" (UID: \"e29f50f9-0bcf-4660-8348-914273f06fc5\") " pod="openshift-monitoring/node-exporter-wp9c7"
Sep 17 17:37:40.997859 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:40.997829    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e29f50f9-0bcf-4660-8348-914273f06fc5-node-exporter-wtmp\") pod \"node-exporter-wp9c7\" (UID: \"e29f50f9-0bcf-4660-8348-914273f06fc5\") " pod="openshift-monitoring/node-exporter-wp9c7"
Sep 17 17:37:40.997859 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:40.997842    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e29f50f9-0bcf-4660-8348-914273f06fc5-sys\") pod \"node-exporter-wp9c7\" (UID: \"e29f50f9-0bcf-4660-8348-914273f06fc5\") " pod="openshift-monitoring/node-exporter-wp9c7"
Sep 17 17:37:40.998038 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:40.997881    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e29f50f9-0bcf-4660-8348-914273f06fc5-sys\") pod \"node-exporter-wp9c7\" (UID: \"e29f50f9-0bcf-4660-8348-914273f06fc5\") " pod="openshift-monitoring/node-exporter-wp9c7"
Sep 17 17:37:40.998038 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:37:40.997837    2585 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 17 17:37:40.998038 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:40.997890    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhdst\" (UniqueName: \"kubernetes.io/projected/e29f50f9-0bcf-4660-8348-914273f06fc5-kube-api-access-vhdst\") pod \"node-exporter-wp9c7\" (UID: \"e29f50f9-0bcf-4660-8348-914273f06fc5\") " pod="openshift-monitoring/node-exporter-wp9c7"
Sep 17 17:37:40.998038 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:37:40.997975    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e29f50f9-0bcf-4660-8348-914273f06fc5-node-exporter-tls podName:e29f50f9-0bcf-4660-8348-914273f06fc5 nodeName:}" failed. No retries permitted until 2026-09-17 17:37:41.497949378 +0000 UTC m=+197.911739809 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/e29f50f9-0bcf-4660-8348-914273f06fc5-node-exporter-tls") pod "node-exporter-wp9c7" (UID: "e29f50f9-0bcf-4660-8348-914273f06fc5") : secret "node-exporter-tls" not found
Sep 17 17:37:40.998038 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:40.997979    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e29f50f9-0bcf-4660-8348-914273f06fc5-node-exporter-textfile\") pod \"node-exporter-wp9c7\" (UID: \"e29f50f9-0bcf-4660-8348-914273f06fc5\") " pod="openshift-monitoring/node-exporter-wp9c7"
Sep 17 17:37:40.998038 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:40.998010    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e29f50f9-0bcf-4660-8348-914273f06fc5-node-exporter-accelerators-collector-config\") pod \"node-exporter-wp9c7\" (UID: \"e29f50f9-0bcf-4660-8348-914273f06fc5\") " pod="openshift-monitoring/node-exporter-wp9c7"
Sep 17 17:37:40.998232 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:40.998064    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e29f50f9-0bcf-4660-8348-914273f06fc5-metrics-client-ca\") pod \"node-exporter-wp9c7\" (UID: \"e29f50f9-0bcf-4660-8348-914273f06fc5\") " pod="openshift-monitoring/node-exporter-wp9c7"
Sep 17 17:37:40.998526 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:40.998486    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e29f50f9-0bcf-4660-8348-914273f06fc5-node-exporter-accelerators-collector-config\") pod \"node-exporter-wp9c7\" (UID: \"e29f50f9-0bcf-4660-8348-914273f06fc5\") " pod="openshift-monitoring/node-exporter-wp9c7"
Sep 17 17:37:40.998526 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:40.998518    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e29f50f9-0bcf-4660-8348-914273f06fc5-metrics-client-ca\") pod \"node-exporter-wp9c7\" (UID: \"e29f50f9-0bcf-4660-8348-914273f06fc5\") " pod="openshift-monitoring/node-exporter-wp9c7"
Sep 17 17:37:41.000054 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:41.000040    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e29f50f9-0bcf-4660-8348-914273f06fc5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wp9c7\" (UID: \"e29f50f9-0bcf-4660-8348-914273f06fc5\") " pod="openshift-monitoring/node-exporter-wp9c7"
Sep 17 17:37:41.009054 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:41.009030    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhdst\" (UniqueName: \"kubernetes.io/projected/e29f50f9-0bcf-4660-8348-914273f06fc5-kube-api-access-vhdst\") pod \"node-exporter-wp9c7\" (UID: \"e29f50f9-0bcf-4660-8348-914273f06fc5\") " pod="openshift-monitoring/node-exporter-wp9c7"
Sep 17 17:37:41.074628 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:41.074554    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:41.173785 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:41.173753    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:41.273481 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:41.273421    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:41.348231 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:41.348165    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:41.444736 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:41.444708    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:41.502032 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:41.502000    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e29f50f9-0bcf-4660-8348-914273f06fc5-node-exporter-tls\") pod \"node-exporter-wp9c7\" (UID: \"e29f50f9-0bcf-4660-8348-914273f06fc5\") " pod="openshift-monitoring/node-exporter-wp9c7"
Sep 17 17:37:41.504344 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:41.504314    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e29f50f9-0bcf-4660-8348-914273f06fc5-node-exporter-tls\") pod \"node-exporter-wp9c7\" (UID: \"e29f50f9-0bcf-4660-8348-914273f06fc5\") " pod="openshift-monitoring/node-exporter-wp9c7"
Sep 17 17:37:41.709078 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:41.709008    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wp9c7"
Sep 17 17:37:41.721398 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:37:41.721355    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode29f50f9_0bcf_4660_8348_914273f06fc5.slice/crio-ddbe6fe228cd6ad613c7ff100a0c93d870284c135e256504ed0849e4582b303c WatchSource:0}: Error finding container ddbe6fe228cd6ad613c7ff100a0c93d870284c135e256504ed0849e4582b303c: Status 404 returned error can't find the container with id ddbe6fe228cd6ad613c7ff100a0c93d870284c135e256504ed0849e4582b303c
Sep 17 17:37:41.757253 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:41.757223    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wp9c7" event={"ID":"e29f50f9-0bcf-4660-8348-914273f06fc5","Type":"ContainerStarted","Data":"ddbe6fe228cd6ad613c7ff100a0c93d870284c135e256504ed0849e4582b303c"}
Sep 17 17:37:41.784097 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:41.784071    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:41.893158 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:41.893132    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:41.957559 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:41.957525    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:42.763243 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:42.763213    2585 generic.go:356] "Generic (PLEG): container finished" podID="e29f50f9-0bcf-4660-8348-914273f06fc5" containerID="c899ff8a25e94aa6ea4ef7f5aa99832bd485ee742de54ee3cefafa0ebc9a11cf" exitCode=0
Sep 17 17:37:42.763563 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:42.763275    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wp9c7" event={"ID":"e29f50f9-0bcf-4660-8348-914273f06fc5","Type":"ContainerDied","Data":"c899ff8a25e94aa6ea4ef7f5aa99832bd485ee742de54ee3cefafa0ebc9a11cf"}
Sep 17 17:37:43.758435 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:43.758403    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:43.768391 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:43.768366    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wp9c7" event={"ID":"e29f50f9-0bcf-4660-8348-914273f06fc5","Type":"ContainerStarted","Data":"62cf7aef17bb2b53bb5922cd5e43a48537857b505c9f781a6fe302b47ef8c526"}
Sep 17 17:37:43.768826 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:43.768412    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wp9c7" event={"ID":"e29f50f9-0bcf-4660-8348-914273f06fc5","Type":"ContainerStarted","Data":"9ed63e4edb045cb8be59bb23a729870144e34b6464a41a19044eb00a1d19b2c1"}
Sep 17 17:37:43.793256 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:43.793215    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-wp9c7" podStartSLOduration=2.860882319 podStartE2EDuration="3.793200049s" totalImagesPullingTime="872.952156ms" totalInitContainerRuntime="59.365574ms" isStatefulPod=true podCreationTimestamp="2026-09-17 17:37:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:37:43.791727816 +0000 UTC m=+200.205518253" watchObservedRunningTime="2026-09-17 17:37:43.793200049 +0000 UTC m=+200.206990486"
Sep 17 17:37:43.859476 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:43.859445    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:44.031348 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:44.031256    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:44.139939 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:44.139907    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:45.570672 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:45.570635    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:45.951338 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:45.951268    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:47.301338 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:47.301307    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:48.516992 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:48.516956    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:48.586390 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:48.586353    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:52.196224 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:52.196186    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:52.291326 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:52.291299    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:52.487973 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:52.487884    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:52.589790 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:52.589759    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:52.690120 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:52.690094    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:52.789413 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:52.789381    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:53.024220 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:53.024179    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:53.152198 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:53.152109    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:53.257295 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:53.257264    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:53.410173 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:53.410101    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:53.509204 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:53.509180    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:53.863197 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:53.863159    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:54.041987 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:54.041956    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:54.362100 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:54.362069    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:55.292231 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:55.292177    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:55.446217 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:55.446190    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:57.515961 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:57.515930    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:57.616258 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:57.616231    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:59.215873 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:59.215840    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:59.325594 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:59.325565    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:37:59.420977 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:37:59.420948    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:00.151256 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:38:00.151223    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:00.251655 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:38:00.251628    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:02.853939 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:38:02.853899    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:02.959797 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:38:02.959765    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:04.059100 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:38:04.059066    2585 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57d467ccf7-rw9lz" podUID="f2ee4712-52c9-434d-b7f3-247871387e09" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 17 17:38:05.683292 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:38:05.683263    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:05.755470 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:38:05.755422    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:06.589830 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:38:06.589798    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:06.658899 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:38:06.658870    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:09.090827 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:38:09.090784    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:11.621863 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:38:11.621829    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:11.725477 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:38:11.725446    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:13.655094 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:38:13.655065    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:13.730480 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:38:13.730453    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:14.059266 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:38:14.059226    2585 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57d467ccf7-rw9lz" podUID="f2ee4712-52c9-434d-b7f3-247871387e09" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 17 17:38:21.653538 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:38:21.653484    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:21.747260 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:38:21.747208    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:24.058918 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:38:24.058886    2585 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57d467ccf7-rw9lz" podUID="f2ee4712-52c9-434d-b7f3-247871387e09" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 17 17:38:24.059285 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:38:24.058956    2585 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57d467ccf7-rw9lz"
Sep 17 17:38:24.059414 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:38:24.059387    2585 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"3a487ce198a0c9886b3db83fbe6a987b0855f9eaa12abdb2b1f4b2659d103b64"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57d467ccf7-rw9lz" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 17 17:38:24.059450 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:38:24.059434    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57d467ccf7-rw9lz" podUID="f2ee4712-52c9-434d-b7f3-247871387e09" containerName="service-proxy" containerID="cri-o://3a487ce198a0c9886b3db83fbe6a987b0855f9eaa12abdb2b1f4b2659d103b64" gracePeriod=30
Sep 17 17:38:24.214546 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:38:24.214515    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:24.310157 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:38:24.310076    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:24.895400 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:38:24.895372    2585 generic.go:356] "Generic (PLEG): container finished" podID="f2ee4712-52c9-434d-b7f3-247871387e09" containerID="3a487ce198a0c9886b3db83fbe6a987b0855f9eaa12abdb2b1f4b2659d103b64" exitCode=2
Sep 17 17:38:24.895581 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:38:24.895412    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57d467ccf7-rw9lz" event={"ID":"f2ee4712-52c9-434d-b7f3-247871387e09","Type":"ContainerDied","Data":"3a487ce198a0c9886b3db83fbe6a987b0855f9eaa12abdb2b1f4b2659d103b64"}
Sep 17 17:38:24.895581 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:38:24.895444    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57d467ccf7-rw9lz" event={"ID":"f2ee4712-52c9-434d-b7f3-247871387e09","Type":"ContainerStarted","Data":"b632c97668c7e26f319d9777bf12a74e33bd731cfc9cd10476835d3f2cf1058b"}
Sep 17 17:38:31.400324 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:38:31.400287    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kgbmb_6c27507b-19a0-4c12-a379-10474f260f84/dns-node-resolver/0.log"
Sep 17 17:38:31.820473 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:38:31.820443    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:31.881858 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:38:31.881825    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:35.884879 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:38:35.884841    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39af04b6-713b-406b-952d-7b2ef2b2b952-metrics-certs\") pod \"network-metrics-daemon-p7vh6\" (UID: \"39af04b6-713b-406b-952d-7b2ef2b2b952\") " pod="openshift-multus/network-metrics-daemon-p7vh6"
Sep 17 17:38:35.887014 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:38:35.886996    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39af04b6-713b-406b-952d-7b2ef2b2b952-metrics-certs\") pod \"network-metrics-daemon-p7vh6\" (UID: \"39af04b6-713b-406b-952d-7b2ef2b2b952\") " pod="openshift-multus/network-metrics-daemon-p7vh6"
Sep 17 17:38:36.004567 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:38:36.004537    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-g24m8\""
Sep 17 17:38:36.012481 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:38:36.012459    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p7vh6"
Sep 17 17:38:36.279146 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:38:36.279118    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-p7vh6"]
Sep 17 17:38:36.282092 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:38:36.282065    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39af04b6_713b_406b_952d_7b2ef2b2b952.slice/crio-26532f2de871457c847a7d0635237a867e57d8a9e032523a64c8ac8f01486825 WatchSource:0}: Error finding container 26532f2de871457c847a7d0635237a867e57d8a9e032523a64c8ac8f01486825: Status 404 returned error can't find the container with id 26532f2de871457c847a7d0635237a867e57d8a9e032523a64c8ac8f01486825
Sep 17 17:38:36.934754 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:38:36.934715    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p7vh6" event={"ID":"39af04b6-713b-406b-952d-7b2ef2b2b952","Type":"ContainerStarted","Data":"26532f2de871457c847a7d0635237a867e57d8a9e032523a64c8ac8f01486825"}
Sep 17 17:38:37.403394 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:38:37.403365    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:37.939817 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:38:37.939783    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p7vh6" event={"ID":"39af04b6-713b-406b-952d-7b2ef2b2b952","Type":"ContainerStarted","Data":"fe7ed268ef63844a7028393fed62816a04db43c66f4fec484e4537a189348701"}
Sep 17 17:38:37.940180 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:38:37.939834    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p7vh6" event={"ID":"39af04b6-713b-406b-952d-7b2ef2b2b952","Type":"ContainerStarted","Data":"7c508760bdafdf4d457c7449222e069d04e0ca67913b16a7a71e416d801e6a67"}
Sep 17 17:38:37.960518 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:38:37.960459    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-p7vh6" podStartSLOduration=252.964795729 podStartE2EDuration="4m13.960446147s" totalImagesPullingTime="995.650418ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:38:37.959299541 +0000 UTC m=+254.373089980" watchObservedRunningTime="2026-09-17 17:38:37.960446147 +0000 UTC m=+254.374236584"
Sep 17 17:38:38.068343 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:38:38.068309    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:38.130927 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:38:38.130895    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:41.618986 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:38:41.618952    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:41.681057 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:38:41.681025    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:58.910607 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:38:58.910534    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:38:58.998163 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:38:58.998131    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:39:00.185430 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:39:00.185395    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:39:00.391193 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:39:00.391160    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:39:02.985117 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:39:02.985087    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:39:03.058565 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:39:03.058523    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:39:03.625255 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:39:03.625212    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-t7fvf" podUID="3973eda4-9544-4956-980a-4c11ba7700c1"
Sep 17 17:39:04.019799 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:39:04.019773    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-t7fvf"
Sep 17 17:39:07.510230 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:39:07.510190    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3973eda4-9544-4956-980a-4c11ba7700c1-metrics-tls\") pod \"dns-default-t7fvf\" (UID: \"3973eda4-9544-4956-980a-4c11ba7700c1\") " pod="openshift-dns/dns-default-t7fvf"
Sep 17 17:39:07.512464 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:39:07.512440    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3973eda4-9544-4956-980a-4c11ba7700c1-metrics-tls\") pod \"dns-default-t7fvf\" (UID: \"3973eda4-9544-4956-980a-4c11ba7700c1\") " pod="openshift-dns/dns-default-t7fvf"
Sep 17 17:39:07.623095 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:39:07.623070    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bqqq4\""
Sep 17 17:39:07.630787 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:39:07.630773    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-t7fvf"
Sep 17 17:39:07.910932 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:39:07.910863    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-t7fvf"]
Sep 17 17:39:07.913979 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:39:07.913954    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3973eda4_9544_4956_980a_4c11ba7700c1.slice/crio-e270dae50ea209e9768e3e528b31816ec21e06531e1802ca955068d8e583d37f WatchSource:0}: Error finding container e270dae50ea209e9768e3e528b31816ec21e06531e1802ca955068d8e583d37f: Status 404 returned error can't find the container with id e270dae50ea209e9768e3e528b31816ec21e06531e1802ca955068d8e583d37f
Sep 17 17:39:08.033361 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:39:08.033323    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-t7fvf" event={"ID":"3973eda4-9544-4956-980a-4c11ba7700c1","Type":"ContainerStarted","Data":"e270dae50ea209e9768e3e528b31816ec21e06531e1802ca955068d8e583d37f"}
Sep 17 17:39:08.094177 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:39:08.094152    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:39:09.041021 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:39:09.040980    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:39:09.141655 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:39:09.141619    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:39:10.042167 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:39:10.042138    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-t7fvf" event={"ID":"3973eda4-9544-4956-980a-4c11ba7700c1","Type":"ContainerStarted","Data":"24eb30e0aad1a08c5d8d87c9f99c9265941289ac96e18006c5e3d7bf67410ab9"}
Sep 17 17:39:10.042671 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:39:10.042184    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-t7fvf" event={"ID":"3973eda4-9544-4956-980a-4c11ba7700c1","Type":"ContainerStarted","Data":"76d5a41143a0a4d83f694ab50d469fec38e07fe80c26275b9d9f4b3d01f5fd80"}
Sep 17 17:39:10.042671 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:39:10.042243    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-t7fvf"
Sep 17 17:39:10.076247 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:39:10.076203    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-t7fvf" podStartSLOduration=251.503034497 podStartE2EDuration="4m13.076190023s" totalImagesPullingTime="1.573155526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:34:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:39:10.074965503 +0000 UTC m=+286.488755941" watchObservedRunningTime="2026-09-17 17:39:10.076190023 +0000 UTC m=+286.489980460"
Sep 17 17:39:11.335368 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:39:11.335335    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:39:11.408888 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:39:11.408856    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:39:20.047251 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:39:20.047211    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-t7fvf"
Sep 17 17:39:20.066857 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:39:20.066832    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:39:20.239246 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:39:20.239215    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:39:23.015196 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:39:23.015160    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:39:23.122303 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:39:23.122271    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:39:23.998717 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:39:23.998682    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rrf79_beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5/ovn-acl-logging/0.log"
Sep 17 17:39:24.005371 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:39:24.005351    2585 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 17:39:24.012397 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:39:24.012382    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rrf79_beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5/ovn-acl-logging/0.log"
Sep 17 17:39:32.136974 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:39:32.136940    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:39:32.243835 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:39:32.243800    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:40:01.414085 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:40:01.414051    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:40:01.506728 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:40:01.506688    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:40:15.433673 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:40:15.433637    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:40:15.515832 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:40:15.515798    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:40:27.324936 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:40:27.324898    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:40:27.425483 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:40:27.425453    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:40:41.633296 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:40:41.633256    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:40:41.734063 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:40:41.734036    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:40:45.036644 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:40:45.036610    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:40:45.114439 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:40:45.114403    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:40:45.191331 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:40:45.191305    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:40:45.807752 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:40:45.807718    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:40:45.868706 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:40:45.868681    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:40:56.484822 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:40:56.484783    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:40:56.577999 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:40:56.577961    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:17.280196 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:17.280159    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:17.373212 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:17.373179    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:17.477280 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:17.477253    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:18.546043 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:18.546007    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1am2n86"]
Sep 17 17:41:18.549152 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:18.549132    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1am2n86"
Sep 17 17:41:18.551920 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:18.551890    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 17:41:18.552978 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:18.552955    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 17:41:18.553063 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:18.553046    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-plk9b\""
Sep 17 17:41:18.559315 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:18.559295    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1am2n86"]
Sep 17 17:41:18.645719 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:18.645690    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfxzh\" (UniqueName: \"kubernetes.io/projected/cb541ebf-968a-4357-b603-52287c87a63f-kube-api-access-lfxzh\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1am2n86\" (UID: \"cb541ebf-968a-4357-b603-52287c87a63f\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1am2n86"
Sep 17 17:41:18.645874 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:18.645727    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb541ebf-968a-4357-b603-52287c87a63f-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1am2n86\" (UID: \"cb541ebf-968a-4357-b603-52287c87a63f\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1am2n86"
Sep 17 17:41:18.645874 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:18.645743    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cb541ebf-968a-4357-b603-52287c87a63f-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1am2n86\" (UID: \"cb541ebf-968a-4357-b603-52287c87a63f\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1am2n86"
Sep 17 17:41:18.746320 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:18.746289    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb541ebf-968a-4357-b603-52287c87a63f-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1am2n86\" (UID: \"cb541ebf-968a-4357-b603-52287c87a63f\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1am2n86"
Sep 17 17:41:18.746320 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:18.746319    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cb541ebf-968a-4357-b603-52287c87a63f-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1am2n86\" (UID: \"cb541ebf-968a-4357-b603-52287c87a63f\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1am2n86"
Sep 17 17:41:18.746520 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:18.746381    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfxzh\" (UniqueName: \"kubernetes.io/projected/cb541ebf-968a-4357-b603-52287c87a63f-kube-api-access-lfxzh\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1am2n86\" (UID: \"cb541ebf-968a-4357-b603-52287c87a63f\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1am2n86"
Sep 17 17:41:18.746736 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:18.746717    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb541ebf-968a-4357-b603-52287c87a63f-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1am2n86\" (UID: \"cb541ebf-968a-4357-b603-52287c87a63f\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1am2n86"
Sep 17 17:41:18.746774 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:18.746726    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cb541ebf-968a-4357-b603-52287c87a63f-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1am2n86\" (UID: \"cb541ebf-968a-4357-b603-52287c87a63f\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1am2n86"
Sep 17 17:41:18.755281 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:18.755256    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfxzh\" (UniqueName: \"kubernetes.io/projected/cb541ebf-968a-4357-b603-52287c87a63f-kube-api-access-lfxzh\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1am2n86\" (UID: \"cb541ebf-968a-4357-b603-52287c87a63f\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1am2n86"
Sep 17 17:41:18.857325 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:18.857258    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1am2n86"
Sep 17 17:41:19.149145 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:19.149122    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1am2n86"]
Sep 17 17:41:19.151366 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:41:19.151335    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb541ebf_968a_4357_b603_52287c87a63f.slice/crio-10e93ca0a36f01417067982abecb8a8716a564371f70b86ba8192501cbbdc4ed WatchSource:0}: Error finding container 10e93ca0a36f01417067982abecb8a8716a564371f70b86ba8192501cbbdc4ed: Status 404 returned error can't find the container with id 10e93ca0a36f01417067982abecb8a8716a564371f70b86ba8192501cbbdc4ed
Sep 17 17:41:19.153219 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:19.153199    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:41:19.435363 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:19.435283    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1am2n86" event={"ID":"cb541ebf-968a-4357-b603-52287c87a63f","Type":"ContainerStarted","Data":"10e93ca0a36f01417067982abecb8a8716a564371f70b86ba8192501cbbdc4ed"}
Sep 17 17:41:25.460292 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:25.460264    2585 generic.go:356] "Generic (PLEG): container finished" podID="cb541ebf-968a-4357-b603-52287c87a63f" containerID="e2eae26c1ae7a82a617b0b01dc0ed8c05d4680a285c448419ca0a7034d10f7bc" exitCode=0
Sep 17 17:41:25.460651 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:25.460350    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1am2n86" event={"ID":"cb541ebf-968a-4357-b603-52287c87a63f","Type":"ContainerDied","Data":"e2eae26c1ae7a82a617b0b01dc0ed8c05d4680a285c448419ca0a7034d10f7bc"}
Sep 17 17:41:28.472505 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:28.472458    2585 generic.go:356] "Generic (PLEG): container finished" podID="cb541ebf-968a-4357-b603-52287c87a63f" containerID="0084e42908b2c740f9bba7f2d0f98afe803a4946ad248c936affabff4d064297" exitCode=0
Sep 17 17:41:28.472871 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:28.472546    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1am2n86" event={"ID":"cb541ebf-968a-4357-b603-52287c87a63f","Type":"ContainerDied","Data":"0084e42908b2c740f9bba7f2d0f98afe803a4946ad248c936affabff4d064297"}
Sep 17 17:41:35.498783 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:35.498745    2585 generic.go:356] "Generic (PLEG): container finished" podID="cb541ebf-968a-4357-b603-52287c87a63f" containerID="5491244f7fb5b131c55477f8735a19af46a8024f500d6eb3b314b00655e8aa0a" exitCode=0
Sep 17 17:41:35.499185 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:35.498805    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1am2n86" event={"ID":"cb541ebf-968a-4357-b603-52287c87a63f","Type":"ContainerDied","Data":"5491244f7fb5b131c55477f8735a19af46a8024f500d6eb3b314b00655e8aa0a"}
Sep 17 17:41:36.616266 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:36.616242    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1am2n86"
Sep 17 17:41:36.679392 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:36.679362    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb541ebf-968a-4357-b603-52287c87a63f-bundle\") pod \"cb541ebf-968a-4357-b603-52287c87a63f\" (UID: \"cb541ebf-968a-4357-b603-52287c87a63f\") "
Sep 17 17:41:36.679551 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:36.679401    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cb541ebf-968a-4357-b603-52287c87a63f-util\") pod \"cb541ebf-968a-4357-b603-52287c87a63f\" (UID: \"cb541ebf-968a-4357-b603-52287c87a63f\") "
Sep 17 17:41:36.679551 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:36.679450    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lfxzh\" (UniqueName: \"kubernetes.io/projected/cb541ebf-968a-4357-b603-52287c87a63f-kube-api-access-lfxzh\") pod \"cb541ebf-968a-4357-b603-52287c87a63f\" (UID: \"cb541ebf-968a-4357-b603-52287c87a63f\") "
Sep 17 17:41:36.679955 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:36.679932    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb541ebf-968a-4357-b603-52287c87a63f-bundle" (OuterVolumeSpecName: "bundle") pod "cb541ebf-968a-4357-b603-52287c87a63f" (UID: "cb541ebf-968a-4357-b603-52287c87a63f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:41:36.681642 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:36.681619    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb541ebf-968a-4357-b603-52287c87a63f-kube-api-access-lfxzh" (OuterVolumeSpecName: "kube-api-access-lfxzh") pod "cb541ebf-968a-4357-b603-52287c87a63f" (UID: "cb541ebf-968a-4357-b603-52287c87a63f"). InnerVolumeSpecName "kube-api-access-lfxzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:41:36.683460 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:36.683441    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb541ebf-968a-4357-b603-52287c87a63f-util" (OuterVolumeSpecName: "util") pod "cb541ebf-968a-4357-b603-52287c87a63f" (UID: "cb541ebf-968a-4357-b603-52287c87a63f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:41:36.780051 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:36.779976    2585 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cb541ebf-968a-4357-b603-52287c87a63f-bundle\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 17:41:36.780051 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:36.780010    2585 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cb541ebf-968a-4357-b603-52287c87a63f-util\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 17:41:36.780051 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:36.780020    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfxzh\" (UniqueName: \"kubernetes.io/projected/cb541ebf-968a-4357-b603-52287c87a63f-kube-api-access-lfxzh\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 17:41:37.507564 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:37.507530    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1am2n86" event={"ID":"cb541ebf-968a-4357-b603-52287c87a63f","Type":"ContainerDied","Data":"10e93ca0a36f01417067982abecb8a8716a564371f70b86ba8192501cbbdc4ed"}
Sep 17 17:41:37.507754 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:37.507588    2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10e93ca0a36f01417067982abecb8a8716a564371f70b86ba8192501cbbdc4ed"
Sep 17 17:41:37.507754 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:37.507546    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1am2n86"
Sep 17 17:41:40.167849 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:40.167816    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:41.497835 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:41.497806    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:47.094984 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:47.094904    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:47.384963 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:47.384865    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:47.394225 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:47.394200    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-5bh9f"]
Sep 17 17:41:47.394431 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:47.394419    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb541ebf-968a-4357-b603-52287c87a63f" containerName="util"
Sep 17 17:41:47.394475 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:47.394433    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb541ebf-968a-4357-b603-52287c87a63f" containerName="util"
Sep 17 17:41:47.394475 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:47.394447    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb541ebf-968a-4357-b603-52287c87a63f" containerName="extract"
Sep 17 17:41:47.394475 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:47.394453    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb541ebf-968a-4357-b603-52287c87a63f" containerName="extract"
Sep 17 17:41:47.394475 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:47.394464    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb541ebf-968a-4357-b603-52287c87a63f" containerName="pull"
Sep 17 17:41:47.394475 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:47.394470    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb541ebf-968a-4357-b603-52287c87a63f" containerName="pull"
Sep 17 17:41:47.394649 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:47.394518    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb541ebf-968a-4357-b603-52287c87a63f" containerName="extract"
Sep 17 17:41:47.397827 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:47.397813    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-5bh9f"
Sep 17 17:41:47.401120 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:47.401098    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 17 17:41:47.401209 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:47.401181    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 17 17:41:47.401268 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:47.401182    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-h6dsj\""
Sep 17 17:41:47.402118 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:47.402100    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 17 17:41:47.402224 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:47.402139    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 17 17:41:47.402224 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:47.402165    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 17 17:41:47.409902 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:47.409883    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-5bh9f"]
Sep 17 17:41:47.545964 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:47.545934    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/7a079be6-55f8-4571-8490-2e575c9639e5-cabundle0\") pod \"keda-operator-579c8958f7-5bh9f\" (UID: \"7a079be6-55f8-4571-8490-2e575c9639e5\") " pod="openshift-keda/keda-operator-579c8958f7-5bh9f"
Sep 17 17:41:47.545964 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:47.545963    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6klhc\" (UniqueName: \"kubernetes.io/projected/7a079be6-55f8-4571-8490-2e575c9639e5-kube-api-access-6klhc\") pod \"keda-operator-579c8958f7-5bh9f\" (UID: \"7a079be6-55f8-4571-8490-2e575c9639e5\") " pod="openshift-keda/keda-operator-579c8958f7-5bh9f"
Sep 17 17:41:47.546157 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:47.545993    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7a079be6-55f8-4571-8490-2e575c9639e5-certificates\") pod \"keda-operator-579c8958f7-5bh9f\" (UID: \"7a079be6-55f8-4571-8490-2e575c9639e5\") " pod="openshift-keda/keda-operator-579c8958f7-5bh9f"
Sep 17 17:41:47.625439 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:47.625412    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:47.646872 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:47.646802    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/7a079be6-55f8-4571-8490-2e575c9639e5-cabundle0\") pod \"keda-operator-579c8958f7-5bh9f\" (UID: \"7a079be6-55f8-4571-8490-2e575c9639e5\") " pod="openshift-keda/keda-operator-579c8958f7-5bh9f"
Sep 17 17:41:47.646872 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:47.646832    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6klhc\" (UniqueName: \"kubernetes.io/projected/7a079be6-55f8-4571-8490-2e575c9639e5-kube-api-access-6klhc\") pod \"keda-operator-579c8958f7-5bh9f\" (UID: \"7a079be6-55f8-4571-8490-2e575c9639e5\") " pod="openshift-keda/keda-operator-579c8958f7-5bh9f"
Sep 17 17:41:47.646872 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:47.646869    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7a079be6-55f8-4571-8490-2e575c9639e5-certificates\") pod \"keda-operator-579c8958f7-5bh9f\" (UID: \"7a079be6-55f8-4571-8490-2e575c9639e5\") " pod="openshift-keda/keda-operator-579c8958f7-5bh9f"
Sep 17 17:41:47.647096 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:41:47.646973    2585 secret.go:281] references non-existent secret key: ca.crt
Sep 17 17:41:47.647096 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:41:47.646987    2585 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 17 17:41:47.647096 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:41:47.646997    2585 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-5bh9f: references non-existent secret key: ca.crt
Sep 17 17:41:47.647096 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:41:47.647055    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a079be6-55f8-4571-8490-2e575c9639e5-certificates podName:7a079be6-55f8-4571-8490-2e575c9639e5 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:48.147037089 +0000 UTC m=+444.560827509 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/7a079be6-55f8-4571-8490-2e575c9639e5-certificates") pod "keda-operator-579c8958f7-5bh9f" (UID: "7a079be6-55f8-4571-8490-2e575c9639e5") : references non-existent secret key: ca.crt
Sep 17 17:41:47.647456 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:47.647438    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/7a079be6-55f8-4571-8490-2e575c9639e5-cabundle0\") pod \"keda-operator-579c8958f7-5bh9f\" (UID: \"7a079be6-55f8-4571-8490-2e575c9639e5\") " pod="openshift-keda/keda-operator-579c8958f7-5bh9f"
Sep 17 17:41:47.657070 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:47.657048    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6klhc\" (UniqueName: \"kubernetes.io/projected/7a079be6-55f8-4571-8490-2e575c9639e5-kube-api-access-6klhc\") pod \"keda-operator-579c8958f7-5bh9f\" (UID: \"7a079be6-55f8-4571-8490-2e575c9639e5\") " pod="openshift-keda/keda-operator-579c8958f7-5bh9f"
Sep 17 17:41:47.725108 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:47.725078    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:47.829077 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:47.829048    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-6hvtp"]
Sep 17 17:41:47.831009 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:47.830994    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-6hvtp"
Sep 17 17:41:47.833187 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:47.833171    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 17 17:41:47.843063 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:47.843034    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-6hvtp"]
Sep 17 17:41:47.920621 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:47.920540    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:47.948221 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:47.948199    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/b70daa64-f068-4601-85c0-3837dcfda09c-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-6hvtp\" (UID: \"b70daa64-f068-4601-85c0-3837dcfda09c\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-6hvtp"
Sep 17 17:41:47.948303 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:47.948237    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b70daa64-f068-4601-85c0-3837dcfda09c-certificates\") pod \"keda-metrics-apiserver-77857b7b98-6hvtp\" (UID: \"b70daa64-f068-4601-85c0-3837dcfda09c\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-6hvtp"
Sep 17 17:41:47.948340 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:47.948316    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m78bw\" (UniqueName: \"kubernetes.io/projected/b70daa64-f068-4601-85c0-3837dcfda09c-kube-api-access-m78bw\") pod \"keda-metrics-apiserver-77857b7b98-6hvtp\" (UID: \"b70daa64-f068-4601-85c0-3837dcfda09c\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-6hvtp"
Sep 17 17:41:48.049587 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:48.049552    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m78bw\" (UniqueName: \"kubernetes.io/projected/b70daa64-f068-4601-85c0-3837dcfda09c-kube-api-access-m78bw\") pod \"keda-metrics-apiserver-77857b7b98-6hvtp\" (UID: \"b70daa64-f068-4601-85c0-3837dcfda09c\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-6hvtp"
Sep 17 17:41:48.049804 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:48.049595    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/b70daa64-f068-4601-85c0-3837dcfda09c-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-6hvtp\" (UID: \"b70daa64-f068-4601-85c0-3837dcfda09c\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-6hvtp"
Sep 17 17:41:48.049804 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:48.049614    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b70daa64-f068-4601-85c0-3837dcfda09c-certificates\") pod \"keda-metrics-apiserver-77857b7b98-6hvtp\" (UID: \"b70daa64-f068-4601-85c0-3837dcfda09c\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-6hvtp"
Sep 17 17:41:48.049992 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:48.049975    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/b70daa64-f068-4601-85c0-3837dcfda09c-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-6hvtp\" (UID: \"b70daa64-f068-4601-85c0-3837dcfda09c\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-6hvtp"
Sep 17 17:41:48.052127 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:48.052100    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b70daa64-f068-4601-85c0-3837dcfda09c-certificates\") pod \"keda-metrics-apiserver-77857b7b98-6hvtp\" (UID: \"b70daa64-f068-4601-85c0-3837dcfda09c\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-6hvtp"
Sep 17 17:41:48.058879 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:48.058862    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m78bw\" (UniqueName: \"kubernetes.io/projected/b70daa64-f068-4601-85c0-3837dcfda09c-kube-api-access-m78bw\") pod \"keda-metrics-apiserver-77857b7b98-6hvtp\" (UID: \"b70daa64-f068-4601-85c0-3837dcfda09c\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-6hvtp"
Sep 17 17:41:48.140683 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:48.140653    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-6hvtp"
Sep 17 17:41:48.144545 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:48.144514    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:48.150436 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:48.150415    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7a079be6-55f8-4571-8490-2e575c9639e5-certificates\") pod \"keda-operator-579c8958f7-5bh9f\" (UID: \"7a079be6-55f8-4571-8490-2e575c9639e5\") " pod="openshift-keda/keda-operator-579c8958f7-5bh9f"
Sep 17 17:41:48.152840 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:48.152819    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7a079be6-55f8-4571-8490-2e575c9639e5-certificates\") pod \"keda-operator-579c8958f7-5bh9f\" (UID: \"7a079be6-55f8-4571-8490-2e575c9639e5\") " pod="openshift-keda/keda-operator-579c8958f7-5bh9f"
Sep 17 17:41:48.245302 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:48.245272    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:48.307203 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:48.307178    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-5bh9f"
Sep 17 17:41:48.352449 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:48.352398    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:48.412565 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:48.412540    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-6hvtp"]
Sep 17 17:41:48.414984 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:41:48.414961    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb70daa64_f068_4601_85c0_3837dcfda09c.slice/crio-c395ca483d985943429e45af547d6b13d4b066682245c63855f978a6c64d6116 WatchSource:0}: Error finding container c395ca483d985943429e45af547d6b13d4b066682245c63855f978a6c64d6116: Status 404 returned error can't find the container with id c395ca483d985943429e45af547d6b13d4b066682245c63855f978a6c64d6116
Sep 17 17:41:48.484061 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:48.484003    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:48.543976 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:48.543955    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-6hvtp" event={"ID":"b70daa64-f068-4601-85c0-3837dcfda09c","Type":"ContainerStarted","Data":"c395ca483d985943429e45af547d6b13d4b066682245c63855f978a6c64d6116"}
Sep 17 17:41:48.575424 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:48.575400    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-5bh9f"]
Sep 17 17:41:48.578206 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:41:48.578183    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a079be6_55f8_4571_8490_2e575c9639e5.slice/crio-723177475a9e34460d7abbe2ed2e30c7da3ccc8e966af1b37c43214df90eed0b WatchSource:0}: Error finding container 723177475a9e34460d7abbe2ed2e30c7da3ccc8e966af1b37c43214df90eed0b: Status 404 returned error can't find the container with id 723177475a9e34460d7abbe2ed2e30c7da3ccc8e966af1b37c43214df90eed0b
Sep 17 17:41:48.698911 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:48.698874    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:48.793646 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:48.793618    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:48.894221 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:48.894199    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:49.550753 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:49.550711    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-5bh9f" event={"ID":"7a079be6-55f8-4571-8490-2e575c9639e5","Type":"ContainerStarted","Data":"723177475a9e34460d7abbe2ed2e30c7da3ccc8e966af1b37c43214df90eed0b"}
Sep 17 17:41:52.564607 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:52.564574    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-6hvtp" event={"ID":"b70daa64-f068-4601-85c0-3837dcfda09c","Type":"ContainerStarted","Data":"f8d8e4a675972f8d920d1165b94d9c5c2b0764da44c1f0722617f117d2bb509b"}
Sep 17 17:41:52.565076 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:52.564678    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-6hvtp"
Sep 17 17:41:52.566200 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:52.566180    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-5bh9f" event={"ID":"7a079be6-55f8-4571-8490-2e575c9639e5","Type":"ContainerStarted","Data":"cfef18553979235088615800c9c6ecad4d17043e8ad817f006763918f598fcfc"}
Sep 17 17:41:52.566300 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:52.566228    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-5bh9f"
Sep 17 17:41:52.584285 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:52.584243    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-6hvtp" podStartSLOduration=1.8139185740000001 podStartE2EDuration="5.5842317s" totalImagesPullingTime="3.770313126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:52.58228926 +0000 UTC m=+448.996079700" watchObservedRunningTime="2026-09-17 17:41:52.5842317 +0000 UTC m=+448.998022147"
Sep 17 17:41:52.601333 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:52.601290    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-5bh9f" podStartSLOduration=1.9875120229999999 podStartE2EDuration="5.601276651s" totalImagesPullingTime="3.613764628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:52.600081025 +0000 UTC m=+449.013871476" watchObservedRunningTime="2026-09-17 17:41:52.601276651 +0000 UTC m=+449.015067088"
Sep 17 17:41:54.123276 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:54.123241    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:41:54.207522 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:41:54.207482    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:03.574526 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:42:03.574479    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-6hvtp"
Sep 17 17:42:13.572571 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:42:13.572540    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-5bh9f"
Sep 17 17:42:39.259913 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:42:39.259870    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:42:39.360915 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:42:39.360876    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:03.491793 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:03.491757    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:03.565886 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:03.565856    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:07.660217 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:07.660180    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:07.732929 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:07.732900    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-7f654675b-kgv87"]
Sep 17 17:43:07.735884 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:07.735871    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7f654675b-kgv87"
Sep 17 17:43:07.739338 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:07.739315    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 17 17:43:07.739338 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:07.739326    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-skfht\""
Sep 17 17:43:07.739530 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:07.739331    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 17:43:07.739530 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:07.739351    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 17:43:07.746586 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:07.746563    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7f654675b-kgv87"]
Sep 17 17:43:07.753971 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:07.753943    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:07.811046 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:07.811021    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pt9x\" (UniqueName: \"kubernetes.io/projected/5d9d3373-15fd-4653-a148-95412f37d3c4-kube-api-access-8pt9x\") pod \"kserve-controller-manager-7f654675b-kgv87\" (UID: \"5d9d3373-15fd-4653-a148-95412f37d3c4\") " pod="kserve/kserve-controller-manager-7f654675b-kgv87"
Sep 17 17:43:07.811165 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:07.811052    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5d9d3373-15fd-4653-a148-95412f37d3c4-cert\") pod \"kserve-controller-manager-7f654675b-kgv87\" (UID: \"5d9d3373-15fd-4653-a148-95412f37d3c4\") " pod="kserve/kserve-controller-manager-7f654675b-kgv87"
Sep 17 17:43:07.877133 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:07.877108    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:07.911668 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:07.911608    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5d9d3373-15fd-4653-a148-95412f37d3c4-cert\") pod \"kserve-controller-manager-7f654675b-kgv87\" (UID: \"5d9d3373-15fd-4653-a148-95412f37d3c4\") " pod="kserve/kserve-controller-manager-7f654675b-kgv87"
Sep 17 17:43:07.911765 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:07.911668    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pt9x\" (UniqueName: \"kubernetes.io/projected/5d9d3373-15fd-4653-a148-95412f37d3c4-kube-api-access-8pt9x\") pod \"kserve-controller-manager-7f654675b-kgv87\" (UID: \"5d9d3373-15fd-4653-a148-95412f37d3c4\") " pod="kserve/kserve-controller-manager-7f654675b-kgv87"
Sep 17 17:43:07.913920 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:07.913893    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5d9d3373-15fd-4653-a148-95412f37d3c4-cert\") pod \"kserve-controller-manager-7f654675b-kgv87\" (UID: \"5d9d3373-15fd-4653-a148-95412f37d3c4\") " pod="kserve/kserve-controller-manager-7f654675b-kgv87"
Sep 17 17:43:07.919501 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:07.919468    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pt9x\" (UniqueName: \"kubernetes.io/projected/5d9d3373-15fd-4653-a148-95412f37d3c4-kube-api-access-8pt9x\") pod \"kserve-controller-manager-7f654675b-kgv87\" (UID: \"5d9d3373-15fd-4653-a148-95412f37d3c4\") " pod="kserve/kserve-controller-manager-7f654675b-kgv87"
Sep 17 17:43:07.978506 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:07.978467    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:08.044999 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:08.044980    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7f654675b-kgv87"
Sep 17 17:43:08.193361 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:08.193298    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:08.310438 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:08.310417    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7f654675b-kgv87"]
Sep 17 17:43:08.313150 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:43:08.313123    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d9d3373_15fd_4653_a148_95412f37d3c4.slice/crio-5b61b3953b3cf0e93f714094cdef1c54f4cbdcf5786ef51aa2d163a3e1cbc52d WatchSource:0}: Error finding container 5b61b3953b3cf0e93f714094cdef1c54f4cbdcf5786ef51aa2d163a3e1cbc52d: Status 404 returned error can't find the container with id 5b61b3953b3cf0e93f714094cdef1c54f4cbdcf5786ef51aa2d163a3e1cbc52d
Sep 17 17:43:08.344015 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:08.343990    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:08.444875 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:08.444793    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:08.509178 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:08.509141    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:08.828010 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:08.827979    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7f654675b-kgv87" event={"ID":"5d9d3373-15fd-4653-a148-95412f37d3c4","Type":"ContainerStarted","Data":"5b61b3953b3cf0e93f714094cdef1c54f4cbdcf5786ef51aa2d163a3e1cbc52d"}
Sep 17 17:43:10.838687 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:10.838655    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7f654675b-kgv87" event={"ID":"5d9d3373-15fd-4653-a148-95412f37d3c4","Type":"ContainerStarted","Data":"7511b8b04e3e4f2af3fcf56a9a3d74165918ab6766ad909b49f675195f9f6808"}
Sep 17 17:43:10.839134 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:10.838738    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-7f654675b-kgv87"
Sep 17 17:43:10.857255 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:10.857211    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-7f654675b-kgv87" podStartSLOduration=1.413685088 podStartE2EDuration="3.857196339s" totalImagesPullingTime="2.443511251s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:43:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:43:10.8563216 +0000 UTC m=+527.270112039" watchObservedRunningTime="2026-09-17 17:43:10.857196339 +0000 UTC m=+527.270986803"
Sep 17 17:43:14.042668 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:14.042628    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:14.143130 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:14.143095    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:14.207176 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:14.207144    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:43:41.847065 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:41.847033    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-7f654675b-kgv87"
Sep 17 17:43:51.616731 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:51.616696    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-7f654675b-kgv87"]
Sep 17 17:43:51.617113 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:51.616882    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-7f654675b-kgv87" podUID="5d9d3373-15fd-4653-a148-95412f37d3c4" containerName="manager" containerID="cri-o://7511b8b04e3e4f2af3fcf56a9a3d74165918ab6766ad909b49f675195f9f6808" gracePeriod=10
Sep 17 17:43:51.677351 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:51.677325    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-7f654675b-7vhmc"]
Sep 17 17:43:51.680228 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:51.680214    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7f654675b-7vhmc"
Sep 17 17:43:51.694566 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:51.694545    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7f654675b-7vhmc"]
Sep 17 17:43:51.825986 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:51.825946    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e8ed622f-3f48-4e28-af3b-fc634c8e52dd-cert\") pod \"kserve-controller-manager-7f654675b-7vhmc\" (UID: \"e8ed622f-3f48-4e28-af3b-fc634c8e52dd\") " pod="kserve/kserve-controller-manager-7f654675b-7vhmc"
Sep 17 17:43:51.826132 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:51.826046    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjcbd\" (UniqueName: \"kubernetes.io/projected/e8ed622f-3f48-4e28-af3b-fc634c8e52dd-kube-api-access-cjcbd\") pod \"kserve-controller-manager-7f654675b-7vhmc\" (UID: \"e8ed622f-3f48-4e28-af3b-fc634c8e52dd\") " pod="kserve/kserve-controller-manager-7f654675b-7vhmc"
Sep 17 17:43:51.848875 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:51.848855    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7f654675b-kgv87"
Sep 17 17:43:51.926764 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:51.926685    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjcbd\" (UniqueName: \"kubernetes.io/projected/e8ed622f-3f48-4e28-af3b-fc634c8e52dd-kube-api-access-cjcbd\") pod \"kserve-controller-manager-7f654675b-7vhmc\" (UID: \"e8ed622f-3f48-4e28-af3b-fc634c8e52dd\") " pod="kserve/kserve-controller-manager-7f654675b-7vhmc"
Sep 17 17:43:51.926764 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:51.926736    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e8ed622f-3f48-4e28-af3b-fc634c8e52dd-cert\") pod \"kserve-controller-manager-7f654675b-7vhmc\" (UID: \"e8ed622f-3f48-4e28-af3b-fc634c8e52dd\") " pod="kserve/kserve-controller-manager-7f654675b-7vhmc"
Sep 17 17:43:51.928904 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:51.928881    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e8ed622f-3f48-4e28-af3b-fc634c8e52dd-cert\") pod \"kserve-controller-manager-7f654675b-7vhmc\" (UID: \"e8ed622f-3f48-4e28-af3b-fc634c8e52dd\") " pod="kserve/kserve-controller-manager-7f654675b-7vhmc"
Sep 17 17:43:51.936326 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:51.936304    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjcbd\" (UniqueName: \"kubernetes.io/projected/e8ed622f-3f48-4e28-af3b-fc634c8e52dd-kube-api-access-cjcbd\") pod \"kserve-controller-manager-7f654675b-7vhmc\" (UID: \"e8ed622f-3f48-4e28-af3b-fc634c8e52dd\") " pod="kserve/kserve-controller-manager-7f654675b-7vhmc"
Sep 17 17:43:51.981752 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:51.981721    2585 generic.go:356] "Generic (PLEG): container finished" podID="5d9d3373-15fd-4653-a148-95412f37d3c4" containerID="7511b8b04e3e4f2af3fcf56a9a3d74165918ab6766ad909b49f675195f9f6808" exitCode=0
Sep 17 17:43:51.981863 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:51.981772    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7f654675b-kgv87"
Sep 17 17:43:51.981863 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:51.981813    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7f654675b-kgv87" event={"ID":"5d9d3373-15fd-4653-a148-95412f37d3c4","Type":"ContainerDied","Data":"7511b8b04e3e4f2af3fcf56a9a3d74165918ab6766ad909b49f675195f9f6808"}
Sep 17 17:43:51.981863 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:51.981861    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7f654675b-kgv87" event={"ID":"5d9d3373-15fd-4653-a148-95412f37d3c4","Type":"ContainerDied","Data":"5b61b3953b3cf0e93f714094cdef1c54f4cbdcf5786ef51aa2d163a3e1cbc52d"}
Sep 17 17:43:51.981960 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:51.981883    2585 scope.go:117] "RemoveContainer" containerID="7511b8b04e3e4f2af3fcf56a9a3d74165918ab6766ad909b49f675195f9f6808"
Sep 17 17:43:51.989125 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:51.989107    2585 scope.go:117] "RemoveContainer" containerID="7511b8b04e3e4f2af3fcf56a9a3d74165918ab6766ad909b49f675195f9f6808"
Sep 17 17:43:51.989371 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:43:51.989354    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7511b8b04e3e4f2af3fcf56a9a3d74165918ab6766ad909b49f675195f9f6808\": container with ID starting with 7511b8b04e3e4f2af3fcf56a9a3d74165918ab6766ad909b49f675195f9f6808 not found: ID does not exist" containerID="7511b8b04e3e4f2af3fcf56a9a3d74165918ab6766ad909b49f675195f9f6808"
Sep 17 17:43:51.989424 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:51.989379    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7511b8b04e3e4f2af3fcf56a9a3d74165918ab6766ad909b49f675195f9f6808"} err="failed to get container status \"7511b8b04e3e4f2af3fcf56a9a3d74165918ab6766ad909b49f675195f9f6808\": rpc error: code = NotFound desc = could not find container \"7511b8b04e3e4f2af3fcf56a9a3d74165918ab6766ad909b49f675195f9f6808\": container with ID starting with 7511b8b04e3e4f2af3fcf56a9a3d74165918ab6766ad909b49f675195f9f6808 not found: ID does not exist"
Sep 17 17:43:51.992565 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:51.992549    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7f654675b-7vhmc"
Sep 17 17:43:52.027557 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:52.027521    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5d9d3373-15fd-4653-a148-95412f37d3c4-cert\") pod \"5d9d3373-15fd-4653-a148-95412f37d3c4\" (UID: \"5d9d3373-15fd-4653-a148-95412f37d3c4\") "
Sep 17 17:43:52.027679 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:52.027628    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8pt9x\" (UniqueName: \"kubernetes.io/projected/5d9d3373-15fd-4653-a148-95412f37d3c4-kube-api-access-8pt9x\") pod \"5d9d3373-15fd-4653-a148-95412f37d3c4\" (UID: \"5d9d3373-15fd-4653-a148-95412f37d3c4\") "
Sep 17 17:43:52.029587 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:52.029561    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d9d3373-15fd-4653-a148-95412f37d3c4-kube-api-access-8pt9x" (OuterVolumeSpecName: "kube-api-access-8pt9x") pod "5d9d3373-15fd-4653-a148-95412f37d3c4" (UID: "5d9d3373-15fd-4653-a148-95412f37d3c4"). InnerVolumeSpecName "kube-api-access-8pt9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:43:52.029587 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:52.029583    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d9d3373-15fd-4653-a148-95412f37d3c4-cert" (OuterVolumeSpecName: "cert") pod "5d9d3373-15fd-4653-a148-95412f37d3c4" (UID: "5d9d3373-15fd-4653-a148-95412f37d3c4"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:43:52.128520 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:52.128478    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pt9x\" (UniqueName: \"kubernetes.io/projected/5d9d3373-15fd-4653-a148-95412f37d3c4-kube-api-access-8pt9x\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 17:43:52.128520 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:52.128519    2585 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5d9d3373-15fd-4653-a148-95412f37d3c4-cert\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 17:43:52.264288 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:52.264262    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7f654675b-7vhmc"]
Sep 17 17:43:52.266737 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:43:52.266714    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8ed622f_3f48_4e28_af3b_fc634c8e52dd.slice/crio-7cd7d2d4a5f83070968bb3afa097e6b78690c5f68232621eaf2f3149b76f0449 WatchSource:0}: Error finding container 7cd7d2d4a5f83070968bb3afa097e6b78690c5f68232621eaf2f3149b76f0449: Status 404 returned error can't find the container with id 7cd7d2d4a5f83070968bb3afa097e6b78690c5f68232621eaf2f3149b76f0449
Sep 17 17:43:52.302131 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:52.302108    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-7f654675b-kgv87"]
Sep 17 17:43:52.304977 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:52.304956    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-7f654675b-kgv87"]
Sep 17 17:43:52.987307 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:52.987222    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7f654675b-7vhmc" event={"ID":"e8ed622f-3f48-4e28-af3b-fc634c8e52dd","Type":"ContainerStarted","Data":"ab8172a1d24545c8bfe3108f2dfd6bc704717d6a8bced43e8677cc0d34a83908"}
Sep 17 17:43:52.987307 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:52.987278    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7f654675b-7vhmc" event={"ID":"e8ed622f-3f48-4e28-af3b-fc634c8e52dd","Type":"ContainerStarted","Data":"7cd7d2d4a5f83070968bb3afa097e6b78690c5f68232621eaf2f3149b76f0449"}
Sep 17 17:43:52.987756 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:52.987368    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-7f654675b-7vhmc"
Sep 17 17:43:53.007582 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:53.007541    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-7f654675b-7vhmc" podStartSLOduration=1.5706279890000001 podStartE2EDuration="2.00752828s" totalImagesPullingTime="436.900291ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:43:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:43:53.006372888 +0000 UTC m=+569.420163325" watchObservedRunningTime="2026-09-17 17:43:53.00752828 +0000 UTC m=+569.421318714"
Sep 17 17:43:54.107431 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:54.107398    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d9d3373-15fd-4653-a148-95412f37d3c4" path="/var/lib/kubelet/pods/5d9d3373-15fd-4653-a148-95412f37d3c4/volumes"
Sep 17 17:43:56.843588 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:43:56.843533    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve/kserve-controller-manager-7f654675b-kgv87" podUID="5d9d3373-15fd-4653-a148-95412f37d3c4" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.15:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 17 17:44:02.679294 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:44:02.679253    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:23.996397 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:44:23.996367    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-7f654675b-7vhmc"
Sep 17 17:44:24.022282 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:44:24.022258    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rrf79_beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5/ovn-acl-logging/0.log"
Sep 17 17:44:24.033326 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:44:24.033303    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rrf79_beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5/ovn-acl-logging/0.log"
Sep 17 17:44:25.793882 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:44:25.793852    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:25.897986 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:44:25.897957    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:26.005740 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:44:26.005708    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:26.227293 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:44:26.227204    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:27.665934 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:44:27.665904    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:44:27.742111 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:44:27.742072    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:45:43.200965 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:45:43.200930    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:00.243534 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:46:00.243502    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-zj64n"]
Sep 17 17:46:00.243971 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:46:00.243851    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d9d3373-15fd-4653-a148-95412f37d3c4" containerName="manager"
Sep 17 17:46:00.243971 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:46:00.243867    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d9d3373-15fd-4653-a148-95412f37d3c4" containerName="manager"
Sep 17 17:46:00.243971 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:46:00.243933    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d9d3373-15fd-4653-a148-95412f37d3c4" containerName="manager"
Sep 17 17:46:00.246648 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:46:00.246629    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-zj64n"
Sep 17 17:46:00.248694 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:46:00.248669    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:00.249113 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:46:00.249093    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 17 17:46:00.249254 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:46:00.249236    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-kcx4l\""
Sep 17 17:46:00.261623 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:46:00.261605    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-zj64n"]
Sep 17 17:46:00.384948 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:46:00.384914    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqhkg\" (UniqueName: \"kubernetes.io/projected/57c77258-1663-4c39-82b7-f5672422f6af-kube-api-access-gqhkg\") pod \"seaweedfs-tls-custom-6644bf5d6c-zj64n\" (UID: \"57c77258-1663-4c39-82b7-f5672422f6af\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-zj64n"
Sep 17 17:46:00.385110 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:46:00.384976    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/57c77258-1663-4c39-82b7-f5672422f6af-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-zj64n\" (UID: \"57c77258-1663-4c39-82b7-f5672422f6af\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-zj64n"
Sep 17 17:46:00.486174 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:46:00.486140    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqhkg\" (UniqueName: \"kubernetes.io/projected/57c77258-1663-4c39-82b7-f5672422f6af-kube-api-access-gqhkg\") pod \"seaweedfs-tls-custom-6644bf5d6c-zj64n\" (UID: \"57c77258-1663-4c39-82b7-f5672422f6af\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-zj64n"
Sep 17 17:46:00.486349 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:46:00.486201    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/57c77258-1663-4c39-82b7-f5672422f6af-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-zj64n\" (UID: \"57c77258-1663-4c39-82b7-f5672422f6af\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-zj64n"
Sep 17 17:46:00.486576 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:46:00.486556    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/57c77258-1663-4c39-82b7-f5672422f6af-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-zj64n\" (UID: \"57c77258-1663-4c39-82b7-f5672422f6af\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-zj64n"
Sep 17 17:46:00.496043 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:46:00.495988    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqhkg\" (UniqueName: \"kubernetes.io/projected/57c77258-1663-4c39-82b7-f5672422f6af-kube-api-access-gqhkg\") pod \"seaweedfs-tls-custom-6644bf5d6c-zj64n\" (UID: \"57c77258-1663-4c39-82b7-f5672422f6af\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-zj64n"
Sep 17 17:46:00.555032 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:46:00.555005    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-zj64n"
Sep 17 17:46:00.831230 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:46:00.831150    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-zj64n"]
Sep 17 17:46:00.833977 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:46:00.833941    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57c77258_1663_4c39_82b7_f5672422f6af.slice/crio-9f123326577bbb9e6ae38e997ed860115fcedf050969b91ca2cf8e091a9c6686 WatchSource:0}: Error finding container 9f123326577bbb9e6ae38e997ed860115fcedf050969b91ca2cf8e091a9c6686: Status 404 returned error can't find the container with id 9f123326577bbb9e6ae38e997ed860115fcedf050969b91ca2cf8e091a9c6686
Sep 17 17:46:01.417398 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:46:01.417361    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-zj64n" event={"ID":"57c77258-1663-4c39-82b7-f5672422f6af","Type":"ContainerStarted","Data":"9f123326577bbb9e6ae38e997ed860115fcedf050969b91ca2cf8e091a9c6686"}
Sep 17 17:46:04.430364 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:46:04.430325    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-zj64n" event={"ID":"57c77258-1663-4c39-82b7-f5672422f6af","Type":"ContainerStarted","Data":"c5d157dd23ab82e574fc9b54b9e1c7b56630e619b80f743547b94cdfd503ba97"}
Sep 17 17:46:04.451407 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:46:04.451357    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-zj64n" podStartSLOduration=1.794433911 podStartE2EDuration="4.451340808s" totalImagesPullingTime="2.656906897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:46:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:46:04.450319187 +0000 UTC m=+700.864109626" watchObservedRunningTime="2026-09-17 17:46:04.451340808 +0000 UTC m=+700.865131246"
Sep 17 17:46:06.153121 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:46:06.153087    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-zj64n"]
Sep 17 17:46:06.437022 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:46:06.436938    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-zj64n" podUID="57c77258-1663-4c39-82b7-f5672422f6af" containerName="seaweedfs-tls-custom" containerID="cri-o://c5d157dd23ab82e574fc9b54b9e1c7b56630e619b80f743547b94cdfd503ba97" gracePeriod=30
Sep 17 17:46:07.669110 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:46:07.669089    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-zj64n"
Sep 17 17:46:07.837567 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:46:07.837539    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqhkg\" (UniqueName: \"kubernetes.io/projected/57c77258-1663-4c39-82b7-f5672422f6af-kube-api-access-gqhkg\") pod \"57c77258-1663-4c39-82b7-f5672422f6af\" (UID: \"57c77258-1663-4c39-82b7-f5672422f6af\") "
Sep 17 17:46:07.837720 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:46:07.837574    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/57c77258-1663-4c39-82b7-f5672422f6af-data\") pod \"57c77258-1663-4c39-82b7-f5672422f6af\" (UID: \"57c77258-1663-4c39-82b7-f5672422f6af\") "
Sep 17 17:46:07.838767 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:46:07.838736    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57c77258-1663-4c39-82b7-f5672422f6af-data" (OuterVolumeSpecName: "data") pod "57c77258-1663-4c39-82b7-f5672422f6af" (UID: "57c77258-1663-4c39-82b7-f5672422f6af"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:46:07.839559 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:46:07.839540    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57c77258-1663-4c39-82b7-f5672422f6af-kube-api-access-gqhkg" (OuterVolumeSpecName: "kube-api-access-gqhkg") pod "57c77258-1663-4c39-82b7-f5672422f6af" (UID: "57c77258-1663-4c39-82b7-f5672422f6af"). InnerVolumeSpecName "kube-api-access-gqhkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:46:07.938008 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:46:07.937973    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqhkg\" (UniqueName: \"kubernetes.io/projected/57c77258-1663-4c39-82b7-f5672422f6af-kube-api-access-gqhkg\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 17:46:07.938008 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:46:07.938001    2585 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/57c77258-1663-4c39-82b7-f5672422f6af-data\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 17:46:08.444575 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:46:08.444480    2585 generic.go:356] "Generic (PLEG): container finished" podID="57c77258-1663-4c39-82b7-f5672422f6af" containerID="c5d157dd23ab82e574fc9b54b9e1c7b56630e619b80f743547b94cdfd503ba97" exitCode=0
Sep 17 17:46:08.444575 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:46:08.444535    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-zj64n"
Sep 17 17:46:08.444575 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:46:08.444573    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-zj64n" event={"ID":"57c77258-1663-4c39-82b7-f5672422f6af","Type":"ContainerDied","Data":"c5d157dd23ab82e574fc9b54b9e1c7b56630e619b80f743547b94cdfd503ba97"}
Sep 17 17:46:08.444798 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:46:08.444614    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-zj64n" event={"ID":"57c77258-1663-4c39-82b7-f5672422f6af","Type":"ContainerDied","Data":"9f123326577bbb9e6ae38e997ed860115fcedf050969b91ca2cf8e091a9c6686"}
Sep 17 17:46:08.444798 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:46:08.444635    2585 scope.go:117] "RemoveContainer" containerID="c5d157dd23ab82e574fc9b54b9e1c7b56630e619b80f743547b94cdfd503ba97"
Sep 17 17:46:08.453252 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:46:08.453235    2585 scope.go:117] "RemoveContainer" containerID="c5d157dd23ab82e574fc9b54b9e1c7b56630e619b80f743547b94cdfd503ba97"
Sep 17 17:46:08.453593 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:46:08.453575    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5d157dd23ab82e574fc9b54b9e1c7b56630e619b80f743547b94cdfd503ba97\": container with ID starting with c5d157dd23ab82e574fc9b54b9e1c7b56630e619b80f743547b94cdfd503ba97 not found: ID does not exist" containerID="c5d157dd23ab82e574fc9b54b9e1c7b56630e619b80f743547b94cdfd503ba97"
Sep 17 17:46:08.453648 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:46:08.453601    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5d157dd23ab82e574fc9b54b9e1c7b56630e619b80f743547b94cdfd503ba97"} err="failed to get container status \"c5d157dd23ab82e574fc9b54b9e1c7b56630e619b80f743547b94cdfd503ba97\": rpc error: code = NotFound desc = could not find container \"c5d157dd23ab82e574fc9b54b9e1c7b56630e619b80f743547b94cdfd503ba97\": container with ID starting with c5d157dd23ab82e574fc9b54b9e1c7b56630e619b80f743547b94cdfd503ba97 not found: ID does not exist"
Sep 17 17:46:08.461632 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:46:08.461612    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-zj64n"]
Sep 17 17:46:08.466320 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:46:08.466302    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-zj64n"]
Sep 17 17:46:10.103421 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:46:10.103387    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57c77258-1663-4c39-82b7-f5672422f6af" path="/var/lib/kubelet/pods/57c77258-1663-4c39-82b7-f5672422f6af/volumes"
Sep 17 17:46:21.261328 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:46:21.261252    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:21.387882 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:46:21.387846    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:37.076582 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:46:37.076546    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:37.177551 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:46:37.177520    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:48.924259 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:46:48.924224    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:50.183627 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:46:50.183595    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:46:58.841235 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:46:58.841204    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:03.977655 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:47:03.977600    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:04.017252 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:47:04.017221    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-q6kv2"]
Sep 17 17:47:04.017469 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:47:04.017456    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57c77258-1663-4c39-82b7-f5672422f6af" containerName="seaweedfs-tls-custom"
Sep 17 17:47:04.017528 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:47:04.017471    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="57c77258-1663-4c39-82b7-f5672422f6af" containerName="seaweedfs-tls-custom"
Sep 17 17:47:04.017562 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:47:04.017529    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="57c77258-1663-4c39-82b7-f5672422f6af" containerName="seaweedfs-tls-custom"
Sep 17 17:47:04.020352 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:47:04.020333    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-q6kv2"
Sep 17 17:47:04.022879 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:47:04.022860    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:47:04.022879 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:47:04.022870    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:47:04.023060 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:47:04.022865    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
Sep 17 17:47:04.023060 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:47:04.022927    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
Sep 17 17:47:04.023060 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:47:04.023000    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:47:04.023227 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:47:04.023208    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:47:04.023336 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:47:04.023314    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lps5w\""
Sep 17 17:47:04.032690 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:47:04.032671    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-q6kv2"]
Sep 17 17:47:04.132008 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:47:04.131974    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22b2098e-81e3-4873-ace4-7239e2071884-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-q6kv2\" (UID: \"22b2098e-81e3-4873-ace4-7239e2071884\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-q6kv2"
Sep 17 17:47:04.132164 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:47:04.132016    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22b2098e-81e3-4873-ace4-7239e2071884-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-q6kv2\" (UID: \"22b2098e-81e3-4873-ace4-7239e2071884\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-q6kv2"
Sep 17 17:47:04.132164 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:47:04.132072    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22b2098e-81e3-4873-ace4-7239e2071884-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-q6kv2\" (UID: \"22b2098e-81e3-4873-ace4-7239e2071884\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-q6kv2"
Sep 17 17:47:04.132258 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:47:04.132161    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5stm\" (UniqueName: \"kubernetes.io/projected/22b2098e-81e3-4873-ace4-7239e2071884-kube-api-access-v5stm\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-q6kv2\" (UID: \"22b2098e-81e3-4873-ace4-7239e2071884\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-q6kv2"
Sep 17 17:47:04.132258 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:47:04.132184    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22b2098e-81e3-4873-ace4-7239e2071884-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-q6kv2\" (UID: \"22b2098e-81e3-4873-ace4-7239e2071884\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-q6kv2"
Sep 17 17:47:04.233382 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:47:04.233301    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5stm\" (UniqueName: \"kubernetes.io/projected/22b2098e-81e3-4873-ace4-7239e2071884-kube-api-access-v5stm\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-q6kv2\" (UID: \"22b2098e-81e3-4873-ace4-7239e2071884\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-q6kv2"
Sep 17 17:47:04.233382 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:47:04.233341    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22b2098e-81e3-4873-ace4-7239e2071884-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-q6kv2\" (UID: \"22b2098e-81e3-4873-ace4-7239e2071884\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-q6kv2"
Sep 17 17:47:04.233382 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:47:04.233367    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22b2098e-81e3-4873-ace4-7239e2071884-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-q6kv2\" (UID: \"22b2098e-81e3-4873-ace4-7239e2071884\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-q6kv2"
Sep 17 17:47:04.233737 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:47:04.233426    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22b2098e-81e3-4873-ace4-7239e2071884-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-q6kv2\" (UID: \"22b2098e-81e3-4873-ace4-7239e2071884\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-q6kv2"
Sep 17 17:47:04.233737 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:47:04.233460    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22b2098e-81e3-4873-ace4-7239e2071884-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-q6kv2\" (UID: \"22b2098e-81e3-4873-ace4-7239e2071884\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-q6kv2"
Sep 17 17:47:04.233986 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:47:04.233965    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22b2098e-81e3-4873-ace4-7239e2071884-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-q6kv2\" (UID: \"22b2098e-81e3-4873-ace4-7239e2071884\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-q6kv2"
Sep 17 17:47:04.234071 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:47:04.234054    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22b2098e-81e3-4873-ace4-7239e2071884-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-q6kv2\" (UID: \"22b2098e-81e3-4873-ace4-7239e2071884\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-q6kv2"
Sep 17 17:47:04.234182 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:47:04.234163    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22b2098e-81e3-4873-ace4-7239e2071884-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-q6kv2\" (UID: \"22b2098e-81e3-4873-ace4-7239e2071884\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-q6kv2"
Sep 17 17:47:04.235934 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:47:04.235911    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22b2098e-81e3-4873-ace4-7239e2071884-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-q6kv2\" (UID: \"22b2098e-81e3-4873-ace4-7239e2071884\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-q6kv2"
Sep 17 17:47:04.246300 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:47:04.246272    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5stm\" (UniqueName: \"kubernetes.io/projected/22b2098e-81e3-4873-ace4-7239e2071884-kube-api-access-v5stm\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-q6kv2\" (UID: \"22b2098e-81e3-4873-ace4-7239e2071884\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-q6kv2"
Sep 17 17:47:04.254130 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:47:04.254108    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:04.329997 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:47:04.329967    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-q6kv2"
Sep 17 17:47:04.421451 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:47:04.421417    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:04.600803 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:47:04.600771    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-q6kv2"]
Sep 17 17:47:04.604089 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:47:04.604063    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22b2098e_81e3_4873_ace4_7239e2071884.slice/crio-911bdc0c3894b9d322097c88657952893db5ccceb83ff97a4569d0fceb15c4a8 WatchSource:0}: Error finding container 911bdc0c3894b9d322097c88657952893db5ccceb83ff97a4569d0fceb15c4a8: Status 404 returned error can't find the container with id 911bdc0c3894b9d322097c88657952893db5ccceb83ff97a4569d0fceb15c4a8
Sep 17 17:47:04.606691 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:47:04.606679    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:47:04.633235 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:47:04.633208    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-q6kv2" event={"ID":"22b2098e-81e3-4873-ace4-7239e2071884","Type":"ContainerStarted","Data":"911bdc0c3894b9d322097c88657952893db5ccceb83ff97a4569d0fceb15c4a8"}
Sep 17 17:47:08.651140 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:47:08.651102    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-q6kv2" event={"ID":"22b2098e-81e3-4873-ace4-7239e2071884","Type":"ContainerStarted","Data":"5144b827f49707ca9f604ad9ed62c0415fb257003fc54697b88c08db70c1ada5"}
Sep 17 17:47:11.662952 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:47:11.662921    2585 generic.go:356] "Generic (PLEG): container finished" podID="22b2098e-81e3-4873-ace4-7239e2071884" containerID="5144b827f49707ca9f604ad9ed62c0415fb257003fc54697b88c08db70c1ada5" exitCode=0
Sep 17 17:47:11.663238 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:47:11.662979    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-q6kv2" event={"ID":"22b2098e-81e3-4873-ace4-7239e2071884","Type":"ContainerDied","Data":"5144b827f49707ca9f604ad9ed62c0415fb257003fc54697b88c08db70c1ada5"}
Sep 17 17:47:34.766273 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:47:34.766239    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-q6kv2" event={"ID":"22b2098e-81e3-4873-ace4-7239e2071884","Type":"ContainerStarted","Data":"05b305834d39db84a9c15a16a53c937dfcf7e765b27293739a978d44a71625b2"}
Sep 17 17:47:37.779971 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:47:37.779933    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-q6kv2" event={"ID":"22b2098e-81e3-4873-ace4-7239e2071884","Type":"ContainerStarted","Data":"8866ac807b4faa6e517c932cf461ba2584d6882feaea0373a297debeddb402a9"}
Sep 17 17:47:37.780352 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:47:37.780067    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-q6kv2"
Sep 17 17:47:37.801175 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:47:37.801020    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-q6kv2" podStartSLOduration=2.4064883950000002 podStartE2EDuration="34.800985961s" totalImagesPullingTime="28.696142611s" totalInitContainerRuntime="3.698354955s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:47:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:47:37.800296718 +0000 UTC m=+794.214087182" watchObservedRunningTime="2026-09-17 17:47:37.800985961 +0000 UTC m=+794.214776400"
Sep 17 17:47:38.783944 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:47:38.783916    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-q6kv2"
Sep 17 17:47:38.785111 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:47:38.785086    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-q6kv2" podUID="22b2098e-81e3-4873-ace4-7239e2071884" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Sep 17 17:47:39.787061 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:47:39.787024    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-q6kv2" podUID="22b2098e-81e3-4873-ace4-7239e2071884" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Sep 17 17:47:40.010654 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:47:40.010623    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:47:44.791473 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:47:44.791445    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-q6kv2"
Sep 17 17:47:44.791955 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:47:44.791932    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-q6kv2" podUID="22b2098e-81e3-4873-ace4-7239e2071884" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Sep 17 17:47:54.792970 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:47:54.792893    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-q6kv2" podUID="22b2098e-81e3-4873-ace4-7239e2071884" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Sep 17 17:48:04.792124 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:48:04.792085    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-q6kv2" podUID="22b2098e-81e3-4873-ace4-7239e2071884" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Sep 17 17:48:14.792803 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:48:14.792766    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-q6kv2" podUID="22b2098e-81e3-4873-ace4-7239e2071884" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Sep 17 17:48:24.792632 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:48:24.792595    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-q6kv2" podUID="22b2098e-81e3-4873-ace4-7239e2071884" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Sep 17 17:48:33.989801 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:48:33.989762    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:34.201485 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:48:34.201427    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:34.300987 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:48:34.300959    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:48:34.792056 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:48:34.792025    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-q6kv2" podUID="22b2098e-81e3-4873-ace4-7239e2071884" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Sep 17 17:48:44.792867 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:48:44.792823    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-q6kv2" podUID="22b2098e-81e3-4873-ace4-7239e2071884" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Sep 17 17:48:54.793327 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:48:54.793297    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-q6kv2"
Sep 17 17:49:04.629689 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:04.629651    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-q6kv2"]
Sep 17 17:49:04.630109 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:04.629879    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-q6kv2" podUID="22b2098e-81e3-4873-ace4-7239e2071884" containerName="kserve-container" containerID="cri-o://05b305834d39db84a9c15a16a53c937dfcf7e765b27293739a978d44a71625b2" gracePeriod=30
Sep 17 17:49:04.630109 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:04.629946    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-q6kv2" podUID="22b2098e-81e3-4873-ace4-7239e2071884" containerName="kube-rbac-proxy" containerID="cri-o://8866ac807b4faa6e517c932cf461ba2584d6882feaea0373a297debeddb402a9" gracePeriod=30
Sep 17 17:49:04.706336 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:04.706275    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:04.788229 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:04.788194    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-q6kv2" podUID="22b2098e-81e3-4873-ace4-7239e2071884" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.18:8643/healthz\": dial tcp 10.133.0.18:8643: connect: connection refused"
Sep 17 17:49:04.792559 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:04.792540    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-q6kv2" podUID="22b2098e-81e3-4873-ace4-7239e2071884" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.18:8080: connect: connection refused"
Sep 17 17:49:05.088623 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:05.088590    2585 generic.go:356] "Generic (PLEG): container finished" podID="22b2098e-81e3-4873-ace4-7239e2071884" containerID="8866ac807b4faa6e517c932cf461ba2584d6882feaea0373a297debeddb402a9" exitCode=2
Sep 17 17:49:05.088785 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:05.088665    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-q6kv2" event={"ID":"22b2098e-81e3-4873-ace4-7239e2071884","Type":"ContainerDied","Data":"8866ac807b4faa6e517c932cf461ba2584d6882feaea0373a297debeddb402a9"}
Sep 17 17:49:09.368787 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:09.368761    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-q6kv2"
Sep 17 17:49:09.439089 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:09.439062    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22b2098e-81e3-4873-ace4-7239e2071884-cabundle-cert\") pod \"22b2098e-81e3-4873-ace4-7239e2071884\" (UID: \"22b2098e-81e3-4873-ace4-7239e2071884\") "
Sep 17 17:49:09.439089 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:09.439093    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22b2098e-81e3-4873-ace4-7239e2071884-kube-rbac-proxy-sar-config\") pod \"22b2098e-81e3-4873-ace4-7239e2071884\" (UID: \"22b2098e-81e3-4873-ace4-7239e2071884\") "
Sep 17 17:49:09.439281 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:09.439126    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22b2098e-81e3-4873-ace4-7239e2071884-proxy-tls\") pod \"22b2098e-81e3-4873-ace4-7239e2071884\" (UID: \"22b2098e-81e3-4873-ace4-7239e2071884\") "
Sep 17 17:49:09.439281 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:09.439167    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22b2098e-81e3-4873-ace4-7239e2071884-kserve-provision-location\") pod \"22b2098e-81e3-4873-ace4-7239e2071884\" (UID: \"22b2098e-81e3-4873-ace4-7239e2071884\") "
Sep 17 17:49:09.439281 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:09.439193    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5stm\" (UniqueName: \"kubernetes.io/projected/22b2098e-81e3-4873-ace4-7239e2071884-kube-api-access-v5stm\") pod \"22b2098e-81e3-4873-ace4-7239e2071884\" (UID: \"22b2098e-81e3-4873-ace4-7239e2071884\") "
Sep 17 17:49:09.439508 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:09.439458    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22b2098e-81e3-4873-ace4-7239e2071884-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "22b2098e-81e3-4873-ace4-7239e2071884" (UID: "22b2098e-81e3-4873-ace4-7239e2071884"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:49:09.439587 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:09.439466    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22b2098e-81e3-4873-ace4-7239e2071884-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "22b2098e-81e3-4873-ace4-7239e2071884" (UID: "22b2098e-81e3-4873-ace4-7239e2071884"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:49:09.439684 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:09.439659    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22b2098e-81e3-4873-ace4-7239e2071884-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "22b2098e-81e3-4873-ace4-7239e2071884" (UID: "22b2098e-81e3-4873-ace4-7239e2071884"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:49:09.441194 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:09.441169    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22b2098e-81e3-4873-ace4-7239e2071884-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "22b2098e-81e3-4873-ace4-7239e2071884" (UID: "22b2098e-81e3-4873-ace4-7239e2071884"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:49:09.441287 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:09.441261    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22b2098e-81e3-4873-ace4-7239e2071884-kube-api-access-v5stm" (OuterVolumeSpecName: "kube-api-access-v5stm") pod "22b2098e-81e3-4873-ace4-7239e2071884" (UID: "22b2098e-81e3-4873-ace4-7239e2071884"). InnerVolumeSpecName "kube-api-access-v5stm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:49:09.539930 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:09.539901    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/22b2098e-81e3-4873-ace4-7239e2071884-cabundle-cert\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 17:49:09.540041 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:09.539931    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/22b2098e-81e3-4873-ace4-7239e2071884-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 17:49:09.540041 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:09.539945    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/22b2098e-81e3-4873-ace4-7239e2071884-proxy-tls\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 17:49:09.540041 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:09.539956    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22b2098e-81e3-4873-ace4-7239e2071884-kserve-provision-location\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 17:49:09.540041 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:09.539967    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5stm\" (UniqueName: \"kubernetes.io/projected/22b2098e-81e3-4873-ace4-7239e2071884-kube-api-access-v5stm\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 17:49:09.782146 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:09.782111    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:09.820815 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:09.820752    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gjcgr"]
Sep 17 17:49:09.820993 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:09.820982    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22b2098e-81e3-4873-ace4-7239e2071884" containerName="storage-initializer"
Sep 17 17:49:09.821032 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:09.820996    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="22b2098e-81e3-4873-ace4-7239e2071884" containerName="storage-initializer"
Sep 17 17:49:09.821032 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:09.821006    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22b2098e-81e3-4873-ace4-7239e2071884" containerName="kserve-container"
Sep 17 17:49:09.821032 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:09.821012    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="22b2098e-81e3-4873-ace4-7239e2071884" containerName="kserve-container"
Sep 17 17:49:09.821032 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:09.821019    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22b2098e-81e3-4873-ace4-7239e2071884" containerName="kube-rbac-proxy"
Sep 17 17:49:09.821032 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:09.821024    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="22b2098e-81e3-4873-ace4-7239e2071884" containerName="kube-rbac-proxy"
Sep 17 17:49:09.821169 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:09.821069    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="22b2098e-81e3-4873-ace4-7239e2071884" containerName="kube-rbac-proxy"
Sep 17 17:49:09.821169 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:09.821077    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="22b2098e-81e3-4873-ace4-7239e2071884" containerName="kserve-container"
Sep 17 17:49:09.824136 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:09.824121    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gjcgr"
Sep 17 17:49:09.827021 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:09.827001    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
Sep 17 17:49:09.827225 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:09.827212    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
Sep 17 17:49:09.841931 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:09.841904    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4587ada0-48d9-4478-abfc-5d90bfb320de-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-gjcgr\" (UID: \"4587ada0-48d9-4478-abfc-5d90bfb320de\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gjcgr"
Sep 17 17:49:09.842014 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:09.841948    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4587ada0-48d9-4478-abfc-5d90bfb320de-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-gjcgr\" (UID: \"4587ada0-48d9-4478-abfc-5d90bfb320de\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gjcgr"
Sep 17 17:49:09.842014 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:09.841968    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4587ada0-48d9-4478-abfc-5d90bfb320de-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-gjcgr\" (UID: \"4587ada0-48d9-4478-abfc-5d90bfb320de\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gjcgr"
Sep 17 17:49:09.842093 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:09.842029    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4587ada0-48d9-4478-abfc-5d90bfb320de-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-gjcgr\" (UID: \"4587ada0-48d9-4478-abfc-5d90bfb320de\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gjcgr"
Sep 17 17:49:09.842093 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:09.842055    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbqdt\" (UniqueName: \"kubernetes.io/projected/4587ada0-48d9-4478-abfc-5d90bfb320de-kube-api-access-hbqdt\") pod \"isvc-predictive-xgboost-predictor-695f978978-gjcgr\" (UID: \"4587ada0-48d9-4478-abfc-5d90bfb320de\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gjcgr"
Sep 17 17:49:09.844996 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:09.844976    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gjcgr"]
Sep 17 17:49:09.910441 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:09.910411    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:49:09.942738 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:09.942710    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4587ada0-48d9-4478-abfc-5d90bfb320de-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-gjcgr\" (UID: \"4587ada0-48d9-4478-abfc-5d90bfb320de\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gjcgr"
Sep 17 17:49:09.942882 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:09.942743    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4587ada0-48d9-4478-abfc-5d90bfb320de-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-gjcgr\" (UID: \"4587ada0-48d9-4478-abfc-5d90bfb320de\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gjcgr"
Sep 17 17:49:09.942882 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:09.942765    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4587ada0-48d9-4478-abfc-5d90bfb320de-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-gjcgr\" (UID: \"4587ada0-48d9-4478-abfc-5d90bfb320de\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gjcgr"
Sep 17 17:49:09.942882 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:09.942784    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbqdt\" (UniqueName: \"kubernetes.io/projected/4587ada0-48d9-4478-abfc-5d90bfb320de-kube-api-access-hbqdt\") pod \"isvc-predictive-xgboost-predictor-695f978978-gjcgr\" (UID: \"4587ada0-48d9-4478-abfc-5d90bfb320de\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gjcgr"
Sep 17 17:49:09.942882 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:09.942803    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4587ada0-48d9-4478-abfc-5d90bfb320de-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-gjcgr\" (UID: \"4587ada0-48d9-4478-abfc-5d90bfb320de\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gjcgr"
Sep 17 17:49:09.943185 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:09.943167    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4587ada0-48d9-4478-abfc-5d90bfb320de-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-gjcgr\" (UID: \"4587ada0-48d9-4478-abfc-5d90bfb320de\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gjcgr"
Sep 17 17:49:09.943476 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:09.943454    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4587ada0-48d9-4478-abfc-5d90bfb320de-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-gjcgr\" (UID: \"4587ada0-48d9-4478-abfc-5d90bfb320de\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gjcgr"
Sep 17 17:49:09.943573 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:09.943529    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4587ada0-48d9-4478-abfc-5d90bfb320de-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-gjcgr\" (UID: \"4587ada0-48d9-4478-abfc-5d90bfb320de\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gjcgr"
Sep 17 17:49:09.945237 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:09.945219    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4587ada0-48d9-4478-abfc-5d90bfb320de-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-gjcgr\" (UID: \"4587ada0-48d9-4478-abfc-5d90bfb320de\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gjcgr"
Sep 17 17:49:09.952383 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:09.952364    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbqdt\" (UniqueName: \"kubernetes.io/projected/4587ada0-48d9-4478-abfc-5d90bfb320de-kube-api-access-hbqdt\") pod \"isvc-predictive-xgboost-predictor-695f978978-gjcgr\" (UID: \"4587ada0-48d9-4478-abfc-5d90bfb320de\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gjcgr"
Sep 17 17:49:10.109146 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:10.109073    2585 generic.go:356] "Generic (PLEG): container finished" podID="22b2098e-81e3-4873-ace4-7239e2071884" containerID="05b305834d39db84a9c15a16a53c937dfcf7e765b27293739a978d44a71625b2" exitCode=0
Sep 17 17:49:10.109146 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:10.109122    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-q6kv2"
Sep 17 17:49:10.109308 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:10.109156    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-q6kv2" event={"ID":"22b2098e-81e3-4873-ace4-7239e2071884","Type":"ContainerDied","Data":"05b305834d39db84a9c15a16a53c937dfcf7e765b27293739a978d44a71625b2"}
Sep 17 17:49:10.109308 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:10.109216    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-q6kv2" event={"ID":"22b2098e-81e3-4873-ace4-7239e2071884","Type":"ContainerDied","Data":"911bdc0c3894b9d322097c88657952893db5ccceb83ff97a4569d0fceb15c4a8"}
Sep 17 17:49:10.109308 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:10.109254    2585 scope.go:117] "RemoveContainer" containerID="8866ac807b4faa6e517c932cf461ba2584d6882feaea0373a297debeddb402a9"
Sep 17 17:49:10.117232 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:10.117217    2585 scope.go:117] "RemoveContainer" containerID="05b305834d39db84a9c15a16a53c937dfcf7e765b27293739a978d44a71625b2"
Sep 17 17:49:10.124275 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:10.124260    2585 scope.go:117] "RemoveContainer" containerID="5144b827f49707ca9f604ad9ed62c0415fb257003fc54697b88c08db70c1ada5"
Sep 17 17:49:10.130927 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:10.130909    2585 scope.go:117] "RemoveContainer" containerID="8866ac807b4faa6e517c932cf461ba2584d6882feaea0373a297debeddb402a9"
Sep 17 17:49:10.131182 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:49:10.131160    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8866ac807b4faa6e517c932cf461ba2584d6882feaea0373a297debeddb402a9\": container with ID starting with 8866ac807b4faa6e517c932cf461ba2584d6882feaea0373a297debeddb402a9 not found: ID does not exist" containerID="8866ac807b4faa6e517c932cf461ba2584d6882feaea0373a297debeddb402a9"
Sep 17 17:49:10.131247 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:10.131189    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8866ac807b4faa6e517c932cf461ba2584d6882feaea0373a297debeddb402a9"} err="failed to get container status \"8866ac807b4faa6e517c932cf461ba2584d6882feaea0373a297debeddb402a9\": rpc error: code = NotFound desc = could not find container \"8866ac807b4faa6e517c932cf461ba2584d6882feaea0373a297debeddb402a9\": container with ID starting with 8866ac807b4faa6e517c932cf461ba2584d6882feaea0373a297debeddb402a9 not found: ID does not exist"
Sep 17 17:49:10.131247 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:10.131205    2585 scope.go:117] "RemoveContainer" containerID="05b305834d39db84a9c15a16a53c937dfcf7e765b27293739a978d44a71625b2"
Sep 17 17:49:10.131460 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:49:10.131440    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05b305834d39db84a9c15a16a53c937dfcf7e765b27293739a978d44a71625b2\": container with ID starting with 05b305834d39db84a9c15a16a53c937dfcf7e765b27293739a978d44a71625b2 not found: ID does not exist" containerID="05b305834d39db84a9c15a16a53c937dfcf7e765b27293739a978d44a71625b2"
Sep 17 17:49:10.131521 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:10.131470    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05b305834d39db84a9c15a16a53c937dfcf7e765b27293739a978d44a71625b2"} err="failed to get container status \"05b305834d39db84a9c15a16a53c937dfcf7e765b27293739a978d44a71625b2\": rpc error: code = NotFound desc = could not find container \"05b305834d39db84a9c15a16a53c937dfcf7e765b27293739a978d44a71625b2\": container with ID starting with 05b305834d39db84a9c15a16a53c937dfcf7e765b27293739a978d44a71625b2 not found: ID does not exist"
Sep 17 17:49:10.131521 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:10.131505    2585 scope.go:117] "RemoveContainer" containerID="5144b827f49707ca9f604ad9ed62c0415fb257003fc54697b88c08db70c1ada5"
Sep 17 17:49:10.131764 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:49:10.131749    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5144b827f49707ca9f604ad9ed62c0415fb257003fc54697b88c08db70c1ada5\": container with ID starting with 5144b827f49707ca9f604ad9ed62c0415fb257003fc54697b88c08db70c1ada5 not found: ID does not exist" containerID="5144b827f49707ca9f604ad9ed62c0415fb257003fc54697b88c08db70c1ada5"
Sep 17 17:49:10.131811 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:10.131769    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5144b827f49707ca9f604ad9ed62c0415fb257003fc54697b88c08db70c1ada5"} err="failed to get container status \"5144b827f49707ca9f604ad9ed62c0415fb257003fc54697b88c08db70c1ada5\": rpc error: code = NotFound desc = could not find container \"5144b827f49707ca9f604ad9ed62c0415fb257003fc54697b88c08db70c1ada5\": container with ID starting with 5144b827f49707ca9f604ad9ed62c0415fb257003fc54697b88c08db70c1ada5 not found: ID does not exist"
Sep 17 17:49:10.131878 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:10.131863    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-q6kv2"]
Sep 17 17:49:10.133274 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:10.133241    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gjcgr"
Sep 17 17:49:10.135050 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:10.135029    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-q6kv2"]
Sep 17 17:49:10.409171 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:10.409147    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gjcgr"]
Sep 17 17:49:10.410214 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:49:10.410194    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4587ada0_48d9_4478_abfc_5d90bfb320de.slice/crio-5d977d9d1f8993af41819d8d2492891d679a2737e53d10774d395f593ad049c2 WatchSource:0}: Error finding container 5d977d9d1f8993af41819d8d2492891d679a2737e53d10774d395f593ad049c2: Status 404 returned error can't find the container with id 5d977d9d1f8993af41819d8d2492891d679a2737e53d10774d395f593ad049c2
Sep 17 17:49:11.120233 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:11.120188    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gjcgr" event={"ID":"4587ada0-48d9-4478-abfc-5d90bfb320de","Type":"ContainerStarted","Data":"2bb7640bd0ef4aee4d599cee99a653a9ef911d87750f8401d3bcb8779acd3eac"}
Sep 17 17:49:11.120437 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:11.120286    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gjcgr" event={"ID":"4587ada0-48d9-4478-abfc-5d90bfb320de","Type":"ContainerStarted","Data":"5d977d9d1f8993af41819d8d2492891d679a2737e53d10774d395f593ad049c2"}
Sep 17 17:49:12.103809 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:12.103775    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22b2098e-81e3-4873-ace4-7239e2071884" path="/var/lib/kubelet/pods/22b2098e-81e3-4873-ace4-7239e2071884/volumes"
Sep 17 17:49:15.135274 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:15.135233    2585 generic.go:356] "Generic (PLEG): container finished" podID="4587ada0-48d9-4478-abfc-5d90bfb320de" containerID="2bb7640bd0ef4aee4d599cee99a653a9ef911d87750f8401d3bcb8779acd3eac" exitCode=0
Sep 17 17:49:15.135789 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:15.135308    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gjcgr" event={"ID":"4587ada0-48d9-4478-abfc-5d90bfb320de","Type":"ContainerDied","Data":"2bb7640bd0ef4aee4d599cee99a653a9ef911d87750f8401d3bcb8779acd3eac"}
Sep 17 17:49:16.140477 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:16.140446    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gjcgr" event={"ID":"4587ada0-48d9-4478-abfc-5d90bfb320de","Type":"ContainerStarted","Data":"06fa150d9fd7e19deadacb67e7b068829a8c4145600d29a83f108a9d26a56db1"}
Sep 17 17:49:16.140934 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:16.140521    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gjcgr" event={"ID":"4587ada0-48d9-4478-abfc-5d90bfb320de","Type":"ContainerStarted","Data":"ba72293fe8fe05ecad3b58013a19c58113eb0bb312503b1c23697be3df6b7986"}
Sep 17 17:49:16.140934 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:16.140795    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gjcgr"
Sep 17 17:49:16.141008 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:16.140959    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gjcgr"
Sep 17 17:49:16.142319 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:16.142291    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gjcgr" podUID="4587ada0-48d9-4478-abfc-5d90bfb320de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Sep 17 17:49:16.165474 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:16.165431    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gjcgr" podStartSLOduration=3.405281201 podStartE2EDuration="7.165417765s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.760136564s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:49:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:49:16.164304542 +0000 UTC m=+892.578094971" watchObservedRunningTime="2026-09-17 17:49:16.165417765 +0000 UTC m=+892.579208203"
Sep 17 17:49:17.144271 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:17.144187    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gjcgr" podUID="4587ada0-48d9-4478-abfc-5d90bfb320de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Sep 17 17:49:22.148963 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:22.148934    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gjcgr"
Sep 17 17:49:22.149442 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:22.149416    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gjcgr" podUID="4587ada0-48d9-4478-abfc-5d90bfb320de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Sep 17 17:49:24.045368 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:24.045339    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rrf79_beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5/ovn-acl-logging/0.log"
Sep 17 17:49:24.054948 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:24.054932    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rrf79_beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5/ovn-acl-logging/0.log"
Sep 17 17:49:32.149960 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:32.149916    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gjcgr" podUID="4587ada0-48d9-4478-abfc-5d90bfb320de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Sep 17 17:49:42.149477 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:42.149436    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gjcgr" podUID="4587ada0-48d9-4478-abfc-5d90bfb320de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Sep 17 17:49:52.149414 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:49:52.149377    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gjcgr" podUID="4587ada0-48d9-4478-abfc-5d90bfb320de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Sep 17 17:50:02.149864 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:02.149826    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gjcgr" podUID="4587ada0-48d9-4478-abfc-5d90bfb320de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Sep 17 17:50:09.186959 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:09.186929    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:50:09.278550 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:09.278517    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zg99"]
Sep 17 17:50:09.281626 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:09.281604    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zg99"
Sep 17 17:50:09.284665 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:09.284647    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Sep 17 17:50:09.284909 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:09.284893    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Sep 17 17:50:09.300769 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:09.300735    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zg99"]
Sep 17 17:50:09.307741 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:09.307719    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:50:09.363823 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:09.363802    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a10431ef-8b50-4ddb-ab94-e06de1a670fd-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-6zg99\" (UID: \"a10431ef-8b50-4ddb-ab94-e06de1a670fd\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zg99"
Sep 17 17:50:09.363946 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:09.363853    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a10431ef-8b50-4ddb-ab94-e06de1a670fd-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-6zg99\" (UID: \"a10431ef-8b50-4ddb-ab94-e06de1a670fd\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zg99"
Sep 17 17:50:09.363946 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:09.363872    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbvj8\" (UniqueName: \"kubernetes.io/projected/a10431ef-8b50-4ddb-ab94-e06de1a670fd-kube-api-access-bbvj8\") pod \"message-dumper-predictor-6575bcfdfd-6zg99\" (UID: \"a10431ef-8b50-4ddb-ab94-e06de1a670fd\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zg99"
Sep 17 17:50:09.465111 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:09.465036    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a10431ef-8b50-4ddb-ab94-e06de1a670fd-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-6zg99\" (UID: \"a10431ef-8b50-4ddb-ab94-e06de1a670fd\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zg99"
Sep 17 17:50:09.465111 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:09.465078    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbvj8\" (UniqueName: \"kubernetes.io/projected/a10431ef-8b50-4ddb-ab94-e06de1a670fd-kube-api-access-bbvj8\") pod \"message-dumper-predictor-6575bcfdfd-6zg99\" (UID: \"a10431ef-8b50-4ddb-ab94-e06de1a670fd\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zg99"
Sep 17 17:50:09.465284 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:09.465123    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a10431ef-8b50-4ddb-ab94-e06de1a670fd-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-6zg99\" (UID: \"a10431ef-8b50-4ddb-ab94-e06de1a670fd\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zg99"
Sep 17 17:50:09.465773 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:09.465754    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a10431ef-8b50-4ddb-ab94-e06de1a670fd-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-6zg99\" (UID: \"a10431ef-8b50-4ddb-ab94-e06de1a670fd\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zg99"
Sep 17 17:50:09.467507 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:09.467477    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a10431ef-8b50-4ddb-ab94-e06de1a670fd-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-6zg99\" (UID: \"a10431ef-8b50-4ddb-ab94-e06de1a670fd\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zg99"
Sep 17 17:50:09.477467 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:09.477449    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbvj8\" (UniqueName: \"kubernetes.io/projected/a10431ef-8b50-4ddb-ab94-e06de1a670fd-kube-api-access-bbvj8\") pod \"message-dumper-predictor-6575bcfdfd-6zg99\" (UID: \"a10431ef-8b50-4ddb-ab94-e06de1a670fd\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zg99"
Sep 17 17:50:09.551541 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:09.551512    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:50:09.590715 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:09.590695    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zg99"
Sep 17 17:50:09.864756 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:09.864734    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zg99"]
Sep 17 17:50:09.866633 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:50:09.866605    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda10431ef_8b50_4ddb_ab94_e06de1a670fd.slice/crio-800112ac1ff044b2d4062bd10cf4f4f3cc871f0b6450c3c8c88b70bd252fbefb WatchSource:0}: Error finding container 800112ac1ff044b2d4062bd10cf4f4f3cc871f0b6450c3c8c88b70bd252fbefb: Status 404 returned error can't find the container with id 800112ac1ff044b2d4062bd10cf4f4f3cc871f0b6450c3c8c88b70bd252fbefb
Sep 17 17:50:10.328459 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:10.328423    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zg99" event={"ID":"a10431ef-8b50-4ddb-ab94-e06de1a670fd","Type":"ContainerStarted","Data":"800112ac1ff044b2d4062bd10cf4f4f3cc871f0b6450c3c8c88b70bd252fbefb"}
Sep 17 17:50:11.334008 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:11.333967    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zg99" event={"ID":"a10431ef-8b50-4ddb-ab94-e06de1a670fd","Type":"ContainerStarted","Data":"80570f5eed16e8fb6e202660a917491dde6db10d86e8e460662f15a2f4184f7d"}
Sep 17 17:50:11.334406 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:11.334025    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zg99" event={"ID":"a10431ef-8b50-4ddb-ab94-e06de1a670fd","Type":"ContainerStarted","Data":"c227b97139c54cda3dbd460453ce4f7464ef4f84cac805d0868b13d87b180a87"}
Sep 17 17:50:11.334406 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:11.334226    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zg99"
Sep 17 17:50:11.334406 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:11.334361    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zg99"
Sep 17 17:50:11.336130 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:11.336112    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zg99"
Sep 17 17:50:11.358178 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:11.358138    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zg99" podStartSLOduration=1.083875874 podStartE2EDuration="2.358125527s" totalImagesPullingTime="1.274249653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:50:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:50:11.355916552 +0000 UTC m=+947.769706989" watchObservedRunningTime="2026-09-17 17:50:11.358125527 +0000 UTC m=+947.771915963"
Sep 17 17:50:12.149814 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:12.149777    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gjcgr" podUID="4587ada0-48d9-4478-abfc-5d90bfb320de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Sep 17 17:50:18.345012 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:18.344982    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zg99"
Sep 17 17:50:19.264054 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:19.264023    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:50:20.024685 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:20.024638    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:50:22.149852 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:22.149818    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gjcgr" podUID="4587ada0-48d9-4478-abfc-5d90bfb320de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Sep 17 17:50:32.149850 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:32.149823    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gjcgr"
Sep 17 17:50:40.338837 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:40.338807    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gjcgr"]
Sep 17 17:50:40.339237 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:40.339029    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gjcgr" podUID="4587ada0-48d9-4478-abfc-5d90bfb320de" containerName="kserve-container" containerID="cri-o://ba72293fe8fe05ecad3b58013a19c58113eb0bb312503b1c23697be3df6b7986" gracePeriod=30
Sep 17 17:50:40.339237 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:40.339072    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gjcgr" podUID="4587ada0-48d9-4478-abfc-5d90bfb320de" containerName="kube-rbac-proxy" containerID="cri-o://06fa150d9fd7e19deadacb67e7b068829a8c4145600d29a83f108a9d26a56db1" gracePeriod=30
Sep 17 17:50:40.357093 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:40.357069    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:50:41.435953 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:41.435919    2585 generic.go:356] "Generic (PLEG): container finished" podID="4587ada0-48d9-4478-abfc-5d90bfb320de" containerID="06fa150d9fd7e19deadacb67e7b068829a8c4145600d29a83f108a9d26a56db1" exitCode=2
Sep 17 17:50:41.435953 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:41.435960    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gjcgr" event={"ID":"4587ada0-48d9-4478-abfc-5d90bfb320de","Type":"ContainerDied","Data":"06fa150d9fd7e19deadacb67e7b068829a8c4145600d29a83f108a9d26a56db1"}
Sep 17 17:50:42.144363 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:42.144321    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gjcgr" podUID="4587ada0-48d9-4478-abfc-5d90bfb320de" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.19:8643/healthz\": dial tcp 10.133.0.19:8643: connect: connection refused"
Sep 17 17:50:42.149671 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:42.149646    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gjcgr" podUID="4587ada0-48d9-4478-abfc-5d90bfb320de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused"
Sep 17 17:50:45.173396 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.173374    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gjcgr"
Sep 17 17:50:45.312821 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.312737    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4587ada0-48d9-4478-abfc-5d90bfb320de-proxy-tls\") pod \"4587ada0-48d9-4478-abfc-5d90bfb320de\" (UID: \"4587ada0-48d9-4478-abfc-5d90bfb320de\") "
Sep 17 17:50:45.312821 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.312784    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4587ada0-48d9-4478-abfc-5d90bfb320de-cabundle-cert\") pod \"4587ada0-48d9-4478-abfc-5d90bfb320de\" (UID: \"4587ada0-48d9-4478-abfc-5d90bfb320de\") "
Sep 17 17:50:45.312821 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.312809    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4587ada0-48d9-4478-abfc-5d90bfb320de-kube-rbac-proxy-sar-config\") pod \"4587ada0-48d9-4478-abfc-5d90bfb320de\" (UID: \"4587ada0-48d9-4478-abfc-5d90bfb320de\") "
Sep 17 17:50:45.313095 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.312835    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4587ada0-48d9-4478-abfc-5d90bfb320de-kserve-provision-location\") pod \"4587ada0-48d9-4478-abfc-5d90bfb320de\" (UID: \"4587ada0-48d9-4478-abfc-5d90bfb320de\") "
Sep 17 17:50:45.313095 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.312866    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hbqdt\" (UniqueName: \"kubernetes.io/projected/4587ada0-48d9-4478-abfc-5d90bfb320de-kube-api-access-hbqdt\") pod \"4587ada0-48d9-4478-abfc-5d90bfb320de\" (UID: \"4587ada0-48d9-4478-abfc-5d90bfb320de\") "
Sep 17 17:50:45.313188 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.313115    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4587ada0-48d9-4478-abfc-5d90bfb320de-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4587ada0-48d9-4478-abfc-5d90bfb320de" (UID: "4587ada0-48d9-4478-abfc-5d90bfb320de"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:50:45.313188 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.313171    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4587ada0-48d9-4478-abfc-5d90bfb320de-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4587ada0-48d9-4478-abfc-5d90bfb320de" (UID: "4587ada0-48d9-4478-abfc-5d90bfb320de"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:50:45.313251 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.313182    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4587ada0-48d9-4478-abfc-5d90bfb320de-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4587ada0-48d9-4478-abfc-5d90bfb320de" (UID: "4587ada0-48d9-4478-abfc-5d90bfb320de"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:50:45.314880 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.314860    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4587ada0-48d9-4478-abfc-5d90bfb320de-kube-api-access-hbqdt" (OuterVolumeSpecName: "kube-api-access-hbqdt") pod "4587ada0-48d9-4478-abfc-5d90bfb320de" (UID: "4587ada0-48d9-4478-abfc-5d90bfb320de"). InnerVolumeSpecName "kube-api-access-hbqdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:50:45.314880 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.314872    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4587ada0-48d9-4478-abfc-5d90bfb320de-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4587ada0-48d9-4478-abfc-5d90bfb320de" (UID: "4587ada0-48d9-4478-abfc-5d90bfb320de"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:50:45.414193 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.414160    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4587ada0-48d9-4478-abfc-5d90bfb320de-proxy-tls\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 17:50:45.414193 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.414194    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4587ada0-48d9-4478-abfc-5d90bfb320de-cabundle-cert\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 17:50:45.414369 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.414210    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4587ada0-48d9-4478-abfc-5d90bfb320de-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 17:50:45.414369 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.414251    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4587ada0-48d9-4478-abfc-5d90bfb320de-kserve-provision-location\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 17:50:45.414369 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.414266    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbqdt\" (UniqueName: \"kubernetes.io/projected/4587ada0-48d9-4478-abfc-5d90bfb320de-kube-api-access-hbqdt\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 17:50:45.451987 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.451952    2585 generic.go:356] "Generic (PLEG): container finished" podID="4587ada0-48d9-4478-abfc-5d90bfb320de" containerID="ba72293fe8fe05ecad3b58013a19c58113eb0bb312503b1c23697be3df6b7986" exitCode=0
Sep 17 17:50:45.452149 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.452029    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gjcgr" event={"ID":"4587ada0-48d9-4478-abfc-5d90bfb320de","Type":"ContainerDied","Data":"ba72293fe8fe05ecad3b58013a19c58113eb0bb312503b1c23697be3df6b7986"}
Sep 17 17:50:45.452149 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.452019    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gjcgr"
Sep 17 17:50:45.452149 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.452086    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gjcgr" event={"ID":"4587ada0-48d9-4478-abfc-5d90bfb320de","Type":"ContainerDied","Data":"5d977d9d1f8993af41819d8d2492891d679a2737e53d10774d395f593ad049c2"}
Sep 17 17:50:45.452149 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.452112    2585 scope.go:117] "RemoveContainer" containerID="06fa150d9fd7e19deadacb67e7b068829a8c4145600d29a83f108a9d26a56db1"
Sep 17 17:50:45.461263 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.461248    2585 scope.go:117] "RemoveContainer" containerID="ba72293fe8fe05ecad3b58013a19c58113eb0bb312503b1c23697be3df6b7986"
Sep 17 17:50:45.468330 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.468313    2585 scope.go:117] "RemoveContainer" containerID="2bb7640bd0ef4aee4d599cee99a653a9ef911d87750f8401d3bcb8779acd3eac"
Sep 17 17:50:45.477439 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.475811    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gjcgr"]
Sep 17 17:50:45.479670 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.479643    2585 scope.go:117] "RemoveContainer" containerID="06fa150d9fd7e19deadacb67e7b068829a8c4145600d29a83f108a9d26a56db1"
Sep 17 17:50:45.480000 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:50:45.479974    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06fa150d9fd7e19deadacb67e7b068829a8c4145600d29a83f108a9d26a56db1\": container with ID starting with 06fa150d9fd7e19deadacb67e7b068829a8c4145600d29a83f108a9d26a56db1 not found: ID does not exist" containerID="06fa150d9fd7e19deadacb67e7b068829a8c4145600d29a83f108a9d26a56db1"
Sep 17 17:50:45.480055 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.480013    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06fa150d9fd7e19deadacb67e7b068829a8c4145600d29a83f108a9d26a56db1"} err="failed to get container status \"06fa150d9fd7e19deadacb67e7b068829a8c4145600d29a83f108a9d26a56db1\": rpc error: code = NotFound desc = could not find container \"06fa150d9fd7e19deadacb67e7b068829a8c4145600d29a83f108a9d26a56db1\": container with ID starting with 06fa150d9fd7e19deadacb67e7b068829a8c4145600d29a83f108a9d26a56db1 not found: ID does not exist"
Sep 17 17:50:45.480055 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.480039    2585 scope.go:117] "RemoveContainer" containerID="ba72293fe8fe05ecad3b58013a19c58113eb0bb312503b1c23697be3df6b7986"
Sep 17 17:50:45.480337 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:50:45.480312    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba72293fe8fe05ecad3b58013a19c58113eb0bb312503b1c23697be3df6b7986\": container with ID starting with ba72293fe8fe05ecad3b58013a19c58113eb0bb312503b1c23697be3df6b7986 not found: ID does not exist" containerID="ba72293fe8fe05ecad3b58013a19c58113eb0bb312503b1c23697be3df6b7986"
Sep 17 17:50:45.480412 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.480346    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba72293fe8fe05ecad3b58013a19c58113eb0bb312503b1c23697be3df6b7986"} err="failed to get container status \"ba72293fe8fe05ecad3b58013a19c58113eb0bb312503b1c23697be3df6b7986\": rpc error: code = NotFound desc = could not find container \"ba72293fe8fe05ecad3b58013a19c58113eb0bb312503b1c23697be3df6b7986\": container with ID starting with ba72293fe8fe05ecad3b58013a19c58113eb0bb312503b1c23697be3df6b7986 not found: ID does not exist"
Sep 17 17:50:45.480412 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.480367    2585 scope.go:117] "RemoveContainer" containerID="2bb7640bd0ef4aee4d599cee99a653a9ef911d87750f8401d3bcb8779acd3eac"
Sep 17 17:50:45.480693 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:50:45.480669    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bb7640bd0ef4aee4d599cee99a653a9ef911d87750f8401d3bcb8779acd3eac\": container with ID starting with 2bb7640bd0ef4aee4d599cee99a653a9ef911d87750f8401d3bcb8779acd3eac not found: ID does not exist" containerID="2bb7640bd0ef4aee4d599cee99a653a9ef911d87750f8401d3bcb8779acd3eac"
Sep 17 17:50:45.480778 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.480697    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bb7640bd0ef4aee4d599cee99a653a9ef911d87750f8401d3bcb8779acd3eac"} err="failed to get container status \"2bb7640bd0ef4aee4d599cee99a653a9ef911d87750f8401d3bcb8779acd3eac\": rpc error: code = NotFound desc = could not find container \"2bb7640bd0ef4aee4d599cee99a653a9ef911d87750f8401d3bcb8779acd3eac\": container with ID starting with 2bb7640bd0ef4aee4d599cee99a653a9ef911d87750f8401d3bcb8779acd3eac not found: ID does not exist"
Sep 17 17:50:45.483589 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.483568    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-gjcgr"]
Sep 17 17:50:45.540433 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.540395    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:50:45.578617 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.578552    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp"]
Sep 17 17:50:45.578804 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.578793    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4587ada0-48d9-4478-abfc-5d90bfb320de" containerName="storage-initializer"
Sep 17 17:50:45.578845 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.578806    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="4587ada0-48d9-4478-abfc-5d90bfb320de" containerName="storage-initializer"
Sep 17 17:50:45.578845 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.578817    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4587ada0-48d9-4478-abfc-5d90bfb320de" containerName="kserve-container"
Sep 17 17:50:45.578845 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.578823    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="4587ada0-48d9-4478-abfc-5d90bfb320de" containerName="kserve-container"
Sep 17 17:50:45.578845 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.578834    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4587ada0-48d9-4478-abfc-5d90bfb320de" containerName="kube-rbac-proxy"
Sep 17 17:50:45.578845 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.578839    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="4587ada0-48d9-4478-abfc-5d90bfb320de" containerName="kube-rbac-proxy"
Sep 17 17:50:45.578984 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.578876    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="4587ada0-48d9-4478-abfc-5d90bfb320de" containerName="kserve-container"
Sep 17 17:50:45.578984 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.578883    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="4587ada0-48d9-4478-abfc-5d90bfb320de" containerName="kube-rbac-proxy"
Sep 17 17:50:45.582144 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.582127    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp"
Sep 17 17:50:45.585137 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.585117    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
Sep 17 17:50:45.585256 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.585238    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
Sep 17 17:50:45.585311 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.585241    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:50:45.585354 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.585248    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:50:45.597229 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.597209    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp"]
Sep 17 17:50:45.716669 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.716639    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27fc4a3f-9136-4433-885f-a15e46b30d2e-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp\" (UID: \"27fc4a3f-9136-4433-885f-a15e46b30d2e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp"
Sep 17 17:50:45.716831 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.716678    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27fc4a3f-9136-4433-885f-a15e46b30d2e-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp\" (UID: \"27fc4a3f-9136-4433-885f-a15e46b30d2e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp"
Sep 17 17:50:45.716831 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.716705    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzgqh\" (UniqueName: \"kubernetes.io/projected/27fc4a3f-9136-4433-885f-a15e46b30d2e-kube-api-access-dzgqh\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp\" (UID: \"27fc4a3f-9136-4433-885f-a15e46b30d2e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp"
Sep 17 17:50:45.716831 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.716768    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27fc4a3f-9136-4433-885f-a15e46b30d2e-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp\" (UID: \"27fc4a3f-9136-4433-885f-a15e46b30d2e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp"
Sep 17 17:50:45.716831 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.716797    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27fc4a3f-9136-4433-885f-a15e46b30d2e-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp\" (UID: \"27fc4a3f-9136-4433-885f-a15e46b30d2e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp"
Sep 17 17:50:45.728451 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.728428    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:50:45.817466 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.817440    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27fc4a3f-9136-4433-885f-a15e46b30d2e-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp\" (UID: \"27fc4a3f-9136-4433-885f-a15e46b30d2e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp"
Sep 17 17:50:45.817620 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.817478    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27fc4a3f-9136-4433-885f-a15e46b30d2e-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp\" (UID: \"27fc4a3f-9136-4433-885f-a15e46b30d2e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp"
Sep 17 17:50:45.817620 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.817551    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27fc4a3f-9136-4433-885f-a15e46b30d2e-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp\" (UID: \"27fc4a3f-9136-4433-885f-a15e46b30d2e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp"
Sep 17 17:50:45.817620 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.817572    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27fc4a3f-9136-4433-885f-a15e46b30d2e-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp\" (UID: \"27fc4a3f-9136-4433-885f-a15e46b30d2e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp"
Sep 17 17:50:45.817620 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.817589    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzgqh\" (UniqueName: \"kubernetes.io/projected/27fc4a3f-9136-4433-885f-a15e46b30d2e-kube-api-access-dzgqh\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp\" (UID: \"27fc4a3f-9136-4433-885f-a15e46b30d2e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp"
Sep 17 17:50:45.818078 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.818046    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27fc4a3f-9136-4433-885f-a15e46b30d2e-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp\" (UID: \"27fc4a3f-9136-4433-885f-a15e46b30d2e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp"
Sep 17 17:50:45.818272 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.818254    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27fc4a3f-9136-4433-885f-a15e46b30d2e-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp\" (UID: \"27fc4a3f-9136-4433-885f-a15e46b30d2e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp"
Sep 17 17:50:45.818330 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.818315    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27fc4a3f-9136-4433-885f-a15e46b30d2e-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp\" (UID: \"27fc4a3f-9136-4433-885f-a15e46b30d2e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp"
Sep 17 17:50:45.820067 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.820052    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27fc4a3f-9136-4433-885f-a15e46b30d2e-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp\" (UID: \"27fc4a3f-9136-4433-885f-a15e46b30d2e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp"
Sep 17 17:50:45.827818 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.827798    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzgqh\" (UniqueName: \"kubernetes.io/projected/27fc4a3f-9136-4433-885f-a15e46b30d2e-kube-api-access-dzgqh\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp\" (UID: \"27fc4a3f-9136-4433-885f-a15e46b30d2e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp"
Sep 17 17:50:45.892779 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:45.892707    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp"
Sep 17 17:50:46.104709 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:46.104675    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4587ada0-48d9-4478-abfc-5d90bfb320de" path="/var/lib/kubelet/pods/4587ada0-48d9-4478-abfc-5d90bfb320de/volumes"
Sep 17 17:50:46.177533 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:46.177505    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp"]
Sep 17 17:50:46.178404 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:50:46.178379    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27fc4a3f_9136_4433_885f_a15e46b30d2e.slice/crio-364c29c3396a5509b225660eb58b6f26517c3dcb67c07629b1c17ec6d47f5223 WatchSource:0}: Error finding container 364c29c3396a5509b225660eb58b6f26517c3dcb67c07629b1c17ec6d47f5223: Status 404 returned error can't find the container with id 364c29c3396a5509b225660eb58b6f26517c3dcb67c07629b1c17ec6d47f5223
Sep 17 17:50:46.458749 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:46.458658    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp" event={"ID":"27fc4a3f-9136-4433-885f-a15e46b30d2e","Type":"ContainerStarted","Data":"037c3d2cae5f2d0d39fee6b70775a9ba0779ac6cf8681947d95908aaa6712d14"}
Sep 17 17:50:46.458749 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:46.458731    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp" event={"ID":"27fc4a3f-9136-4433-885f-a15e46b30d2e","Type":"ContainerStarted","Data":"364c29c3396a5509b225660eb58b6f26517c3dcb67c07629b1c17ec6d47f5223"}
Sep 17 17:50:50.476667 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:50.476579    2585 generic.go:356] "Generic (PLEG): container finished" podID="27fc4a3f-9136-4433-885f-a15e46b30d2e" containerID="037c3d2cae5f2d0d39fee6b70775a9ba0779ac6cf8681947d95908aaa6712d14" exitCode=0
Sep 17 17:50:50.477012 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:50.476658    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp" event={"ID":"27fc4a3f-9136-4433-885f-a15e46b30d2e","Type":"ContainerDied","Data":"037c3d2cae5f2d0d39fee6b70775a9ba0779ac6cf8681947d95908aaa6712d14"}
Sep 17 17:50:51.483113 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:51.483079    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp" event={"ID":"27fc4a3f-9136-4433-885f-a15e46b30d2e","Type":"ContainerStarted","Data":"3c90f56dcc64e7ab237933fd924443784f11530cbfb22fef823a72c20d03a00e"}
Sep 17 17:50:51.483478 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:51.483142    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp" event={"ID":"27fc4a3f-9136-4433-885f-a15e46b30d2e","Type":"ContainerStarted","Data":"1a0f388736a28c72e321aa2a7bc4fa921f3c24aefc4d54a8cb7737c05d778bc7"}
Sep 17 17:50:51.483478 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:51.483348    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp"
Sep 17 17:50:51.507619 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:51.507568    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp" podStartSLOduration=2.660326709 podStartE2EDuration="6.507551306s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.847224597s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:50:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:50:51.507122622 +0000 UTC m=+987.920913062" watchObservedRunningTime="2026-09-17 17:50:51.507551306 +0000 UTC m=+987.921341741"
Sep 17 17:50:52.486963 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:52.486940    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp"
Sep 17 17:50:52.488154 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:52.488128    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp" podUID="27fc4a3f-9136-4433-885f-a15e46b30d2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 17 17:50:53.491373 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:53.491330    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp" podUID="27fc4a3f-9136-4433-885f-a15e46b30d2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 17 17:50:58.495246 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:58.495217    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp"
Sep 17 17:50:58.495831 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:50:58.495802    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp" podUID="27fc4a3f-9136-4433-885f-a15e46b30d2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 17 17:51:08.495744 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:08.495702    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp" podUID="27fc4a3f-9136-4433-885f-a15e46b30d2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 17 17:51:18.452048 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:18.452013    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:51:18.496656 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:18.496621    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp" podUID="27fc4a3f-9136-4433-885f-a15e46b30d2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 17 17:51:28.495734 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:28.495691    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp" podUID="27fc4a3f-9136-4433-885f-a15e46b30d2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 17 17:51:38.496014 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:38.495975    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp" podUID="27fc4a3f-9136-4433-885f-a15e46b30d2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 17 17:51:48.496341 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:48.496305    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp" podUID="27fc4a3f-9136-4433-885f-a15e46b30d2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 17 17:51:54.250749 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:54.250717    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-6zg99_a10431ef-8b50-4ddb-ab94-e06de1a670fd/kserve-container/0.log"
Sep 17 17:51:54.552292 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:54.552212    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:51:54.686765 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:54.686726    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7rv24"]
Sep 17 17:51:54.690061 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:54.690031    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7rv24"
Sep 17 17:51:54.692811 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:54.692786    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Sep 17 17:51:54.692984 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:54.692962    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Sep 17 17:51:54.741827 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:54.741790    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7rv24"]
Sep 17 17:51:54.792291 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:54.792260    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zg99"]
Sep 17 17:51:54.792535 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:54.792479    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zg99" podUID="a10431ef-8b50-4ddb-ab94-e06de1a670fd" containerName="kserve-container" containerID="cri-o://c227b97139c54cda3dbd460453ce4f7464ef4f84cac805d0868b13d87b180a87" gracePeriod=30
Sep 17 17:51:54.792535 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:54.792517    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zg99" podUID="a10431ef-8b50-4ddb-ab94-e06de1a670fd" containerName="kube-rbac-proxy" containerID="cri-o://80570f5eed16e8fb6e202660a917491dde6db10d86e8e460662f15a2f4184f7d" gracePeriod=30
Sep 17 17:51:54.797525 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:54.797237    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3344339e-2d84-4ee1-b4e5-63cca80cfa48-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-7rv24\" (UID: \"3344339e-2d84-4ee1-b4e5-63cca80cfa48\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7rv24"
Sep 17 17:51:54.797525 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:54.797349    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3344339e-2d84-4ee1-b4e5-63cca80cfa48-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-7rv24\" (UID: \"3344339e-2d84-4ee1-b4e5-63cca80cfa48\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7rv24"
Sep 17 17:51:54.797525 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:54.797418    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mtq9\" (UniqueName: \"kubernetes.io/projected/3344339e-2d84-4ee1-b4e5-63cca80cfa48-kube-api-access-9mtq9\") pod \"isvc-autogluon-v1-predictor-574b5b7495-7rv24\" (UID: \"3344339e-2d84-4ee1-b4e5-63cca80cfa48\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7rv24"
Sep 17 17:51:54.797525 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:54.797516    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3344339e-2d84-4ee1-b4e5-63cca80cfa48-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-7rv24\" (UID: \"3344339e-2d84-4ee1-b4e5-63cca80cfa48\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7rv24"
Sep 17 17:51:54.797751 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:54.797564    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3344339e-2d84-4ee1-b4e5-63cca80cfa48-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-7rv24\" (UID: \"3344339e-2d84-4ee1-b4e5-63cca80cfa48\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7rv24"
Sep 17 17:51:54.898770 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:54.898731    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3344339e-2d84-4ee1-b4e5-63cca80cfa48-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-7rv24\" (UID: \"3344339e-2d84-4ee1-b4e5-63cca80cfa48\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7rv24"
Sep 17 17:51:54.898927 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:54.898781    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mtq9\" (UniqueName: \"kubernetes.io/projected/3344339e-2d84-4ee1-b4e5-63cca80cfa48-kube-api-access-9mtq9\") pod \"isvc-autogluon-v1-predictor-574b5b7495-7rv24\" (UID: \"3344339e-2d84-4ee1-b4e5-63cca80cfa48\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7rv24"
Sep 17 17:51:54.898927 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:54.898849    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3344339e-2d84-4ee1-b4e5-63cca80cfa48-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-7rv24\" (UID: \"3344339e-2d84-4ee1-b4e5-63cca80cfa48\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7rv24"
Sep 17 17:51:54.898927 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:54.898875    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3344339e-2d84-4ee1-b4e5-63cca80cfa48-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-7rv24\" (UID: \"3344339e-2d84-4ee1-b4e5-63cca80cfa48\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7rv24"
Sep 17 17:51:54.898927 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:54.898917    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3344339e-2d84-4ee1-b4e5-63cca80cfa48-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-7rv24\" (UID: \"3344339e-2d84-4ee1-b4e5-63cca80cfa48\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7rv24"
Sep 17 17:51:54.899357 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:54.899334    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3344339e-2d84-4ee1-b4e5-63cca80cfa48-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-7rv24\" (UID: \"3344339e-2d84-4ee1-b4e5-63cca80cfa48\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7rv24"
Sep 17 17:51:54.899760 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:54.899736    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3344339e-2d84-4ee1-b4e5-63cca80cfa48-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-7rv24\" (UID: \"3344339e-2d84-4ee1-b4e5-63cca80cfa48\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7rv24"
Sep 17 17:51:54.900353 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:54.900328    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3344339e-2d84-4ee1-b4e5-63cca80cfa48-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-7rv24\" (UID: \"3344339e-2d84-4ee1-b4e5-63cca80cfa48\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7rv24"
Sep 17 17:51:54.901876 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:54.901852    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3344339e-2d84-4ee1-b4e5-63cca80cfa48-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-7rv24\" (UID: \"3344339e-2d84-4ee1-b4e5-63cca80cfa48\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7rv24"
Sep 17 17:51:54.909041 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:54.909022    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mtq9\" (UniqueName: \"kubernetes.io/projected/3344339e-2d84-4ee1-b4e5-63cca80cfa48-kube-api-access-9mtq9\") pod \"isvc-autogluon-v1-predictor-574b5b7495-7rv24\" (UID: \"3344339e-2d84-4ee1-b4e5-63cca80cfa48\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7rv24"
Sep 17 17:51:55.002023 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:55.001989    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7rv24"
Sep 17 17:51:55.027130 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:55.027104    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zg99"
Sep 17 17:51:55.100935 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:55.100898    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a10431ef-8b50-4ddb-ab94-e06de1a670fd-kube-rbac-proxy-sar-config\") pod \"a10431ef-8b50-4ddb-ab94-e06de1a670fd\" (UID: \"a10431ef-8b50-4ddb-ab94-e06de1a670fd\") "
Sep 17 17:51:55.100935 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:55.100937    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a10431ef-8b50-4ddb-ab94-e06de1a670fd-proxy-tls\") pod \"a10431ef-8b50-4ddb-ab94-e06de1a670fd\" (UID: \"a10431ef-8b50-4ddb-ab94-e06de1a670fd\") "
Sep 17 17:51:55.101148 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:55.100996    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bbvj8\" (UniqueName: \"kubernetes.io/projected/a10431ef-8b50-4ddb-ab94-e06de1a670fd-kube-api-access-bbvj8\") pod \"a10431ef-8b50-4ddb-ab94-e06de1a670fd\" (UID: \"a10431ef-8b50-4ddb-ab94-e06de1a670fd\") "
Sep 17 17:51:55.101281 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:55.101258    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a10431ef-8b50-4ddb-ab94-e06de1a670fd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a10431ef-8b50-4ddb-ab94-e06de1a670fd" (UID: "a10431ef-8b50-4ddb-ab94-e06de1a670fd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:51:55.102898 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:55.102873    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a10431ef-8b50-4ddb-ab94-e06de1a670fd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a10431ef-8b50-4ddb-ab94-e06de1a670fd" (UID: "a10431ef-8b50-4ddb-ab94-e06de1a670fd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:51:55.103018 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:55.102961    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a10431ef-8b50-4ddb-ab94-e06de1a670fd-kube-api-access-bbvj8" (OuterVolumeSpecName: "kube-api-access-bbvj8") pod "a10431ef-8b50-4ddb-ab94-e06de1a670fd" (UID: "a10431ef-8b50-4ddb-ab94-e06de1a670fd"). InnerVolumeSpecName "kube-api-access-bbvj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:51:55.201599 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:55.201566    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbvj8\" (UniqueName: \"kubernetes.io/projected/a10431ef-8b50-4ddb-ab94-e06de1a670fd-kube-api-access-bbvj8\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 17:51:55.201599 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:55.201588    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a10431ef-8b50-4ddb-ab94-e06de1a670fd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 17:51:55.201599 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:55.201598    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a10431ef-8b50-4ddb-ab94-e06de1a670fd-proxy-tls\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 17:51:55.300056 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:55.300018    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7rv24"]
Sep 17 17:51:55.302467 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:51:55.302443    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3344339e_2d84_4ee1_b4e5_63cca80cfa48.slice/crio-687ae5c5070fe43c783137bce26ce69f5642f8ed12d9aedbfa143239dd495910 WatchSource:0}: Error finding container 687ae5c5070fe43c783137bce26ce69f5642f8ed12d9aedbfa143239dd495910: Status 404 returned error can't find the container with id 687ae5c5070fe43c783137bce26ce69f5642f8ed12d9aedbfa143239dd495910
Sep 17 17:51:55.711390 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:55.711348    2585 generic.go:356] "Generic (PLEG): container finished" podID="a10431ef-8b50-4ddb-ab94-e06de1a670fd" containerID="80570f5eed16e8fb6e202660a917491dde6db10d86e8e460662f15a2f4184f7d" exitCode=2
Sep 17 17:51:55.711390 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:55.711380    2585 generic.go:356] "Generic (PLEG): container finished" podID="a10431ef-8b50-4ddb-ab94-e06de1a670fd" containerID="c227b97139c54cda3dbd460453ce4f7464ef4f84cac805d0868b13d87b180a87" exitCode=2
Sep 17 17:51:55.711648 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:55.711408    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zg99"
Sep 17 17:51:55.711648 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:55.711424    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zg99" event={"ID":"a10431ef-8b50-4ddb-ab94-e06de1a670fd","Type":"ContainerDied","Data":"80570f5eed16e8fb6e202660a917491dde6db10d86e8e460662f15a2f4184f7d"}
Sep 17 17:51:55.711648 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:55.711481    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zg99" event={"ID":"a10431ef-8b50-4ddb-ab94-e06de1a670fd","Type":"ContainerDied","Data":"c227b97139c54cda3dbd460453ce4f7464ef4f84cac805d0868b13d87b180a87"}
Sep 17 17:51:55.711648 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:55.711520    2585 scope.go:117] "RemoveContainer" containerID="80570f5eed16e8fb6e202660a917491dde6db10d86e8e460662f15a2f4184f7d"
Sep 17 17:51:55.711648 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:55.711528    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zg99" event={"ID":"a10431ef-8b50-4ddb-ab94-e06de1a670fd","Type":"ContainerDied","Data":"800112ac1ff044b2d4062bd10cf4f4f3cc871f0b6450c3c8c88b70bd252fbefb"}
Sep 17 17:51:55.713236 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:55.713215    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7rv24" event={"ID":"3344339e-2d84-4ee1-b4e5-63cca80cfa48","Type":"ContainerStarted","Data":"dc626136f61103046d383183ee4fbe12ebe5b21aa9b0400ff3e1dfccb6fac095"}
Sep 17 17:51:55.713344 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:55.713268    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7rv24" event={"ID":"3344339e-2d84-4ee1-b4e5-63cca80cfa48","Type":"ContainerStarted","Data":"687ae5c5070fe43c783137bce26ce69f5642f8ed12d9aedbfa143239dd495910"}
Sep 17 17:51:55.720020 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:55.720001    2585 scope.go:117] "RemoveContainer" containerID="c227b97139c54cda3dbd460453ce4f7464ef4f84cac805d0868b13d87b180a87"
Sep 17 17:51:55.728154 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:55.728133    2585 scope.go:117] "RemoveContainer" containerID="80570f5eed16e8fb6e202660a917491dde6db10d86e8e460662f15a2f4184f7d"
Sep 17 17:51:55.728406 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:51:55.728389    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80570f5eed16e8fb6e202660a917491dde6db10d86e8e460662f15a2f4184f7d\": container with ID starting with 80570f5eed16e8fb6e202660a917491dde6db10d86e8e460662f15a2f4184f7d not found: ID does not exist" containerID="80570f5eed16e8fb6e202660a917491dde6db10d86e8e460662f15a2f4184f7d"
Sep 17 17:51:55.728486 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:55.728412    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80570f5eed16e8fb6e202660a917491dde6db10d86e8e460662f15a2f4184f7d"} err="failed to get container status \"80570f5eed16e8fb6e202660a917491dde6db10d86e8e460662f15a2f4184f7d\": rpc error: code = NotFound desc = could not find container \"80570f5eed16e8fb6e202660a917491dde6db10d86e8e460662f15a2f4184f7d\": container with ID starting with 80570f5eed16e8fb6e202660a917491dde6db10d86e8e460662f15a2f4184f7d not found: ID does not exist"
Sep 17 17:51:55.728486 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:55.728429    2585 scope.go:117] "RemoveContainer" containerID="c227b97139c54cda3dbd460453ce4f7464ef4f84cac805d0868b13d87b180a87"
Sep 17 17:51:55.728694 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:51:55.728676    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c227b97139c54cda3dbd460453ce4f7464ef4f84cac805d0868b13d87b180a87\": container with ID starting with c227b97139c54cda3dbd460453ce4f7464ef4f84cac805d0868b13d87b180a87 not found: ID does not exist" containerID="c227b97139c54cda3dbd460453ce4f7464ef4f84cac805d0868b13d87b180a87"
Sep 17 17:51:55.728757 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:55.728698    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c227b97139c54cda3dbd460453ce4f7464ef4f84cac805d0868b13d87b180a87"} err="failed to get container status \"c227b97139c54cda3dbd460453ce4f7464ef4f84cac805d0868b13d87b180a87\": rpc error: code = NotFound desc = could not find container \"c227b97139c54cda3dbd460453ce4f7464ef4f84cac805d0868b13d87b180a87\": container with ID starting with c227b97139c54cda3dbd460453ce4f7464ef4f84cac805d0868b13d87b180a87 not found: ID does not exist"
Sep 17 17:51:55.728757 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:55.728712    2585 scope.go:117] "RemoveContainer" containerID="80570f5eed16e8fb6e202660a917491dde6db10d86e8e460662f15a2f4184f7d"
Sep 17 17:51:55.728905 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:55.728888    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80570f5eed16e8fb6e202660a917491dde6db10d86e8e460662f15a2f4184f7d"} err="failed to get container status \"80570f5eed16e8fb6e202660a917491dde6db10d86e8e460662f15a2f4184f7d\": rpc error: code = NotFound desc = could not find container \"80570f5eed16e8fb6e202660a917491dde6db10d86e8e460662f15a2f4184f7d\": container with ID starting with 80570f5eed16e8fb6e202660a917491dde6db10d86e8e460662f15a2f4184f7d not found: ID does not exist"
Sep 17 17:51:55.728905 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:55.728903    2585 scope.go:117] "RemoveContainer" containerID="c227b97139c54cda3dbd460453ce4f7464ef4f84cac805d0868b13d87b180a87"
Sep 17 17:51:55.729081 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:55.729065    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c227b97139c54cda3dbd460453ce4f7464ef4f84cac805d0868b13d87b180a87"} err="failed to get container status \"c227b97139c54cda3dbd460453ce4f7464ef4f84cac805d0868b13d87b180a87\": rpc error: code = NotFound desc = could not find container \"c227b97139c54cda3dbd460453ce4f7464ef4f84cac805d0868b13d87b180a87\": container with ID starting with c227b97139c54cda3dbd460453ce4f7464ef4f84cac805d0868b13d87b180a87 not found: ID does not exist"
Sep 17 17:51:55.758045 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:55.758024    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zg99"]
Sep 17 17:51:55.759523 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:55.759481    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-6zg99"]
Sep 17 17:51:55.959333 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:55.959288    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:51:56.109832 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:56.106447    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a10431ef-8b50-4ddb-ab94-e06de1a670fd" path="/var/lib/kubelet/pods/a10431ef-8b50-4ddb-ab94-e06de1a670fd/volumes"
Sep 17 17:51:58.496476 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:51:58.496436    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp" podUID="27fc4a3f-9136-4433-885f-a15e46b30d2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 17 17:52:08.497364 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:52:08.497337    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp"
Sep 17 17:52:16.085125 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:52:16.085091    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp"]
Sep 17 17:52:16.085542 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:52:16.085362    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp" podUID="27fc4a3f-9136-4433-885f-a15e46b30d2e" containerName="kserve-container" containerID="cri-o://1a0f388736a28c72e321aa2a7bc4fa921f3c24aefc4d54a8cb7737c05d778bc7" gracePeriod=30
Sep 17 17:52:16.085542 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:52:16.085366    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp" podUID="27fc4a3f-9136-4433-885f-a15e46b30d2e" containerName="kube-rbac-proxy" containerID="cri-o://3c90f56dcc64e7ab237933fd924443784f11530cbfb22fef823a72c20d03a00e" gracePeriod=30
Sep 17 17:52:16.128070 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:52:16.128033    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:52:16.793723 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:52:16.793688    2585 generic.go:356] "Generic (PLEG): container finished" podID="27fc4a3f-9136-4433-885f-a15e46b30d2e" containerID="3c90f56dcc64e7ab237933fd924443784f11530cbfb22fef823a72c20d03a00e" exitCode=2
Sep 17 17:52:16.793861 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:52:16.793763    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp" event={"ID":"27fc4a3f-9136-4433-885f-a15e46b30d2e","Type":"ContainerDied","Data":"3c90f56dcc64e7ab237933fd924443784f11530cbfb22fef823a72c20d03a00e"}
Sep 17 17:52:18.492265 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:52:18.492222    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp" podUID="27fc4a3f-9136-4433-885f-a15e46b30d2e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.21:8643/healthz\": dial tcp 10.133.0.21:8643: connect: connection refused"
Sep 17 17:52:18.496538 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:52:18.496508    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp" podUID="27fc4a3f-9136-4433-885f-a15e46b30d2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused"
Sep 17 17:52:20.811383 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:52:20.811353    2585 generic.go:356] "Generic (PLEG): container finished" podID="27fc4a3f-9136-4433-885f-a15e46b30d2e" containerID="1a0f388736a28c72e321aa2a7bc4fa921f3c24aefc4d54a8cb7737c05d778bc7" exitCode=0
Sep 17 17:52:20.811746 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:52:20.811421    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp" event={"ID":"27fc4a3f-9136-4433-885f-a15e46b30d2e","Type":"ContainerDied","Data":"1a0f388736a28c72e321aa2a7bc4fa921f3c24aefc4d54a8cb7737c05d778bc7"}
Sep 17 17:52:20.824397 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:52:20.824375    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp"
Sep 17 17:52:20.988255 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:52:20.988181    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27fc4a3f-9136-4433-885f-a15e46b30d2e-proxy-tls\") pod \"27fc4a3f-9136-4433-885f-a15e46b30d2e\" (UID: \"27fc4a3f-9136-4433-885f-a15e46b30d2e\") "
Sep 17 17:52:20.988255 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:52:20.988225    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dzgqh\" (UniqueName: \"kubernetes.io/projected/27fc4a3f-9136-4433-885f-a15e46b30d2e-kube-api-access-dzgqh\") pod \"27fc4a3f-9136-4433-885f-a15e46b30d2e\" (UID: \"27fc4a3f-9136-4433-885f-a15e46b30d2e\") "
Sep 17 17:52:20.988255 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:52:20.988251    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27fc4a3f-9136-4433-885f-a15e46b30d2e-kserve-provision-location\") pod \"27fc4a3f-9136-4433-885f-a15e46b30d2e\" (UID: \"27fc4a3f-9136-4433-885f-a15e46b30d2e\") "
Sep 17 17:52:20.988542 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:52:20.988382    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27fc4a3f-9136-4433-885f-a15e46b30d2e-cabundle-cert\") pod \"27fc4a3f-9136-4433-885f-a15e46b30d2e\" (UID: \"27fc4a3f-9136-4433-885f-a15e46b30d2e\") "
Sep 17 17:52:20.988542 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:52:20.988442    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27fc4a3f-9136-4433-885f-a15e46b30d2e-kube-rbac-proxy-sar-config\") pod \"27fc4a3f-9136-4433-885f-a15e46b30d2e\" (UID: \"27fc4a3f-9136-4433-885f-a15e46b30d2e\") "
Sep 17 17:52:20.988652 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:52:20.988578    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27fc4a3f-9136-4433-885f-a15e46b30d2e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "27fc4a3f-9136-4433-885f-a15e46b30d2e" (UID: "27fc4a3f-9136-4433-885f-a15e46b30d2e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:52:20.988739 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:52:20.988720    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27fc4a3f-9136-4433-885f-a15e46b30d2e-kserve-provision-location\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 17:52:20.988854 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:52:20.988792    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27fc4a3f-9136-4433-885f-a15e46b30d2e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "27fc4a3f-9136-4433-885f-a15e46b30d2e" (UID: "27fc4a3f-9136-4433-885f-a15e46b30d2e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:52:20.988854 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:52:20.988798    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27fc4a3f-9136-4433-885f-a15e46b30d2e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "27fc4a3f-9136-4433-885f-a15e46b30d2e" (UID: "27fc4a3f-9136-4433-885f-a15e46b30d2e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:52:20.990785 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:52:20.990767    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27fc4a3f-9136-4433-885f-a15e46b30d2e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "27fc4a3f-9136-4433-885f-a15e46b30d2e" (UID: "27fc4a3f-9136-4433-885f-a15e46b30d2e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:52:20.990853 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:52:20.990783    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27fc4a3f-9136-4433-885f-a15e46b30d2e-kube-api-access-dzgqh" (OuterVolumeSpecName: "kube-api-access-dzgqh") pod "27fc4a3f-9136-4433-885f-a15e46b30d2e" (UID: "27fc4a3f-9136-4433-885f-a15e46b30d2e"). InnerVolumeSpecName "kube-api-access-dzgqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:52:21.089179 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:52:21.089150    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzgqh\" (UniqueName: \"kubernetes.io/projected/27fc4a3f-9136-4433-885f-a15e46b30d2e-kube-api-access-dzgqh\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 17:52:21.089179 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:52:21.089174    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27fc4a3f-9136-4433-885f-a15e46b30d2e-cabundle-cert\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 17:52:21.089179 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:52:21.089184    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27fc4a3f-9136-4433-885f-a15e46b30d2e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 17:52:21.089381 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:52:21.089193    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27fc4a3f-9136-4433-885f-a15e46b30d2e-proxy-tls\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 17:52:21.267729 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:52:21.267695    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:52:21.520605 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:52:21.520518    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:52:21.816467 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:52:21.816385    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp" event={"ID":"27fc4a3f-9136-4433-885f-a15e46b30d2e","Type":"ContainerDied","Data":"364c29c3396a5509b225660eb58b6f26517c3dcb67c07629b1c17ec6d47f5223"}
Sep 17 17:52:21.816467 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:52:21.816460    2585 scope.go:117] "RemoveContainer" containerID="3c90f56dcc64e7ab237933fd924443784f11530cbfb22fef823a72c20d03a00e"
Sep 17 17:52:21.816928 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:52:21.816394    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp"
Sep 17 17:52:21.824762 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:52:21.824746    2585 scope.go:117] "RemoveContainer" containerID="1a0f388736a28c72e321aa2a7bc4fa921f3c24aefc4d54a8cb7737c05d778bc7"
Sep 17 17:52:21.831537 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:52:21.831517    2585 scope.go:117] "RemoveContainer" containerID="037c3d2cae5f2d0d39fee6b70775a9ba0779ac6cf8681947d95908aaa6712d14"
Sep 17 17:52:21.843645 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:52:21.843620    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp"]
Sep 17 17:52:21.848745 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:52:21.848727    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-qs4pp"]
Sep 17 17:52:22.103811 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:52:22.103740    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27fc4a3f-9136-4433-885f-a15e46b30d2e" path="/var/lib/kubelet/pods/27fc4a3f-9136-4433-885f-a15e46b30d2e/volumes"
Sep 17 17:52:27.839954 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:52:27.839921    2585 generic.go:356] "Generic (PLEG): container finished" podID="3344339e-2d84-4ee1-b4e5-63cca80cfa48" containerID="dc626136f61103046d383183ee4fbe12ebe5b21aa9b0400ff3e1dfccb6fac095" exitCode=0
Sep 17 17:52:27.839954 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:52:27.839957    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7rv24" event={"ID":"3344339e-2d84-4ee1-b4e5-63cca80cfa48","Type":"ContainerDied","Data":"dc626136f61103046d383183ee4fbe12ebe5b21aa9b0400ff3e1dfccb6fac095"}
Sep 17 17:52:27.841117 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:52:27.841103    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 17:53:39.267136 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:53:39.267100    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:54:11.551713 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:54:11.551459    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:54:16.646383 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:54:16.646342    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:54:16.943890 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:54:16.943785    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:54:28.373587 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:54:28.373555    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rrf79_beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5/ovn-acl-logging/0.log"
Sep 17 17:54:28.373587 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:54:28.373588    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rrf79_beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5/ovn-acl-logging/0.log"
Sep 17 17:54:29.314411 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:54:29.314376    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7rv24" event={"ID":"3344339e-2d84-4ee1-b4e5-63cca80cfa48","Type":"ContainerStarted","Data":"9c8a28f2dd18c6d4016cf80b25ecfdbb21c7b521446e547866d7838f734a2fc0"}
Sep 17 17:54:29.314578 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:54:29.314431    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7rv24" event={"ID":"3344339e-2d84-4ee1-b4e5-63cca80cfa48","Type":"ContainerStarted","Data":"0b112805dd60681556238a5c88d65dba7932b129b31b1fa88720008e01719ea4"}
Sep 17 17:54:29.314631 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:54:29.314619    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7rv24"
Sep 17 17:54:29.341188 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:54:29.341132    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7rv24" podStartSLOduration=2.809488045 podStartE2EDuration="2m35.341113157s" totalImagesPullingTime="2m0.983217797s" totalInitContainerRuntime="31.548407315s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:51:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:54:29.338248514 +0000 UTC m=+1205.752038951" watchObservedRunningTime="2026-09-17 17:54:29.341113157 +0000 UTC m=+1205.754903607"
Sep 17 17:54:30.317859 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:54:30.317828    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7rv24"
Sep 17 17:54:30.319065 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:54:30.319041    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7rv24" podUID="3344339e-2d84-4ee1-b4e5-63cca80cfa48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 17 17:54:31.321736 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:54:31.321693    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7rv24" podUID="3344339e-2d84-4ee1-b4e5-63cca80cfa48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 17 17:54:36.329576 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:54:36.329535    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7rv24"
Sep 17 17:54:36.330228 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:54:36.330115    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7rv24" podUID="3344339e-2d84-4ee1-b4e5-63cca80cfa48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Sep 17 17:54:46.330995 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:54:46.330959    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7rv24"
Sep 17 17:55:00.749462 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:00.749430    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7rv24"]
Sep 17 17:55:00.749933 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:00.749659    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7rv24" podUID="3344339e-2d84-4ee1-b4e5-63cca80cfa48" containerName="kserve-container" containerID="cri-o://0b112805dd60681556238a5c88d65dba7932b129b31b1fa88720008e01719ea4" gracePeriod=30
Sep 17 17:55:00.749933 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:00.749713    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7rv24" podUID="3344339e-2d84-4ee1-b4e5-63cca80cfa48" containerName="kube-rbac-proxy" containerID="cri-o://9c8a28f2dd18c6d4016cf80b25ecfdbb21c7b521446e547866d7838f734a2fc0" gracePeriod=30
Sep 17 17:55:00.780059 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:00.780010    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:55:01.322945 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:01.322906    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7rv24" podUID="3344339e-2d84-4ee1-b4e5-63cca80cfa48" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": dial tcp 10.133.0.22:8643: connect: connection refused"
Sep 17 17:55:01.429587 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:01.429546    2585 generic.go:356] "Generic (PLEG): container finished" podID="3344339e-2d84-4ee1-b4e5-63cca80cfa48" containerID="9c8a28f2dd18c6d4016cf80b25ecfdbb21c7b521446e547866d7838f734a2fc0" exitCode=2
Sep 17 17:55:01.429739 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:01.429615    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7rv24" event={"ID":"3344339e-2d84-4ee1-b4e5-63cca80cfa48","Type":"ContainerDied","Data":"9c8a28f2dd18c6d4016cf80b25ecfdbb21c7b521446e547866d7838f734a2fc0"}
Sep 17 17:55:02.395944 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:02.208321    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7rv24"
Sep 17 17:55:02.395944 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:02.302824    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3344339e-2d84-4ee1-b4e5-63cca80cfa48-cabundle-cert\") pod \"3344339e-2d84-4ee1-b4e5-63cca80cfa48\" (UID: \"3344339e-2d84-4ee1-b4e5-63cca80cfa48\") "
Sep 17 17:55:02.395944 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:02.302862    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3344339e-2d84-4ee1-b4e5-63cca80cfa48-kserve-provision-location\") pod \"3344339e-2d84-4ee1-b4e5-63cca80cfa48\" (UID: \"3344339e-2d84-4ee1-b4e5-63cca80cfa48\") "
Sep 17 17:55:02.395944 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:02.302882    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3344339e-2d84-4ee1-b4e5-63cca80cfa48-proxy-tls\") pod \"3344339e-2d84-4ee1-b4e5-63cca80cfa48\" (UID: \"3344339e-2d84-4ee1-b4e5-63cca80cfa48\") "
Sep 17 17:55:02.395944 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:02.302924    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3344339e-2d84-4ee1-b4e5-63cca80cfa48-kube-rbac-proxy-sar-config\") pod \"3344339e-2d84-4ee1-b4e5-63cca80cfa48\" (UID: \"3344339e-2d84-4ee1-b4e5-63cca80cfa48\") "
Sep 17 17:55:02.395944 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:02.302953    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9mtq9\" (UniqueName: \"kubernetes.io/projected/3344339e-2d84-4ee1-b4e5-63cca80cfa48-kube-api-access-9mtq9\") pod \"3344339e-2d84-4ee1-b4e5-63cca80cfa48\" (UID: \"3344339e-2d84-4ee1-b4e5-63cca80cfa48\") "
Sep 17 17:55:02.395944 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:02.303258    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3344339e-2d84-4ee1-b4e5-63cca80cfa48-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3344339e-2d84-4ee1-b4e5-63cca80cfa48" (UID: "3344339e-2d84-4ee1-b4e5-63cca80cfa48"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:55:02.395944 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:02.303296    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3344339e-2d84-4ee1-b4e5-63cca80cfa48-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3344339e-2d84-4ee1-b4e5-63cca80cfa48" (UID: "3344339e-2d84-4ee1-b4e5-63cca80cfa48"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:55:02.395944 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:02.305916    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3344339e-2d84-4ee1-b4e5-63cca80cfa48-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3344339e-2d84-4ee1-b4e5-63cca80cfa48" (UID: "3344339e-2d84-4ee1-b4e5-63cca80cfa48"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:55:02.395944 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:02.306056    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3344339e-2d84-4ee1-b4e5-63cca80cfa48-kube-api-access-9mtq9" (OuterVolumeSpecName: "kube-api-access-9mtq9") pod "3344339e-2d84-4ee1-b4e5-63cca80cfa48" (UID: "3344339e-2d84-4ee1-b4e5-63cca80cfa48"). InnerVolumeSpecName "kube-api-access-9mtq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:55:02.395944 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:02.308408    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3344339e-2d84-4ee1-b4e5-63cca80cfa48-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3344339e-2d84-4ee1-b4e5-63cca80cfa48" (UID: "3344339e-2d84-4ee1-b4e5-63cca80cfa48"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:55:02.404333 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:02.404306    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3344339e-2d84-4ee1-b4e5-63cca80cfa48-proxy-tls\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 17:55:02.404333 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:02.404330    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3344339e-2d84-4ee1-b4e5-63cca80cfa48-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 17:55:02.404476 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:02.404340    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mtq9\" (UniqueName: \"kubernetes.io/projected/3344339e-2d84-4ee1-b4e5-63cca80cfa48-kube-api-access-9mtq9\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 17:55:02.404476 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:02.404349    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3344339e-2d84-4ee1-b4e5-63cca80cfa48-cabundle-cert\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 17:55:02.404476 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:02.404360    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3344339e-2d84-4ee1-b4e5-63cca80cfa48-kserve-provision-location\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 17:55:02.434911 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:02.434878    2585 generic.go:356] "Generic (PLEG): container finished" podID="3344339e-2d84-4ee1-b4e5-63cca80cfa48" containerID="0b112805dd60681556238a5c88d65dba7932b129b31b1fa88720008e01719ea4" exitCode=0
Sep 17 17:55:02.435024 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:02.434944    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7rv24"
Sep 17 17:55:02.435024 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:02.434961    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7rv24" event={"ID":"3344339e-2d84-4ee1-b4e5-63cca80cfa48","Type":"ContainerDied","Data":"0b112805dd60681556238a5c88d65dba7932b129b31b1fa88720008e01719ea4"}
Sep 17 17:55:02.435094 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:02.435025    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7rv24" event={"ID":"3344339e-2d84-4ee1-b4e5-63cca80cfa48","Type":"ContainerDied","Data":"687ae5c5070fe43c783137bce26ce69f5642f8ed12d9aedbfa143239dd495910"}
Sep 17 17:55:02.435094 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:02.435053    2585 scope.go:117] "RemoveContainer" containerID="9c8a28f2dd18c6d4016cf80b25ecfdbb21c7b521446e547866d7838f734a2fc0"
Sep 17 17:55:02.447948 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:02.447931    2585 scope.go:117] "RemoveContainer" containerID="0b112805dd60681556238a5c88d65dba7932b129b31b1fa88720008e01719ea4"
Sep 17 17:55:02.454728 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:02.454712    2585 scope.go:117] "RemoveContainer" containerID="dc626136f61103046d383183ee4fbe12ebe5b21aa9b0400ff3e1dfccb6fac095"
Sep 17 17:55:02.461647 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:02.461623    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7rv24"]
Sep 17 17:55:02.461647 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:02.461646    2585 scope.go:117] "RemoveContainer" containerID="9c8a28f2dd18c6d4016cf80b25ecfdbb21c7b521446e547866d7838f734a2fc0"
Sep 17 17:55:02.461915 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:55:02.461894    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c8a28f2dd18c6d4016cf80b25ecfdbb21c7b521446e547866d7838f734a2fc0\": container with ID starting with 9c8a28f2dd18c6d4016cf80b25ecfdbb21c7b521446e547866d7838f734a2fc0 not found: ID does not exist" containerID="9c8a28f2dd18c6d4016cf80b25ecfdbb21c7b521446e547866d7838f734a2fc0"
Sep 17 17:55:02.462028 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:02.461919    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c8a28f2dd18c6d4016cf80b25ecfdbb21c7b521446e547866d7838f734a2fc0"} err="failed to get container status \"9c8a28f2dd18c6d4016cf80b25ecfdbb21c7b521446e547866d7838f734a2fc0\": rpc error: code = NotFound desc = could not find container \"9c8a28f2dd18c6d4016cf80b25ecfdbb21c7b521446e547866d7838f734a2fc0\": container with ID starting with 9c8a28f2dd18c6d4016cf80b25ecfdbb21c7b521446e547866d7838f734a2fc0 not found: ID does not exist"
Sep 17 17:55:02.462028 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:02.461936    2585 scope.go:117] "RemoveContainer" containerID="0b112805dd60681556238a5c88d65dba7932b129b31b1fa88720008e01719ea4"
Sep 17 17:55:02.462206 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:55:02.462177    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b112805dd60681556238a5c88d65dba7932b129b31b1fa88720008e01719ea4\": container with ID starting with 0b112805dd60681556238a5c88d65dba7932b129b31b1fa88720008e01719ea4 not found: ID does not exist" containerID="0b112805dd60681556238a5c88d65dba7932b129b31b1fa88720008e01719ea4"
Sep 17 17:55:02.462286 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:02.462215    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b112805dd60681556238a5c88d65dba7932b129b31b1fa88720008e01719ea4"} err="failed to get container status \"0b112805dd60681556238a5c88d65dba7932b129b31b1fa88720008e01719ea4\": rpc error: code = NotFound desc = could not find container \"0b112805dd60681556238a5c88d65dba7932b129b31b1fa88720008e01719ea4\": container with ID starting with 0b112805dd60681556238a5c88d65dba7932b129b31b1fa88720008e01719ea4 not found: ID does not exist"
Sep 17 17:55:02.462286 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:02.462230    2585 scope.go:117] "RemoveContainer" containerID="dc626136f61103046d383183ee4fbe12ebe5b21aa9b0400ff3e1dfccb6fac095"
Sep 17 17:55:02.462557 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:55:02.462531    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc626136f61103046d383183ee4fbe12ebe5b21aa9b0400ff3e1dfccb6fac095\": container with ID starting with dc626136f61103046d383183ee4fbe12ebe5b21aa9b0400ff3e1dfccb6fac095 not found: ID does not exist" containerID="dc626136f61103046d383183ee4fbe12ebe5b21aa9b0400ff3e1dfccb6fac095"
Sep 17 17:55:02.462699 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:02.462675    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc626136f61103046d383183ee4fbe12ebe5b21aa9b0400ff3e1dfccb6fac095"} err="failed to get container status \"dc626136f61103046d383183ee4fbe12ebe5b21aa9b0400ff3e1dfccb6fac095\": rpc error: code = NotFound desc = could not find container \"dc626136f61103046d383183ee4fbe12ebe5b21aa9b0400ff3e1dfccb6fac095\": container with ID starting with dc626136f61103046d383183ee4fbe12ebe5b21aa9b0400ff3e1dfccb6fac095 not found: ID does not exist"
Sep 17 17:55:02.464018 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:02.464000    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-7rv24"]
Sep 17 17:55:04.103189 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:04.103147    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3344339e-2d84-4ee1-b4e5-63cca80cfa48" path="/var/lib/kubelet/pods/3344339e-2d84-4ee1-b4e5-63cca80cfa48/volumes"
Sep 17 17:55:05.846732 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:05.846702    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:55:05.890246 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:05.890214    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sqrhq"]
Sep 17 17:55:05.890475 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:05.890462    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3344339e-2d84-4ee1-b4e5-63cca80cfa48" containerName="storage-initializer"
Sep 17 17:55:05.890541 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:05.890480    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="3344339e-2d84-4ee1-b4e5-63cca80cfa48" containerName="storage-initializer"
Sep 17 17:55:05.890541 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:05.890506    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a10431ef-8b50-4ddb-ab94-e06de1a670fd" containerName="kserve-container"
Sep 17 17:55:05.890541 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:05.890511    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="a10431ef-8b50-4ddb-ab94-e06de1a670fd" containerName="kserve-container"
Sep 17 17:55:05.890541 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:05.890517    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a10431ef-8b50-4ddb-ab94-e06de1a670fd" containerName="kube-rbac-proxy"
Sep 17 17:55:05.890541 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:05.890522    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="a10431ef-8b50-4ddb-ab94-e06de1a670fd" containerName="kube-rbac-proxy"
Sep 17 17:55:05.890541 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:05.890532    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3344339e-2d84-4ee1-b4e5-63cca80cfa48" containerName="kube-rbac-proxy"
Sep 17 17:55:05.890541 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:05.890538    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="3344339e-2d84-4ee1-b4e5-63cca80cfa48" containerName="kube-rbac-proxy"
Sep 17 17:55:05.890740 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:05.890545    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3344339e-2d84-4ee1-b4e5-63cca80cfa48" containerName="kserve-container"
Sep 17 17:55:05.890740 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:05.890553    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="3344339e-2d84-4ee1-b4e5-63cca80cfa48" containerName="kserve-container"
Sep 17 17:55:05.890740 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:05.890568    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27fc4a3f-9136-4433-885f-a15e46b30d2e" containerName="storage-initializer"
Sep 17 17:55:05.890740 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:05.890573    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="27fc4a3f-9136-4433-885f-a15e46b30d2e" containerName="storage-initializer"
Sep 17 17:55:05.890740 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:05.890579    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27fc4a3f-9136-4433-885f-a15e46b30d2e" containerName="kserve-container"
Sep 17 17:55:05.890740 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:05.890584    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="27fc4a3f-9136-4433-885f-a15e46b30d2e" containerName="kserve-container"
Sep 17 17:55:05.890740 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:05.890589    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27fc4a3f-9136-4433-885f-a15e46b30d2e" containerName="kube-rbac-proxy"
Sep 17 17:55:05.890740 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:05.890594    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="27fc4a3f-9136-4433-885f-a15e46b30d2e" containerName="kube-rbac-proxy"
Sep 17 17:55:05.890740 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:05.890637    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="27fc4a3f-9136-4433-885f-a15e46b30d2e" containerName="kserve-container"
Sep 17 17:55:05.890740 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:05.890646    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="a10431ef-8b50-4ddb-ab94-e06de1a670fd" containerName="kserve-container"
Sep 17 17:55:05.890740 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:05.890651    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="a10431ef-8b50-4ddb-ab94-e06de1a670fd" containerName="kube-rbac-proxy"
Sep 17 17:55:05.890740 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:05.890655    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="3344339e-2d84-4ee1-b4e5-63cca80cfa48" containerName="kube-rbac-proxy"
Sep 17 17:55:05.890740 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:05.890660    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="27fc4a3f-9136-4433-885f-a15e46b30d2e" containerName="kube-rbac-proxy"
Sep 17 17:55:05.890740 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:05.890664    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="3344339e-2d84-4ee1-b4e5-63cca80cfa48" containerName="kserve-container"
Sep 17 17:55:05.893943 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:05.893926    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sqrhq"
Sep 17 17:55:05.896598 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:05.896580    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:55:05.896817 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:05.896792    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Sep 17 17:55:05.896817 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:05.896806    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lps5w\""
Sep 17 17:55:05.896991 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:05.896978    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:55:05.897521 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:05.897503    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:55:05.897585 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:05.897555    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Sep 17 17:55:05.897689 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:05.897675    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:55:05.908317 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:05.908296    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sqrhq"]
Sep 17 17:55:05.996653 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:05.996615    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:55:06.028625 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:06.028594    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff59e744-2a7e-48ed-b84e-5a2f40ac0c36-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-sqrhq\" (UID: \"ff59e744-2a7e-48ed-b84e-5a2f40ac0c36\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sqrhq"
Sep 17 17:55:06.028760 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:06.028629    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf5hq\" (UniqueName: \"kubernetes.io/projected/ff59e744-2a7e-48ed-b84e-5a2f40ac0c36-kube-api-access-bf5hq\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-sqrhq\" (UID: \"ff59e744-2a7e-48ed-b84e-5a2f40ac0c36\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sqrhq"
Sep 17 17:55:06.028760 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:06.028658    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff59e744-2a7e-48ed-b84e-5a2f40ac0c36-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-sqrhq\" (UID: \"ff59e744-2a7e-48ed-b84e-5a2f40ac0c36\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sqrhq"
Sep 17 17:55:06.028868 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:06.028755    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff59e744-2a7e-48ed-b84e-5a2f40ac0c36-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-sqrhq\" (UID: \"ff59e744-2a7e-48ed-b84e-5a2f40ac0c36\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sqrhq"
Sep 17 17:55:06.028868 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:06.028779    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff59e744-2a7e-48ed-b84e-5a2f40ac0c36-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-sqrhq\" (UID: \"ff59e744-2a7e-48ed-b84e-5a2f40ac0c36\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sqrhq"
Sep 17 17:55:06.130045 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:06.129976    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff59e744-2a7e-48ed-b84e-5a2f40ac0c36-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-sqrhq\" (UID: \"ff59e744-2a7e-48ed-b84e-5a2f40ac0c36\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sqrhq"
Sep 17 17:55:06.130045 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:06.130009    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf5hq\" (UniqueName: \"kubernetes.io/projected/ff59e744-2a7e-48ed-b84e-5a2f40ac0c36-kube-api-access-bf5hq\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-sqrhq\" (UID: \"ff59e744-2a7e-48ed-b84e-5a2f40ac0c36\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sqrhq"
Sep 17 17:55:06.130045 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:06.130037    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff59e744-2a7e-48ed-b84e-5a2f40ac0c36-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-sqrhq\" (UID: \"ff59e744-2a7e-48ed-b84e-5a2f40ac0c36\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sqrhq"
Sep 17 17:55:06.130248 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:06.130078    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff59e744-2a7e-48ed-b84e-5a2f40ac0c36-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-sqrhq\" (UID: \"ff59e744-2a7e-48ed-b84e-5a2f40ac0c36\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sqrhq"
Sep 17 17:55:06.130248 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:06.130105    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff59e744-2a7e-48ed-b84e-5a2f40ac0c36-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-sqrhq\" (UID: \"ff59e744-2a7e-48ed-b84e-5a2f40ac0c36\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sqrhq"
Sep 17 17:55:06.130399 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:06.130382    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff59e744-2a7e-48ed-b84e-5a2f40ac0c36-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-sqrhq\" (UID: \"ff59e744-2a7e-48ed-b84e-5a2f40ac0c36\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sqrhq"
Sep 17 17:55:06.130759 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:06.130735    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff59e744-2a7e-48ed-b84e-5a2f40ac0c36-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-sqrhq\" (UID: \"ff59e744-2a7e-48ed-b84e-5a2f40ac0c36\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sqrhq"
Sep 17 17:55:06.130759 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:06.130746    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff59e744-2a7e-48ed-b84e-5a2f40ac0c36-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-sqrhq\" (UID: \"ff59e744-2a7e-48ed-b84e-5a2f40ac0c36\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sqrhq"
Sep 17 17:55:06.132522 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:06.132507    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff59e744-2a7e-48ed-b84e-5a2f40ac0c36-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-sqrhq\" (UID: \"ff59e744-2a7e-48ed-b84e-5a2f40ac0c36\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sqrhq"
Sep 17 17:55:06.139756 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:06.139739    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf5hq\" (UniqueName: \"kubernetes.io/projected/ff59e744-2a7e-48ed-b84e-5a2f40ac0c36-kube-api-access-bf5hq\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-sqrhq\" (UID: \"ff59e744-2a7e-48ed-b84e-5a2f40ac0c36\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sqrhq"
Sep 17 17:55:06.203773 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:06.203730    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sqrhq"
Sep 17 17:55:06.497509 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:06.497459    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sqrhq"]
Sep 17 17:55:06.502041 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:55:06.502009    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff59e744_2a7e_48ed_b84e_5a2f40ac0c36.slice/crio-9cbee41560d885d32288018576621da6f3088629b9baf63f6203ae56bd33f551 WatchSource:0}: Error finding container 9cbee41560d885d32288018576621da6f3088629b9baf63f6203ae56bd33f551: Status 404 returned error can't find the container with id 9cbee41560d885d32288018576621da6f3088629b9baf63f6203ae56bd33f551
Sep 17 17:55:07.453973 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:07.453938    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sqrhq" event={"ID":"ff59e744-2a7e-48ed-b84e-5a2f40ac0c36","Type":"ContainerStarted","Data":"4e13773d628831a377f6f5f1552f46286ed5236be517f75fba65a1e5806e3add"}
Sep 17 17:55:07.454307 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:07.454015    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sqrhq" event={"ID":"ff59e744-2a7e-48ed-b84e-5a2f40ac0c36","Type":"ContainerStarted","Data":"9cbee41560d885d32288018576621da6f3088629b9baf63f6203ae56bd33f551"}
Sep 17 17:55:27.529479 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:27.529446    2585 generic.go:356] "Generic (PLEG): container finished" podID="ff59e744-2a7e-48ed-b84e-5a2f40ac0c36" containerID="4e13773d628831a377f6f5f1552f46286ed5236be517f75fba65a1e5806e3add" exitCode=0
Sep 17 17:55:27.529874 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:27.529514    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sqrhq" event={"ID":"ff59e744-2a7e-48ed-b84e-5a2f40ac0c36","Type":"ContainerDied","Data":"4e13773d628831a377f6f5f1552f46286ed5236be517f75fba65a1e5806e3add"}
Sep 17 17:55:28.534817 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:28.534782    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sqrhq" event={"ID":"ff59e744-2a7e-48ed-b84e-5a2f40ac0c36","Type":"ContainerStarted","Data":"30d5630ddbc56acb88a520d53ec8631e039d001a0678b4ab71c4e00b0f1e110d"}
Sep 17 17:55:28.535401 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:28.534861    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sqrhq" event={"ID":"ff59e744-2a7e-48ed-b84e-5a2f40ac0c36","Type":"ContainerStarted","Data":"3b79307a93907c1142557ba8f0a826ec3dba0be49dcaaeb697011661d47cb05a"}
Sep 17 17:55:28.535401 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:28.535129    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sqrhq"
Sep 17 17:55:28.565436 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:28.565380    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sqrhq" podStartSLOduration=3.19378436 podStartE2EDuration="23.565365332s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.371580972s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:55:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:55:28.563727401 +0000 UTC m=+1264.977517846" watchObservedRunningTime="2026-09-17 17:55:28.565365332 +0000 UTC m=+1264.979155770"
Sep 17 17:55:29.538425 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:29.538395    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sqrhq"
Sep 17 17:55:35.548098 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:35.548064    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sqrhq"
Sep 17 17:55:46.789760 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:46.789723    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:55:51.899375 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:51.899342    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:55:52.325336 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:55:52.325299    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:05.131929 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:05.131859    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sqrhq"
Sep 17 17:57:11.839826 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:11.839792    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sqrhq"]
Sep 17 17:57:11.840345 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:11.840107    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sqrhq" podUID="ff59e744-2a7e-48ed-b84e-5a2f40ac0c36" containerName="kserve-container" containerID="cri-o://3b79307a93907c1142557ba8f0a826ec3dba0be49dcaaeb697011661d47cb05a" gracePeriod=30
Sep 17 17:57:11.840345 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:11.840122    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sqrhq" podUID="ff59e744-2a7e-48ed-b84e-5a2f40ac0c36" containerName="kube-rbac-proxy" containerID="cri-o://30d5630ddbc56acb88a520d53ec8631e039d001a0678b4ab71c4e00b0f1e110d" gracePeriod=30
Sep 17 17:57:11.866535 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:11.866509    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:12.903440 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:12.903409    2585 generic.go:356] "Generic (PLEG): container finished" podID="ff59e744-2a7e-48ed-b84e-5a2f40ac0c36" containerID="30d5630ddbc56acb88a520d53ec8631e039d001a0678b4ab71c4e00b0f1e110d" exitCode=2
Sep 17 17:57:12.903867 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:12.903463    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sqrhq" event={"ID":"ff59e744-2a7e-48ed-b84e-5a2f40ac0c36","Type":"ContainerDied","Data":"30d5630ddbc56acb88a520d53ec8631e039d001a0678b4ab71c4e00b0f1e110d"}
Sep 17 17:57:13.271252 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:13.271229    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sqrhq"
Sep 17 17:57:13.419560 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:13.419457    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bf5hq\" (UniqueName: \"kubernetes.io/projected/ff59e744-2a7e-48ed-b84e-5a2f40ac0c36-kube-api-access-bf5hq\") pod \"ff59e744-2a7e-48ed-b84e-5a2f40ac0c36\" (UID: \"ff59e744-2a7e-48ed-b84e-5a2f40ac0c36\") "
Sep 17 17:57:13.419560 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:13.419529    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff59e744-2a7e-48ed-b84e-5a2f40ac0c36-kube-rbac-proxy-sar-config\") pod \"ff59e744-2a7e-48ed-b84e-5a2f40ac0c36\" (UID: \"ff59e744-2a7e-48ed-b84e-5a2f40ac0c36\") "
Sep 17 17:57:13.419560 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:13.419562    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff59e744-2a7e-48ed-b84e-5a2f40ac0c36-proxy-tls\") pod \"ff59e744-2a7e-48ed-b84e-5a2f40ac0c36\" (UID: \"ff59e744-2a7e-48ed-b84e-5a2f40ac0c36\") "
Sep 17 17:57:13.419841 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:13.419606    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff59e744-2a7e-48ed-b84e-5a2f40ac0c36-cabundle-cert\") pod \"ff59e744-2a7e-48ed-b84e-5a2f40ac0c36\" (UID: \"ff59e744-2a7e-48ed-b84e-5a2f40ac0c36\") "
Sep 17 17:57:13.419841 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:13.419675    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff59e744-2a7e-48ed-b84e-5a2f40ac0c36-kserve-provision-location\") pod \"ff59e744-2a7e-48ed-b84e-5a2f40ac0c36\" (UID: \"ff59e744-2a7e-48ed-b84e-5a2f40ac0c36\") "
Sep 17 17:57:13.419941 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:13.419905    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff59e744-2a7e-48ed-b84e-5a2f40ac0c36-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ff59e744-2a7e-48ed-b84e-5a2f40ac0c36" (UID: "ff59e744-2a7e-48ed-b84e-5a2f40ac0c36"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:57:13.419980 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:13.419941    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff59e744-2a7e-48ed-b84e-5a2f40ac0c36-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ff59e744-2a7e-48ed-b84e-5a2f40ac0c36" (UID: "ff59e744-2a7e-48ed-b84e-5a2f40ac0c36"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:57:13.421642 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:13.421612    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff59e744-2a7e-48ed-b84e-5a2f40ac0c36-kube-api-access-bf5hq" (OuterVolumeSpecName: "kube-api-access-bf5hq") pod "ff59e744-2a7e-48ed-b84e-5a2f40ac0c36" (UID: "ff59e744-2a7e-48ed-b84e-5a2f40ac0c36"). InnerVolumeSpecName "kube-api-access-bf5hq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:57:13.421747 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:13.421659    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff59e744-2a7e-48ed-b84e-5a2f40ac0c36-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ff59e744-2a7e-48ed-b84e-5a2f40ac0c36" (UID: "ff59e744-2a7e-48ed-b84e-5a2f40ac0c36"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:57:13.423955 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:13.423932    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff59e744-2a7e-48ed-b84e-5a2f40ac0c36-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ff59e744-2a7e-48ed-b84e-5a2f40ac0c36" (UID: "ff59e744-2a7e-48ed-b84e-5a2f40ac0c36"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:57:13.521073 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:13.521038    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff59e744-2a7e-48ed-b84e-5a2f40ac0c36-cabundle-cert\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 17:57:13.521073 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:13.521066    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff59e744-2a7e-48ed-b84e-5a2f40ac0c36-kserve-provision-location\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 17:57:13.521073 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:13.521076    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bf5hq\" (UniqueName: \"kubernetes.io/projected/ff59e744-2a7e-48ed-b84e-5a2f40ac0c36-kube-api-access-bf5hq\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 17:57:13.521073 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:13.521084    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff59e744-2a7e-48ed-b84e-5a2f40ac0c36-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 17:57:13.521324 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:13.521094    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff59e744-2a7e-48ed-b84e-5a2f40ac0c36-proxy-tls\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 17:57:13.909094 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:13.909057    2585 generic.go:356] "Generic (PLEG): container finished" podID="ff59e744-2a7e-48ed-b84e-5a2f40ac0c36" containerID="3b79307a93907c1142557ba8f0a826ec3dba0be49dcaaeb697011661d47cb05a" exitCode=0
Sep 17 17:57:13.909511 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:13.909099    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sqrhq" event={"ID":"ff59e744-2a7e-48ed-b84e-5a2f40ac0c36","Type":"ContainerDied","Data":"3b79307a93907c1142557ba8f0a826ec3dba0be49dcaaeb697011661d47cb05a"}
Sep 17 17:57:13.909511 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:13.909120    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sqrhq"
Sep 17 17:57:13.909511 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:13.909161    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sqrhq" event={"ID":"ff59e744-2a7e-48ed-b84e-5a2f40ac0c36","Type":"ContainerDied","Data":"9cbee41560d885d32288018576621da6f3088629b9baf63f6203ae56bd33f551"}
Sep 17 17:57:13.909511 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:13.909190    2585 scope.go:117] "RemoveContainer" containerID="30d5630ddbc56acb88a520d53ec8631e039d001a0678b4ab71c4e00b0f1e110d"
Sep 17 17:57:13.916830 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:13.916813    2585 scope.go:117] "RemoveContainer" containerID="3b79307a93907c1142557ba8f0a826ec3dba0be49dcaaeb697011661d47cb05a"
Sep 17 17:57:13.923610 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:13.923591    2585 scope.go:117] "RemoveContainer" containerID="4e13773d628831a377f6f5f1552f46286ed5236be517f75fba65a1e5806e3add"
Sep 17 17:57:13.930167 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:13.930143    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sqrhq"]
Sep 17 17:57:13.930958 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:13.930939    2585 scope.go:117] "RemoveContainer" containerID="30d5630ddbc56acb88a520d53ec8631e039d001a0678b4ab71c4e00b0f1e110d"
Sep 17 17:57:13.931206 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:57:13.931189    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30d5630ddbc56acb88a520d53ec8631e039d001a0678b4ab71c4e00b0f1e110d\": container with ID starting with 30d5630ddbc56acb88a520d53ec8631e039d001a0678b4ab71c4e00b0f1e110d not found: ID does not exist" containerID="30d5630ddbc56acb88a520d53ec8631e039d001a0678b4ab71c4e00b0f1e110d"
Sep 17 17:57:13.931254 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:13.931214    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30d5630ddbc56acb88a520d53ec8631e039d001a0678b4ab71c4e00b0f1e110d"} err="failed to get container status \"30d5630ddbc56acb88a520d53ec8631e039d001a0678b4ab71c4e00b0f1e110d\": rpc error: code = NotFound desc = could not find container \"30d5630ddbc56acb88a520d53ec8631e039d001a0678b4ab71c4e00b0f1e110d\": container with ID starting with 30d5630ddbc56acb88a520d53ec8631e039d001a0678b4ab71c4e00b0f1e110d not found: ID does not exist"
Sep 17 17:57:13.931254 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:13.931230    2585 scope.go:117] "RemoveContainer" containerID="3b79307a93907c1142557ba8f0a826ec3dba0be49dcaaeb697011661d47cb05a"
Sep 17 17:57:13.931471 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:57:13.931455    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b79307a93907c1142557ba8f0a826ec3dba0be49dcaaeb697011661d47cb05a\": container with ID starting with 3b79307a93907c1142557ba8f0a826ec3dba0be49dcaaeb697011661d47cb05a not found: ID does not exist" containerID="3b79307a93907c1142557ba8f0a826ec3dba0be49dcaaeb697011661d47cb05a"
Sep 17 17:57:13.931528 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:13.931476    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b79307a93907c1142557ba8f0a826ec3dba0be49dcaaeb697011661d47cb05a"} err="failed to get container status \"3b79307a93907c1142557ba8f0a826ec3dba0be49dcaaeb697011661d47cb05a\": rpc error: code = NotFound desc = could not find container \"3b79307a93907c1142557ba8f0a826ec3dba0be49dcaaeb697011661d47cb05a\": container with ID starting with 3b79307a93907c1142557ba8f0a826ec3dba0be49dcaaeb697011661d47cb05a not found: ID does not exist"
Sep 17 17:57:13.931528 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:13.931505    2585 scope.go:117] "RemoveContainer" containerID="4e13773d628831a377f6f5f1552f46286ed5236be517f75fba65a1e5806e3add"
Sep 17 17:57:13.931699 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:57:13.931685    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e13773d628831a377f6f5f1552f46286ed5236be517f75fba65a1e5806e3add\": container with ID starting with 4e13773d628831a377f6f5f1552f46286ed5236be517f75fba65a1e5806e3add not found: ID does not exist" containerID="4e13773d628831a377f6f5f1552f46286ed5236be517f75fba65a1e5806e3add"
Sep 17 17:57:13.931737 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:13.931703    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e13773d628831a377f6f5f1552f46286ed5236be517f75fba65a1e5806e3add"} err="failed to get container status \"4e13773d628831a377f6f5f1552f46286ed5236be517f75fba65a1e5806e3add\": rpc error: code = NotFound desc = could not find container \"4e13773d628831a377f6f5f1552f46286ed5236be517f75fba65a1e5806e3add\": container with ID starting with 4e13773d628831a377f6f5f1552f46286ed5236be517f75fba65a1e5806e3add not found: ID does not exist"
Sep 17 17:57:13.935138 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:13.935116    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-sqrhq"]
Sep 17 17:57:14.104367 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:14.104335    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff59e744-2a7e-48ed-b84e-5a2f40ac0c36" path="/var/lib/kubelet/pods/ff59e744-2a7e-48ed-b84e-5a2f40ac0c36/volumes"
Sep 17 17:57:17.021422 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:17.021392    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:17.072074 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:17.072040    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-vqnpj"]
Sep 17 17:57:17.072298 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:17.072287    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff59e744-2a7e-48ed-b84e-5a2f40ac0c36" containerName="kube-rbac-proxy"
Sep 17 17:57:17.072365 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:17.072301    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff59e744-2a7e-48ed-b84e-5a2f40ac0c36" containerName="kube-rbac-proxy"
Sep 17 17:57:17.072365 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:17.072317    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff59e744-2a7e-48ed-b84e-5a2f40ac0c36" containerName="kserve-container"
Sep 17 17:57:17.072365 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:17.072323    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff59e744-2a7e-48ed-b84e-5a2f40ac0c36" containerName="kserve-container"
Sep 17 17:57:17.072365 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:17.072329    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff59e744-2a7e-48ed-b84e-5a2f40ac0c36" containerName="storage-initializer"
Sep 17 17:57:17.072365 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:17.072337    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff59e744-2a7e-48ed-b84e-5a2f40ac0c36" containerName="storage-initializer"
Sep 17 17:57:17.072536 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:17.072379    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff59e744-2a7e-48ed-b84e-5a2f40ac0c36" containerName="kube-rbac-proxy"
Sep 17 17:57:17.072536 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:17.072391    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff59e744-2a7e-48ed-b84e-5a2f40ac0c36" containerName="kserve-container"
Sep 17 17:57:17.077094 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:17.077075    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-vqnpj"
Sep 17 17:57:17.080754 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:17.080735    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:57:17.080865 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:17.080825    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lps5w\""
Sep 17 17:57:17.080969 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:17.080910    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Sep 17 17:57:17.081022 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:17.080987    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Sep 17 17:57:17.081886 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:17.081868    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:57:17.081965 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:17.081900    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:57:17.081965 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:17.081934    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:57:17.106023 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:17.105997    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-vqnpj"]
Sep 17 17:57:17.144259 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:17.144222    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxpwv\" (UniqueName: \"kubernetes.io/projected/5f2298ac-7de6-4188-812f-d496c8afb895-kube-api-access-sxpwv\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-vqnpj\" (UID: \"5f2298ac-7de6-4188-812f-d496c8afb895\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-vqnpj"
Sep 17 17:57:17.144394 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:17.144271    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f2298ac-7de6-4188-812f-d496c8afb895-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-vqnpj\" (UID: \"5f2298ac-7de6-4188-812f-d496c8afb895\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-vqnpj"
Sep 17 17:57:17.144394 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:17.144309    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f2298ac-7de6-4188-812f-d496c8afb895-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-vqnpj\" (UID: \"5f2298ac-7de6-4188-812f-d496c8afb895\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-vqnpj"
Sep 17 17:57:17.144394 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:17.144345    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f2298ac-7de6-4188-812f-d496c8afb895-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-vqnpj\" (UID: \"5f2298ac-7de6-4188-812f-d496c8afb895\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-vqnpj"
Sep 17 17:57:17.144394 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:17.144378    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f2298ac-7de6-4188-812f-d496c8afb895-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-vqnpj\" (UID: \"5f2298ac-7de6-4188-812f-d496c8afb895\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-vqnpj"
Sep 17 17:57:17.206926 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:17.206891    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:17.244925 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:17.244898    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxpwv\" (UniqueName: \"kubernetes.io/projected/5f2298ac-7de6-4188-812f-d496c8afb895-kube-api-access-sxpwv\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-vqnpj\" (UID: \"5f2298ac-7de6-4188-812f-d496c8afb895\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-vqnpj"
Sep 17 17:57:17.245055 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:17.244931    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f2298ac-7de6-4188-812f-d496c8afb895-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-vqnpj\" (UID: \"5f2298ac-7de6-4188-812f-d496c8afb895\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-vqnpj"
Sep 17 17:57:17.245055 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:17.244961    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f2298ac-7de6-4188-812f-d496c8afb895-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-vqnpj\" (UID: \"5f2298ac-7de6-4188-812f-d496c8afb895\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-vqnpj"
Sep 17 17:57:17.245055 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:17.244992    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f2298ac-7de6-4188-812f-d496c8afb895-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-vqnpj\" (UID: \"5f2298ac-7de6-4188-812f-d496c8afb895\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-vqnpj"
Sep 17 17:57:17.245055 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:17.245022    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f2298ac-7de6-4188-812f-d496c8afb895-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-vqnpj\" (UID: \"5f2298ac-7de6-4188-812f-d496c8afb895\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-vqnpj"
Sep 17 17:57:17.245404 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:17.245386    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f2298ac-7de6-4188-812f-d496c8afb895-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-vqnpj\" (UID: \"5f2298ac-7de6-4188-812f-d496c8afb895\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-vqnpj"
Sep 17 17:57:17.245596 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:17.245579    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f2298ac-7de6-4188-812f-d496c8afb895-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-vqnpj\" (UID: \"5f2298ac-7de6-4188-812f-d496c8afb895\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-vqnpj"
Sep 17 17:57:17.245673 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:17.245658    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f2298ac-7de6-4188-812f-d496c8afb895-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-vqnpj\" (UID: \"5f2298ac-7de6-4188-812f-d496c8afb895\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-vqnpj"
Sep 17 17:57:17.247331 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:17.247317    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f2298ac-7de6-4188-812f-d496c8afb895-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-vqnpj\" (UID: \"5f2298ac-7de6-4188-812f-d496c8afb895\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-vqnpj"
Sep 17 17:57:17.254035 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:17.254012    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxpwv\" (UniqueName: \"kubernetes.io/projected/5f2298ac-7de6-4188-812f-d496c8afb895-kube-api-access-sxpwv\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-vqnpj\" (UID: \"5f2298ac-7de6-4188-812f-d496c8afb895\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-vqnpj"
Sep 17 17:57:17.386913 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:17.386889    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-vqnpj"
Sep 17 17:57:17.661957 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:17.661935    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-vqnpj"]
Sep 17 17:57:17.664117 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:57:17.664091    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f2298ac_7de6_4188_812f_d496c8afb895.slice/crio-322158983efe9699e881e54e550465107603329ea503552a4a337263b2c1f8a4 WatchSource:0}: Error finding container 322158983efe9699e881e54e550465107603329ea503552a4a337263b2c1f8a4: Status 404 returned error can't find the container with id 322158983efe9699e881e54e550465107603329ea503552a4a337263b2c1f8a4
Sep 17 17:57:17.926784 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:17.926695    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-vqnpj" event={"ID":"5f2298ac-7de6-4188-812f-d496c8afb895","Type":"ContainerStarted","Data":"dd06a32228a54030894527d8b950f9e4f347484cd72aa44d6f63df83d34d9faf"}
Sep 17 17:57:17.926784 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:17.926766    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-vqnpj" event={"ID":"5f2298ac-7de6-4188-812f-d496c8afb895","Type":"ContainerStarted","Data":"322158983efe9699e881e54e550465107603329ea503552a4a337263b2c1f8a4"}
Sep 17 17:57:22.092028 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:22.091999    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:22.945441 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:22.945358    2585 generic.go:356] "Generic (PLEG): container finished" podID="5f2298ac-7de6-4188-812f-d496c8afb895" containerID="dd06a32228a54030894527d8b950f9e4f347484cd72aa44d6f63df83d34d9faf" exitCode=0
Sep 17 17:57:22.945441 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:22.945410    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-vqnpj" event={"ID":"5f2298ac-7de6-4188-812f-d496c8afb895","Type":"ContainerDied","Data":"dd06a32228a54030894527d8b950f9e4f347484cd72aa44d6f63df83d34d9faf"}
Sep 17 17:57:23.952170 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:23.952135    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-vqnpj" event={"ID":"5f2298ac-7de6-4188-812f-d496c8afb895","Type":"ContainerStarted","Data":"0ff77f645e80e3002b68a4ed607c8249287369105948f8d846999c7716048e36"}
Sep 17 17:57:23.952647 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:23.952206    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-vqnpj" event={"ID":"5f2298ac-7de6-4188-812f-d496c8afb895","Type":"ContainerStarted","Data":"84e21b1198a77e7500a28bcd77bc83a487460c3dcadae0e96bc397e7f0891fe1"}
Sep 17 17:57:23.952647 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:23.952422    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-vqnpj"
Sep 17 17:57:23.977893 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:23.977837    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-vqnpj" podStartSLOduration=2.118637887 podStartE2EDuration="6.977821564s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.859183677s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:57:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:57:23.976985206 +0000 UTC m=+1380.390775646" watchObservedRunningTime="2026-09-17 17:57:23.977821564 +0000 UTC m=+1380.391612001"
Sep 17 17:57:24.955363 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:24.955335    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-vqnpj"
Sep 17 17:57:27.176268 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:27.176235    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:27.372758 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:27.372729    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:28.084314 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:28.084277    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:29.336212 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:29.336179    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:57:30.964541 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:57:30.964503    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-vqnpj"
Sep 17 17:58:01.567076 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:58:01.567042    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:01.907413 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:58:01.907342    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:58:56.133426 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:58:56.133388    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-vqnpj"
Sep 17 17:59:03.434175 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:03.434141    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-vqnpj"]
Sep 17 17:59:03.434632 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:03.434420    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-vqnpj" podUID="5f2298ac-7de6-4188-812f-d496c8afb895" containerName="kserve-container" containerID="cri-o://84e21b1198a77e7500a28bcd77bc83a487460c3dcadae0e96bc397e7f0891fe1" gracePeriod=30
Sep 17 17:59:03.434632 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:03.434532    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-vqnpj" podUID="5f2298ac-7de6-4188-812f-d496c8afb895" containerName="kube-rbac-proxy" containerID="cri-o://0ff77f645e80e3002b68a4ed607c8249287369105948f8d846999c7716048e36" gracePeriod=30
Sep 17 17:59:03.462544 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:03.462508    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:04.300092 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:04.300060    2585 generic.go:356] "Generic (PLEG): container finished" podID="5f2298ac-7de6-4188-812f-d496c8afb895" containerID="0ff77f645e80e3002b68a4ed607c8249287369105948f8d846999c7716048e36" exitCode=2
Sep 17 17:59:04.300257 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:04.300132    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-vqnpj" event={"ID":"5f2298ac-7de6-4188-812f-d496c8afb895","Type":"ContainerDied","Data":"0ff77f645e80e3002b68a4ed607c8249287369105948f8d846999c7716048e36"}
Sep 17 17:59:04.769229 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:04.769207    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-vqnpj"
Sep 17 17:59:04.871908 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:04.871830    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxpwv\" (UniqueName: \"kubernetes.io/projected/5f2298ac-7de6-4188-812f-d496c8afb895-kube-api-access-sxpwv\") pod \"5f2298ac-7de6-4188-812f-d496c8afb895\" (UID: \"5f2298ac-7de6-4188-812f-d496c8afb895\") "
Sep 17 17:59:04.872045 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:04.871949    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f2298ac-7de6-4188-812f-d496c8afb895-proxy-tls\") pod \"5f2298ac-7de6-4188-812f-d496c8afb895\" (UID: \"5f2298ac-7de6-4188-812f-d496c8afb895\") "
Sep 17 17:59:04.872045 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:04.871982    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f2298ac-7de6-4188-812f-d496c8afb895-kube-rbac-proxy-sar-config\") pod \"5f2298ac-7de6-4188-812f-d496c8afb895\" (UID: \"5f2298ac-7de6-4188-812f-d496c8afb895\") "
Sep 17 17:59:04.872045 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:04.872009    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f2298ac-7de6-4188-812f-d496c8afb895-kserve-provision-location\") pod \"5f2298ac-7de6-4188-812f-d496c8afb895\" (UID: \"5f2298ac-7de6-4188-812f-d496c8afb895\") "
Sep 17 17:59:04.872045 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:04.872042    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f2298ac-7de6-4188-812f-d496c8afb895-cabundle-cert\") pod \"5f2298ac-7de6-4188-812f-d496c8afb895\" (UID: \"5f2298ac-7de6-4188-812f-d496c8afb895\") "
Sep 17 17:59:04.872438 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:04.872409    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f2298ac-7de6-4188-812f-d496c8afb895-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5f2298ac-7de6-4188-812f-d496c8afb895" (UID: "5f2298ac-7de6-4188-812f-d496c8afb895"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:59:04.872547 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:04.872519    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f2298ac-7de6-4188-812f-d496c8afb895-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5f2298ac-7de6-4188-812f-d496c8afb895" (UID: "5f2298ac-7de6-4188-812f-d496c8afb895"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 17:59:04.873917 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:04.873896    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f2298ac-7de6-4188-812f-d496c8afb895-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5f2298ac-7de6-4188-812f-d496c8afb895" (UID: "5f2298ac-7de6-4188-812f-d496c8afb895"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 17:59:04.874002 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:04.873965    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f2298ac-7de6-4188-812f-d496c8afb895-kube-api-access-sxpwv" (OuterVolumeSpecName: "kube-api-access-sxpwv") pod "5f2298ac-7de6-4188-812f-d496c8afb895" (UID: "5f2298ac-7de6-4188-812f-d496c8afb895"). InnerVolumeSpecName "kube-api-access-sxpwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 17:59:04.875996 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:04.875973    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f2298ac-7de6-4188-812f-d496c8afb895-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5f2298ac-7de6-4188-812f-d496c8afb895" (UID: "5f2298ac-7de6-4188-812f-d496c8afb895"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 17:59:04.972955 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:04.972919    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f2298ac-7de6-4188-812f-d496c8afb895-cabundle-cert\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 17:59:04.972955 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:04.972951    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxpwv\" (UniqueName: \"kubernetes.io/projected/5f2298ac-7de6-4188-812f-d496c8afb895-kube-api-access-sxpwv\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 17:59:04.972955 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:04.972962    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f2298ac-7de6-4188-812f-d496c8afb895-proxy-tls\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 17:59:04.972955 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:04.972970    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f2298ac-7de6-4188-812f-d496c8afb895-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 17:59:04.973174 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:04.972978    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f2298ac-7de6-4188-812f-d496c8afb895-kserve-provision-location\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 17:59:05.305013 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:05.304977    2585 generic.go:356] "Generic (PLEG): container finished" podID="5f2298ac-7de6-4188-812f-d496c8afb895" containerID="84e21b1198a77e7500a28bcd77bc83a487460c3dcadae0e96bc397e7f0891fe1" exitCode=0
Sep 17 17:59:05.305169 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:05.305024    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-vqnpj"
Sep 17 17:59:05.305169 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:05.305062    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-vqnpj" event={"ID":"5f2298ac-7de6-4188-812f-d496c8afb895","Type":"ContainerDied","Data":"84e21b1198a77e7500a28bcd77bc83a487460c3dcadae0e96bc397e7f0891fe1"}
Sep 17 17:59:05.305169 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:05.305117    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-vqnpj" event={"ID":"5f2298ac-7de6-4188-812f-d496c8afb895","Type":"ContainerDied","Data":"322158983efe9699e881e54e550465107603329ea503552a4a337263b2c1f8a4"}
Sep 17 17:59:05.305169 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:05.305145    2585 scope.go:117] "RemoveContainer" containerID="0ff77f645e80e3002b68a4ed607c8249287369105948f8d846999c7716048e36"
Sep 17 17:59:05.315139 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:05.315112    2585 scope.go:117] "RemoveContainer" containerID="84e21b1198a77e7500a28bcd77bc83a487460c3dcadae0e96bc397e7f0891fe1"
Sep 17 17:59:05.323056 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:05.323040    2585 scope.go:117] "RemoveContainer" containerID="dd06a32228a54030894527d8b950f9e4f347484cd72aa44d6f63df83d34d9faf"
Sep 17 17:59:05.330057 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:05.330029    2585 scope.go:117] "RemoveContainer" containerID="0ff77f645e80e3002b68a4ed607c8249287369105948f8d846999c7716048e36"
Sep 17 17:59:05.330431 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:59:05.330381    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ff77f645e80e3002b68a4ed607c8249287369105948f8d846999c7716048e36\": container with ID starting with 0ff77f645e80e3002b68a4ed607c8249287369105948f8d846999c7716048e36 not found: ID does not exist" containerID="0ff77f645e80e3002b68a4ed607c8249287369105948f8d846999c7716048e36"
Sep 17 17:59:05.330536 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:05.330414    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ff77f645e80e3002b68a4ed607c8249287369105948f8d846999c7716048e36"} err="failed to get container status \"0ff77f645e80e3002b68a4ed607c8249287369105948f8d846999c7716048e36\": rpc error: code = NotFound desc = could not find container \"0ff77f645e80e3002b68a4ed607c8249287369105948f8d846999c7716048e36\": container with ID starting with 0ff77f645e80e3002b68a4ed607c8249287369105948f8d846999c7716048e36 not found: ID does not exist"
Sep 17 17:59:05.330646 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:05.330545    2585 scope.go:117] "RemoveContainer" containerID="84e21b1198a77e7500a28bcd77bc83a487460c3dcadae0e96bc397e7f0891fe1"
Sep 17 17:59:05.330840 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:59:05.330824    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84e21b1198a77e7500a28bcd77bc83a487460c3dcadae0e96bc397e7f0891fe1\": container with ID starting with 84e21b1198a77e7500a28bcd77bc83a487460c3dcadae0e96bc397e7f0891fe1 not found: ID does not exist" containerID="84e21b1198a77e7500a28bcd77bc83a487460c3dcadae0e96bc397e7f0891fe1"
Sep 17 17:59:05.330906 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:05.330846    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84e21b1198a77e7500a28bcd77bc83a487460c3dcadae0e96bc397e7f0891fe1"} err="failed to get container status \"84e21b1198a77e7500a28bcd77bc83a487460c3dcadae0e96bc397e7f0891fe1\": rpc error: code = NotFound desc = could not find container \"84e21b1198a77e7500a28bcd77bc83a487460c3dcadae0e96bc397e7f0891fe1\": container with ID starting with 84e21b1198a77e7500a28bcd77bc83a487460c3dcadae0e96bc397e7f0891fe1 not found: ID does not exist"
Sep 17 17:59:05.330906 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:05.330866    2585 scope.go:117] "RemoveContainer" containerID="dd06a32228a54030894527d8b950f9e4f347484cd72aa44d6f63df83d34d9faf"
Sep 17 17:59:05.331120 ip-10-0-133-139 kubenswrapper[2585]: E0917 17:59:05.331104    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd06a32228a54030894527d8b950f9e4f347484cd72aa44d6f63df83d34d9faf\": container with ID starting with dd06a32228a54030894527d8b950f9e4f347484cd72aa44d6f63df83d34d9faf not found: ID does not exist" containerID="dd06a32228a54030894527d8b950f9e4f347484cd72aa44d6f63df83d34d9faf"
Sep 17 17:59:05.331157 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:05.331125    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd06a32228a54030894527d8b950f9e4f347484cd72aa44d6f63df83d34d9faf"} err="failed to get container status \"dd06a32228a54030894527d8b950f9e4f347484cd72aa44d6f63df83d34d9faf\": rpc error: code = NotFound desc = could not find container \"dd06a32228a54030894527d8b950f9e4f347484cd72aa44d6f63df83d34d9faf\": container with ID starting with dd06a32228a54030894527d8b950f9e4f347484cd72aa44d6f63df83d34d9faf not found: ID does not exist"
Sep 17 17:59:05.331514 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:05.331480    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-vqnpj"]
Sep 17 17:59:05.338351 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:05.338331    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-vqnpj"]
Sep 17 17:59:06.104651 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:06.104621    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f2298ac-7de6-4188-812f-d496c8afb895" path="/var/lib/kubelet/pods/5f2298ac-7de6-4188-812f-d496c8afb895/volumes"
Sep 17 17:59:08.622246 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:08.622212    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:08.663563 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:08.663532    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vg4g9"]
Sep 17 17:59:08.663787 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:08.663775    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f2298ac-7de6-4188-812f-d496c8afb895" containerName="storage-initializer"
Sep 17 17:59:08.663825 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:08.663788    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f2298ac-7de6-4188-812f-d496c8afb895" containerName="storage-initializer"
Sep 17 17:59:08.663825 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:08.663796    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f2298ac-7de6-4188-812f-d496c8afb895" containerName="kserve-container"
Sep 17 17:59:08.663825 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:08.663803    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f2298ac-7de6-4188-812f-d496c8afb895" containerName="kserve-container"
Sep 17 17:59:08.663825 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:08.663810    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f2298ac-7de6-4188-812f-d496c8afb895" containerName="kube-rbac-proxy"
Sep 17 17:59:08.663825 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:08.663815    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f2298ac-7de6-4188-812f-d496c8afb895" containerName="kube-rbac-proxy"
Sep 17 17:59:08.663969 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:08.663858    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f2298ac-7de6-4188-812f-d496c8afb895" containerName="kube-rbac-proxy"
Sep 17 17:59:08.663969 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:08.663865    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f2298ac-7de6-4188-812f-d496c8afb895" containerName="kserve-container"
Sep 17 17:59:08.668625 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:08.668605    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vg4g9"
Sep 17 17:59:08.673774 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:08.673754    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 17:59:08.673892 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:08.673758    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Sep 17 17:59:08.674683 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:08.674610    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 17:59:08.674683 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:08.674636    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Sep 17 17:59:08.674810 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:08.674603    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 17:59:08.674907 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:08.674893    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 17:59:08.675522 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:08.675483    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lps5w\""
Sep 17 17:59:08.690507 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:08.690468    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vg4g9"]
Sep 17 17:59:08.705688 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:08.705668    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30b29da0-ef86-4963-8342-e366ddf0a445-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-vg4g9\" (UID: \"30b29da0-ef86-4963-8342-e366ddf0a445\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vg4g9"
Sep 17 17:59:08.705793 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:08.705706    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30b29da0-ef86-4963-8342-e366ddf0a445-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-vg4g9\" (UID: \"30b29da0-ef86-4963-8342-e366ddf0a445\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vg4g9"
Sep 17 17:59:08.705793 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:08.705772    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30b29da0-ef86-4963-8342-e366ddf0a445-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-vg4g9\" (UID: \"30b29da0-ef86-4963-8342-e366ddf0a445\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vg4g9"
Sep 17 17:59:08.705883 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:08.705814    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crwb2\" (UniqueName: \"kubernetes.io/projected/30b29da0-ef86-4963-8342-e366ddf0a445-kube-api-access-crwb2\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-vg4g9\" (UID: \"30b29da0-ef86-4963-8342-e366ddf0a445\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vg4g9"
Sep 17 17:59:08.705883 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:08.705857    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30b29da0-ef86-4963-8342-e366ddf0a445-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-vg4g9\" (UID: \"30b29da0-ef86-4963-8342-e366ddf0a445\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vg4g9"
Sep 17 17:59:08.747250 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:08.747225    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:08.807010 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:08.806989    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30b29da0-ef86-4963-8342-e366ddf0a445-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-vg4g9\" (UID: \"30b29da0-ef86-4963-8342-e366ddf0a445\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vg4g9"
Sep 17 17:59:08.807119 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:08.807015    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30b29da0-ef86-4963-8342-e366ddf0a445-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-vg4g9\" (UID: \"30b29da0-ef86-4963-8342-e366ddf0a445\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vg4g9"
Sep 17 17:59:08.807119 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:08.807039    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30b29da0-ef86-4963-8342-e366ddf0a445-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-vg4g9\" (UID: \"30b29da0-ef86-4963-8342-e366ddf0a445\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vg4g9"
Sep 17 17:59:08.807119 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:08.807060    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crwb2\" (UniqueName: \"kubernetes.io/projected/30b29da0-ef86-4963-8342-e366ddf0a445-kube-api-access-crwb2\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-vg4g9\" (UID: \"30b29da0-ef86-4963-8342-e366ddf0a445\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vg4g9"
Sep 17 17:59:08.807262 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:08.807182    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30b29da0-ef86-4963-8342-e366ddf0a445-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-vg4g9\" (UID: \"30b29da0-ef86-4963-8342-e366ddf0a445\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vg4g9"
Sep 17 17:59:08.807591 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:08.807569    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30b29da0-ef86-4963-8342-e366ddf0a445-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-vg4g9\" (UID: \"30b29da0-ef86-4963-8342-e366ddf0a445\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vg4g9"
Sep 17 17:59:08.807739 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:08.807721    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30b29da0-ef86-4963-8342-e366ddf0a445-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-vg4g9\" (UID: \"30b29da0-ef86-4963-8342-e366ddf0a445\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vg4g9"
Sep 17 17:59:08.807796 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:08.807724    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30b29da0-ef86-4963-8342-e366ddf0a445-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-vg4g9\" (UID: \"30b29da0-ef86-4963-8342-e366ddf0a445\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vg4g9"
Sep 17 17:59:08.809345 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:08.809329    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30b29da0-ef86-4963-8342-e366ddf0a445-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-vg4g9\" (UID: \"30b29da0-ef86-4963-8342-e366ddf0a445\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vg4g9"
Sep 17 17:59:08.816469 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:08.816450    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crwb2\" (UniqueName: \"kubernetes.io/projected/30b29da0-ef86-4963-8342-e366ddf0a445-kube-api-access-crwb2\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-vg4g9\" (UID: \"30b29da0-ef86-4963-8342-e366ddf0a445\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vg4g9"
Sep 17 17:59:08.977980 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:08.977920    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vg4g9"
Sep 17 17:59:09.252035 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:09.252013    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vg4g9"]
Sep 17 17:59:09.254552 ip-10-0-133-139 kubenswrapper[2585]: W0917 17:59:09.254525    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30b29da0_ef86_4963_8342_e366ddf0a445.slice/crio-03b68ec6ab3c7e42f6ef5dbef4ecacd564ee1d1879f407742d81858df5a5ead1 WatchSource:0}: Error finding container 03b68ec6ab3c7e42f6ef5dbef4ecacd564ee1d1879f407742d81858df5a5ead1: Status 404 returned error can't find the container with id 03b68ec6ab3c7e42f6ef5dbef4ecacd564ee1d1879f407742d81858df5a5ead1
Sep 17 17:59:09.321951 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:09.321924    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vg4g9" event={"ID":"30b29da0-ef86-4963-8342-e366ddf0a445","Type":"ContainerStarted","Data":"06ab657507ea215bdaba7883758b8f16587036f464f1a2765b0d1ea3e90b6d91"}
Sep 17 17:59:09.322081 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:09.321988    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vg4g9" event={"ID":"30b29da0-ef86-4963-8342-e366ddf0a445","Type":"ContainerStarted","Data":"03b68ec6ab3c7e42f6ef5dbef4ecacd564ee1d1879f407742d81858df5a5ead1"}
Sep 17 17:59:28.398060 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:28.398029    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rrf79_beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5/ovn-acl-logging/0.log"
Sep 17 17:59:28.398601 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:28.398478    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rrf79_beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5/ovn-acl-logging/0.log"
Sep 17 17:59:31.764275 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:31.764222    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:36.842404 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:36.842369    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:37.149410 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:37.149327    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 17:59:38.420612 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:38.420578    2585 generic.go:356] "Generic (PLEG): container finished" podID="30b29da0-ef86-4963-8342-e366ddf0a445" containerID="06ab657507ea215bdaba7883758b8f16587036f464f1a2765b0d1ea3e90b6d91" exitCode=0
Sep 17 17:59:38.420964 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:38.420641    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vg4g9" event={"ID":"30b29da0-ef86-4963-8342-e366ddf0a445","Type":"ContainerDied","Data":"06ab657507ea215bdaba7883758b8f16587036f464f1a2765b0d1ea3e90b6d91"}
Sep 17 17:59:39.427105 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:39.427071    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vg4g9" event={"ID":"30b29da0-ef86-4963-8342-e366ddf0a445","Type":"ContainerStarted","Data":"610faa9260d58aa66c300358b168a6ad30223c7c10f9adaf87f01baeed8229ee"}
Sep 17 17:59:39.427462 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:39.427133    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vg4g9" event={"ID":"30b29da0-ef86-4963-8342-e366ddf0a445","Type":"ContainerStarted","Data":"976fa6bed947bf385a8ee260c18ffd7ead3bacb17f7e64c57a06447d1d0bceb8"}
Sep 17 17:59:39.427462 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:39.427321    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vg4g9"
Sep 17 17:59:39.453400 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:39.453355    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vg4g9" podStartSLOduration=3.008531083 podStartE2EDuration="31.453341223s" totalImagesPullingTime="0s" totalInitContainerRuntime="28.44481014s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:59:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:59:39.452243998 +0000 UTC m=+1515.866034448" watchObservedRunningTime="2026-09-17 17:59:39.453341223 +0000 UTC m=+1515.867131660"
Sep 17 17:59:40.430824 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:40.430794    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vg4g9"
Sep 17 17:59:46.440141 ip-10-0-133-139 kubenswrapper[2585]: I0917 17:59:46.440114    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vg4g9"
Sep 17 18:00:08.669440 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:00:08.669401    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:13.133372 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:13.133341    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vg4g9"
Sep 17 18:01:24.141670 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:24.141623    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vg4g9" podUID="30b29da0-ef86-4963-8342-e366ddf0a445" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.25:8080/v2/models/isvc-autogluon-v2-noslash/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 17 18:01:24.679727 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:24.679698    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vg4g9"]
Sep 17 18:01:24.679950 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:24.679909    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vg4g9" podUID="30b29da0-ef86-4963-8342-e366ddf0a445" containerName="kserve-container" containerID="cri-o://976fa6bed947bf385a8ee260c18ffd7ead3bacb17f7e64c57a06447d1d0bceb8" gracePeriod=30
Sep 17 18:01:24.680017 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:24.679946    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vg4g9" podUID="30b29da0-ef86-4963-8342-e366ddf0a445" containerName="kube-rbac-proxy" containerID="cri-o://610faa9260d58aa66c300358b168a6ad30223c7c10f9adaf87f01baeed8229ee" gracePeriod=30
Sep 17 18:01:25.796866 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:25.796829    2585 generic.go:356] "Generic (PLEG): container finished" podID="30b29da0-ef86-4963-8342-e366ddf0a445" containerID="610faa9260d58aa66c300358b168a6ad30223c7c10f9adaf87f01baeed8229ee" exitCode=2
Sep 17 18:01:25.797228 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:25.796906    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vg4g9" event={"ID":"30b29da0-ef86-4963-8342-e366ddf0a445","Type":"ContainerDied","Data":"610faa9260d58aa66c300358b168a6ad30223c7c10f9adaf87f01baeed8229ee"}
Sep 17 18:01:26.011722 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:26.011697    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vg4g9"
Sep 17 18:01:26.072619 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:26.072546    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:26.166958 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:26.166925    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30b29da0-ef86-4963-8342-e366ddf0a445-cabundle-cert\") pod \"30b29da0-ef86-4963-8342-e366ddf0a445\" (UID: \"30b29da0-ef86-4963-8342-e366ddf0a445\") "
Sep 17 18:01:26.167145 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:26.166966    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crwb2\" (UniqueName: \"kubernetes.io/projected/30b29da0-ef86-4963-8342-e366ddf0a445-kube-api-access-crwb2\") pod \"30b29da0-ef86-4963-8342-e366ddf0a445\" (UID: \"30b29da0-ef86-4963-8342-e366ddf0a445\") "
Sep 17 18:01:26.167145 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:26.167011    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30b29da0-ef86-4963-8342-e366ddf0a445-proxy-tls\") pod \"30b29da0-ef86-4963-8342-e366ddf0a445\" (UID: \"30b29da0-ef86-4963-8342-e366ddf0a445\") "
Sep 17 18:01:26.167145 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:26.167035    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30b29da0-ef86-4963-8342-e366ddf0a445-kserve-provision-location\") pod \"30b29da0-ef86-4963-8342-e366ddf0a445\" (UID: \"30b29da0-ef86-4963-8342-e366ddf0a445\") "
Sep 17 18:01:26.167145 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:26.167080    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30b29da0-ef86-4963-8342-e366ddf0a445-kube-rbac-proxy-sar-config\") pod \"30b29da0-ef86-4963-8342-e366ddf0a445\" (UID: \"30b29da0-ef86-4963-8342-e366ddf0a445\") "
Sep 17 18:01:26.167435 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:26.167405    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30b29da0-ef86-4963-8342-e366ddf0a445-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "30b29da0-ef86-4963-8342-e366ddf0a445" (UID: "30b29da0-ef86-4963-8342-e366ddf0a445"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:01:26.167578 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:26.167524    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30b29da0-ef86-4963-8342-e366ddf0a445-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "30b29da0-ef86-4963-8342-e366ddf0a445" (UID: "30b29da0-ef86-4963-8342-e366ddf0a445"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:01:26.169105 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:26.169077    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30b29da0-ef86-4963-8342-e366ddf0a445-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "30b29da0-ef86-4963-8342-e366ddf0a445" (UID: "30b29da0-ef86-4963-8342-e366ddf0a445"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:01:26.169405 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:26.169384    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30b29da0-ef86-4963-8342-e366ddf0a445-kube-api-access-crwb2" (OuterVolumeSpecName: "kube-api-access-crwb2") pod "30b29da0-ef86-4963-8342-e366ddf0a445" (UID: "30b29da0-ef86-4963-8342-e366ddf0a445"). InnerVolumeSpecName "kube-api-access-crwb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:01:26.172543 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:26.172521    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30b29da0-ef86-4963-8342-e366ddf0a445-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "30b29da0-ef86-4963-8342-e366ddf0a445" (UID: "30b29da0-ef86-4963-8342-e366ddf0a445"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:01:26.267996 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:26.267960    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30b29da0-ef86-4963-8342-e366ddf0a445-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:01:26.267996 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:26.267991    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/30b29da0-ef86-4963-8342-e366ddf0a445-cabundle-cert\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:01:26.267996 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:26.268005    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crwb2\" (UniqueName: \"kubernetes.io/projected/30b29da0-ef86-4963-8342-e366ddf0a445-kube-api-access-crwb2\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:01:26.268235 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:26.268017    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30b29da0-ef86-4963-8342-e366ddf0a445-proxy-tls\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:01:26.268235 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:26.268028    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30b29da0-ef86-4963-8342-e366ddf0a445-kserve-provision-location\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:01:26.802725 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:26.802683    2585 generic.go:356] "Generic (PLEG): container finished" podID="30b29da0-ef86-4963-8342-e366ddf0a445" containerID="976fa6bed947bf385a8ee260c18ffd7ead3bacb17f7e64c57a06447d1d0bceb8" exitCode=0
Sep 17 18:01:26.803173 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:26.802734    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vg4g9"
Sep 17 18:01:26.803173 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:26.802741    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vg4g9" event={"ID":"30b29da0-ef86-4963-8342-e366ddf0a445","Type":"ContainerDied","Data":"976fa6bed947bf385a8ee260c18ffd7ead3bacb17f7e64c57a06447d1d0bceb8"}
Sep 17 18:01:26.803173 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:26.802802    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vg4g9" event={"ID":"30b29da0-ef86-4963-8342-e366ddf0a445","Type":"ContainerDied","Data":"03b68ec6ab3c7e42f6ef5dbef4ecacd564ee1d1879f407742d81858df5a5ead1"}
Sep 17 18:01:26.803173 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:26.802842    2585 scope.go:117] "RemoveContainer" containerID="610faa9260d58aa66c300358b168a6ad30223c7c10f9adaf87f01baeed8229ee"
Sep 17 18:01:26.811523 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:26.811501    2585 scope.go:117] "RemoveContainer" containerID="976fa6bed947bf385a8ee260c18ffd7ead3bacb17f7e64c57a06447d1d0bceb8"
Sep 17 18:01:26.818330 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:26.818310    2585 scope.go:117] "RemoveContainer" containerID="06ab657507ea215bdaba7883758b8f16587036f464f1a2765b0d1ea3e90b6d91"
Sep 17 18:01:26.825287 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:26.825211    2585 scope.go:117] "RemoveContainer" containerID="610faa9260d58aa66c300358b168a6ad30223c7c10f9adaf87f01baeed8229ee"
Sep 17 18:01:26.825651 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:01:26.825630    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"610faa9260d58aa66c300358b168a6ad30223c7c10f9adaf87f01baeed8229ee\": container with ID starting with 610faa9260d58aa66c300358b168a6ad30223c7c10f9adaf87f01baeed8229ee not found: ID does not exist" containerID="610faa9260d58aa66c300358b168a6ad30223c7c10f9adaf87f01baeed8229ee"
Sep 17 18:01:26.825747 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:26.825657    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"610faa9260d58aa66c300358b168a6ad30223c7c10f9adaf87f01baeed8229ee"} err="failed to get container status \"610faa9260d58aa66c300358b168a6ad30223c7c10f9adaf87f01baeed8229ee\": rpc error: code = NotFound desc = could not find container \"610faa9260d58aa66c300358b168a6ad30223c7c10f9adaf87f01baeed8229ee\": container with ID starting with 610faa9260d58aa66c300358b168a6ad30223c7c10f9adaf87f01baeed8229ee not found: ID does not exist"
Sep 17 18:01:26.825747 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:26.825676    2585 scope.go:117] "RemoveContainer" containerID="976fa6bed947bf385a8ee260c18ffd7ead3bacb17f7e64c57a06447d1d0bceb8"
Sep 17 18:01:26.825947 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:01:26.825930    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"976fa6bed947bf385a8ee260c18ffd7ead3bacb17f7e64c57a06447d1d0bceb8\": container with ID starting with 976fa6bed947bf385a8ee260c18ffd7ead3bacb17f7e64c57a06447d1d0bceb8 not found: ID does not exist" containerID="976fa6bed947bf385a8ee260c18ffd7ead3bacb17f7e64c57a06447d1d0bceb8"
Sep 17 18:01:26.825990 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:26.825953    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"976fa6bed947bf385a8ee260c18ffd7ead3bacb17f7e64c57a06447d1d0bceb8"} err="failed to get container status \"976fa6bed947bf385a8ee260c18ffd7ead3bacb17f7e64c57a06447d1d0bceb8\": rpc error: code = NotFound desc = could not find container \"976fa6bed947bf385a8ee260c18ffd7ead3bacb17f7e64c57a06447d1d0bceb8\": container with ID starting with 976fa6bed947bf385a8ee260c18ffd7ead3bacb17f7e64c57a06447d1d0bceb8 not found: ID does not exist"
Sep 17 18:01:26.825990 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:26.825969    2585 scope.go:117] "RemoveContainer" containerID="06ab657507ea215bdaba7883758b8f16587036f464f1a2765b0d1ea3e90b6d91"
Sep 17 18:01:26.826248 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:01:26.826225    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06ab657507ea215bdaba7883758b8f16587036f464f1a2765b0d1ea3e90b6d91\": container with ID starting with 06ab657507ea215bdaba7883758b8f16587036f464f1a2765b0d1ea3e90b6d91 not found: ID does not exist" containerID="06ab657507ea215bdaba7883758b8f16587036f464f1a2765b0d1ea3e90b6d91"
Sep 17 18:01:26.826288 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:26.826258    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06ab657507ea215bdaba7883758b8f16587036f464f1a2765b0d1ea3e90b6d91"} err="failed to get container status \"06ab657507ea215bdaba7883758b8f16587036f464f1a2765b0d1ea3e90b6d91\": rpc error: code = NotFound desc = could not find container \"06ab657507ea215bdaba7883758b8f16587036f464f1a2765b0d1ea3e90b6d91\": container with ID starting with 06ab657507ea215bdaba7883758b8f16587036f464f1a2765b0d1ea3e90b6d91 not found: ID does not exist"
Sep 17 18:01:26.827174 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:26.827157    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vg4g9"]
Sep 17 18:01:26.835258 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:26.835240    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-vg4g9"]
Sep 17 18:01:28.103742 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:28.103713    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30b29da0-ef86-4963-8342-e366ddf0a445" path="/var/lib/kubelet/pods/30b29da0-ef86-4963-8342-e366ddf0a445/volumes"
Sep 17 18:01:29.804973 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:29.804939    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:29.879908 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:29.879878    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk"]
Sep 17 18:01:29.880132 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:29.880121    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30b29da0-ef86-4963-8342-e366ddf0a445" containerName="kserve-container"
Sep 17 18:01:29.880187 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:29.880134    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="30b29da0-ef86-4963-8342-e366ddf0a445" containerName="kserve-container"
Sep 17 18:01:29.880187 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:29.880155    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30b29da0-ef86-4963-8342-e366ddf0a445" containerName="storage-initializer"
Sep 17 18:01:29.880187 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:29.880160    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="30b29da0-ef86-4963-8342-e366ddf0a445" containerName="storage-initializer"
Sep 17 18:01:29.880187 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:29.880167    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30b29da0-ef86-4963-8342-e366ddf0a445" containerName="kube-rbac-proxy"
Sep 17 18:01:29.880187 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:29.880172    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="30b29da0-ef86-4963-8342-e366ddf0a445" containerName="kube-rbac-proxy"
Sep 17 18:01:29.880340 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:29.880216    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="30b29da0-ef86-4963-8342-e366ddf0a445" containerName="kube-rbac-proxy"
Sep 17 18:01:29.880340 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:29.880224    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="30b29da0-ef86-4963-8342-e366ddf0a445" containerName="kserve-container"
Sep 17 18:01:29.883315 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:29.883299    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk"
Sep 17 18:01:29.888177 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:29.888154    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:01:29.888296 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:29.888248    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Sep 17 18:01:29.888296 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:29.888266    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:01:29.888296 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:29.888268    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:01:29.888437 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:29.888300    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:01:29.888437 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:29.888356    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Sep 17 18:01:29.888872 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:29.888834    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lps5w\""
Sep 17 18:01:29.896023 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:29.896003    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk"]
Sep 17 18:01:29.992730 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:29.992703    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86e0f7c8-3f94-4ab8-9fd6-463dca55b391-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk\" (UID: \"86e0f7c8-3f94-4ab8-9fd6-463dca55b391\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk"
Sep 17 18:01:29.992860 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:29.992745    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86e0f7c8-3f94-4ab8-9fd6-463dca55b391-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk\" (UID: \"86e0f7c8-3f94-4ab8-9fd6-463dca55b391\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk"
Sep 17 18:01:29.992860 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:29.992773    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86e0f7c8-3f94-4ab8-9fd6-463dca55b391-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk\" (UID: \"86e0f7c8-3f94-4ab8-9fd6-463dca55b391\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk"
Sep 17 18:01:29.992860 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:29.992804    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l7jk\" (UniqueName: \"kubernetes.io/projected/86e0f7c8-3f94-4ab8-9fd6-463dca55b391-kube-api-access-2l7jk\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk\" (UID: \"86e0f7c8-3f94-4ab8-9fd6-463dca55b391\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk"
Sep 17 18:01:29.992958 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:29.992867    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86e0f7c8-3f94-4ab8-9fd6-463dca55b391-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk\" (UID: \"86e0f7c8-3f94-4ab8-9fd6-463dca55b391\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk"
Sep 17 18:01:30.093561 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:30.093502    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86e0f7c8-3f94-4ab8-9fd6-463dca55b391-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk\" (UID: \"86e0f7c8-3f94-4ab8-9fd6-463dca55b391\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk"
Sep 17 18:01:30.093561 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:30.093532    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86e0f7c8-3f94-4ab8-9fd6-463dca55b391-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk\" (UID: \"86e0f7c8-3f94-4ab8-9fd6-463dca55b391\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk"
Sep 17 18:01:30.093561 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:30.093554    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l7jk\" (UniqueName: \"kubernetes.io/projected/86e0f7c8-3f94-4ab8-9fd6-463dca55b391-kube-api-access-2l7jk\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk\" (UID: \"86e0f7c8-3f94-4ab8-9fd6-463dca55b391\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk"
Sep 17 18:01:30.093770 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:30.093751    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86e0f7c8-3f94-4ab8-9fd6-463dca55b391-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk\" (UID: \"86e0f7c8-3f94-4ab8-9fd6-463dca55b391\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk"
Sep 17 18:01:30.093837 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:30.093821    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86e0f7c8-3f94-4ab8-9fd6-463dca55b391-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk\" (UID: \"86e0f7c8-3f94-4ab8-9fd6-463dca55b391\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk"
Sep 17 18:01:30.093944 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:30.093922    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86e0f7c8-3f94-4ab8-9fd6-463dca55b391-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk\" (UID: \"86e0f7c8-3f94-4ab8-9fd6-463dca55b391\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk"
Sep 17 18:01:30.093999 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:01:30.093955    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-serving-cert: secret "isvc-autogluon-ts-v1-predictor-serving-cert" not found
Sep 17 18:01:30.094050 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:01:30.094029    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86e0f7c8-3f94-4ab8-9fd6-463dca55b391-proxy-tls podName:86e0f7c8-3f94-4ab8-9fd6-463dca55b391 nodeName:}" failed. No retries permitted until 2026-09-17 18:01:30.594010439 +0000 UTC m=+1627.007800868 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/86e0f7c8-3f94-4ab8-9fd6-463dca55b391-proxy-tls") pod "isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk" (UID: "86e0f7c8-3f94-4ab8-9fd6-463dca55b391") : secret "isvc-autogluon-ts-v1-predictor-serving-cert" not found
Sep 17 18:01:30.094260 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:30.094243    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86e0f7c8-3f94-4ab8-9fd6-463dca55b391-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk\" (UID: \"86e0f7c8-3f94-4ab8-9fd6-463dca55b391\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk"
Sep 17 18:01:30.094313 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:30.094291    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86e0f7c8-3f94-4ab8-9fd6-463dca55b391-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk\" (UID: \"86e0f7c8-3f94-4ab8-9fd6-463dca55b391\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk"
Sep 17 18:01:30.105634 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:30.105616    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l7jk\" (UniqueName: \"kubernetes.io/projected/86e0f7c8-3f94-4ab8-9fd6-463dca55b391-kube-api-access-2l7jk\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk\" (UID: \"86e0f7c8-3f94-4ab8-9fd6-463dca55b391\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk"
Sep 17 18:01:30.192631 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:30.192599    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:01:30.597104 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:30.597072    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86e0f7c8-3f94-4ab8-9fd6-463dca55b391-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk\" (UID: \"86e0f7c8-3f94-4ab8-9fd6-463dca55b391\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk"
Sep 17 18:01:30.599406 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:30.599378    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86e0f7c8-3f94-4ab8-9fd6-463dca55b391-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk\" (UID: \"86e0f7c8-3f94-4ab8-9fd6-463dca55b391\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk"
Sep 17 18:01:30.792347 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:30.792312    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk"
Sep 17 18:01:31.067666 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:31.067642    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk"]
Sep 17 18:01:31.070089 ip-10-0-133-139 kubenswrapper[2585]: W0917 18:01:31.070060    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86e0f7c8_3f94_4ab8_9fd6_463dca55b391.slice/crio-6b6e0e0656cfe330189fb69e980f048c7c0bf61479435e826ba8ef123a2bdd75 WatchSource:0}: Error finding container 6b6e0e0656cfe330189fb69e980f048c7c0bf61479435e826ba8ef123a2bdd75: Status 404 returned error can't find the container with id 6b6e0e0656cfe330189fb69e980f048c7c0bf61479435e826ba8ef123a2bdd75
Sep 17 18:01:31.823010 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:31.822977    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk" event={"ID":"86e0f7c8-3f94-4ab8-9fd6-463dca55b391","Type":"ContainerStarted","Data":"8d13a8b2228c0950b4b4a91ace00986789c5c5aa30ff89e07c252fe7f21b0743"}
Sep 17 18:01:31.823201 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:31.823032    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk" event={"ID":"86e0f7c8-3f94-4ab8-9fd6-463dca55b391","Type":"ContainerStarted","Data":"6b6e0e0656cfe330189fb69e980f048c7c0bf61479435e826ba8ef123a2bdd75"}
Sep 17 18:01:40.857104 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:40.857072    2585 generic.go:356] "Generic (PLEG): container finished" podID="86e0f7c8-3f94-4ab8-9fd6-463dca55b391" containerID="8d13a8b2228c0950b4b4a91ace00986789c5c5aa30ff89e07c252fe7f21b0743" exitCode=0
Sep 17 18:01:40.857481 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:40.857113    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk" event={"ID":"86e0f7c8-3f94-4ab8-9fd6-463dca55b391","Type":"ContainerDied","Data":"8d13a8b2228c0950b4b4a91ace00986789c5c5aa30ff89e07c252fe7f21b0743"}
Sep 17 18:01:41.862932 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:41.862900    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk" event={"ID":"86e0f7c8-3f94-4ab8-9fd6-463dca55b391","Type":"ContainerStarted","Data":"946cad50dcd4e23889293a65cb3dbd96efbdd0ca8304ebe779d5ac54682d8999"}
Sep 17 18:01:41.863332 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:41.862966    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk" event={"ID":"86e0f7c8-3f94-4ab8-9fd6-463dca55b391","Type":"ContainerStarted","Data":"66ec9e531920de6a980fd21294dc734c6fcea63c484557d1eb70d5337dc3506a"}
Sep 17 18:01:41.863332 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:41.863146    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk"
Sep 17 18:01:41.883953 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:41.883909    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk" podStartSLOduration=4.145096894 podStartE2EDuration="12.883896127s" totalImagesPullingTime="0s" totalInitContainerRuntime="8.738799233s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:01:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:01:41.882545017 +0000 UTC m=+1638.296335445" watchObservedRunningTime="2026-09-17 18:01:41.883896127 +0000 UTC m=+1638.297686585"
Sep 17 18:01:42.867596 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:42.867556    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk"
Sep 17 18:01:42.868950 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:42.868925    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk" podUID="86e0f7c8-3f94-4ab8-9fd6-463dca55b391" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 18:01:43.871978 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:43.871939    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk" podUID="86e0f7c8-3f94-4ab8-9fd6-463dca55b391" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 18:01:48.876362 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:48.876331    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk"
Sep 17 18:01:48.876910 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:48.876884    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk" podUID="86e0f7c8-3f94-4ab8-9fd6-463dca55b391" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Sep 17 18:01:58.877661 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:01:58.877631    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk"
Sep 17 18:02:12.285752 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:12.285715    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk"]
Sep 17 18:02:12.286281 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:12.286013    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk" podUID="86e0f7c8-3f94-4ab8-9fd6-463dca55b391" containerName="kserve-container" containerID="cri-o://66ec9e531920de6a980fd21294dc734c6fcea63c484557d1eb70d5337dc3506a" gracePeriod=30
Sep 17 18:02:12.286281 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:12.286059    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk" podUID="86e0f7c8-3f94-4ab8-9fd6-463dca55b391" containerName="kube-rbac-proxy" containerID="cri-o://946cad50dcd4e23889293a65cb3dbd96efbdd0ca8304ebe779d5ac54682d8999" gracePeriod=30
Sep 17 18:02:12.303250 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:12.303217    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:02:12.983292 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:12.983254    2585 generic.go:356] "Generic (PLEG): container finished" podID="86e0f7c8-3f94-4ab8-9fd6-463dca55b391" containerID="946cad50dcd4e23889293a65cb3dbd96efbdd0ca8304ebe779d5ac54682d8999" exitCode=2
Sep 17 18:02:12.983292 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:12.983299    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk" event={"ID":"86e0f7c8-3f94-4ab8-9fd6-463dca55b391","Type":"ContainerDied","Data":"946cad50dcd4e23889293a65cb3dbd96efbdd0ca8304ebe779d5ac54682d8999"}
Sep 17 18:02:13.956040 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:13.956015    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk"
Sep 17 18:02:13.989301 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:13.989268    2585 generic.go:356] "Generic (PLEG): container finished" podID="86e0f7c8-3f94-4ab8-9fd6-463dca55b391" containerID="66ec9e531920de6a980fd21294dc734c6fcea63c484557d1eb70d5337dc3506a" exitCode=0
Sep 17 18:02:13.989438 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:13.989314    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk"
Sep 17 18:02:13.989438 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:13.989320    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk" event={"ID":"86e0f7c8-3f94-4ab8-9fd6-463dca55b391","Type":"ContainerDied","Data":"66ec9e531920de6a980fd21294dc734c6fcea63c484557d1eb70d5337dc3506a"}
Sep 17 18:02:13.989438 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:13.989386    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk" event={"ID":"86e0f7c8-3f94-4ab8-9fd6-463dca55b391","Type":"ContainerDied","Data":"6b6e0e0656cfe330189fb69e980f048c7c0bf61479435e826ba8ef123a2bdd75"}
Sep 17 18:02:13.989438 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:13.989427    2585 scope.go:117] "RemoveContainer" containerID="946cad50dcd4e23889293a65cb3dbd96efbdd0ca8304ebe779d5ac54682d8999"
Sep 17 18:02:13.997329 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:13.997307    2585 scope.go:117] "RemoveContainer" containerID="66ec9e531920de6a980fd21294dc734c6fcea63c484557d1eb70d5337dc3506a"
Sep 17 18:02:14.004439 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:14.004421    2585 scope.go:117] "RemoveContainer" containerID="8d13a8b2228c0950b4b4a91ace00986789c5c5aa30ff89e07c252fe7f21b0743"
Sep 17 18:02:14.011428 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:14.011412    2585 scope.go:117] "RemoveContainer" containerID="946cad50dcd4e23889293a65cb3dbd96efbdd0ca8304ebe779d5ac54682d8999"
Sep 17 18:02:14.011704 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:02:14.011685    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"946cad50dcd4e23889293a65cb3dbd96efbdd0ca8304ebe779d5ac54682d8999\": container with ID starting with 946cad50dcd4e23889293a65cb3dbd96efbdd0ca8304ebe779d5ac54682d8999 not found: ID does not exist" containerID="946cad50dcd4e23889293a65cb3dbd96efbdd0ca8304ebe779d5ac54682d8999"
Sep 17 18:02:14.011759 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:14.011714    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"946cad50dcd4e23889293a65cb3dbd96efbdd0ca8304ebe779d5ac54682d8999"} err="failed to get container status \"946cad50dcd4e23889293a65cb3dbd96efbdd0ca8304ebe779d5ac54682d8999\": rpc error: code = NotFound desc = could not find container \"946cad50dcd4e23889293a65cb3dbd96efbdd0ca8304ebe779d5ac54682d8999\": container with ID starting with 946cad50dcd4e23889293a65cb3dbd96efbdd0ca8304ebe779d5ac54682d8999 not found: ID does not exist"
Sep 17 18:02:14.011759 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:14.011729    2585 scope.go:117] "RemoveContainer" containerID="66ec9e531920de6a980fd21294dc734c6fcea63c484557d1eb70d5337dc3506a"
Sep 17 18:02:14.012004 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:02:14.011983    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66ec9e531920de6a980fd21294dc734c6fcea63c484557d1eb70d5337dc3506a\": container with ID starting with 66ec9e531920de6a980fd21294dc734c6fcea63c484557d1eb70d5337dc3506a not found: ID does not exist" containerID="66ec9e531920de6a980fd21294dc734c6fcea63c484557d1eb70d5337dc3506a"
Sep 17 18:02:14.012055 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:14.012011    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66ec9e531920de6a980fd21294dc734c6fcea63c484557d1eb70d5337dc3506a"} err="failed to get container status \"66ec9e531920de6a980fd21294dc734c6fcea63c484557d1eb70d5337dc3506a\": rpc error: code = NotFound desc = could not find container \"66ec9e531920de6a980fd21294dc734c6fcea63c484557d1eb70d5337dc3506a\": container with ID starting with 66ec9e531920de6a980fd21294dc734c6fcea63c484557d1eb70d5337dc3506a not found: ID does not exist"
Sep 17 18:02:14.012055 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:14.012035    2585 scope.go:117] "RemoveContainer" containerID="8d13a8b2228c0950b4b4a91ace00986789c5c5aa30ff89e07c252fe7f21b0743"
Sep 17 18:02:14.012267 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:02:14.012252    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d13a8b2228c0950b4b4a91ace00986789c5c5aa30ff89e07c252fe7f21b0743\": container with ID starting with 8d13a8b2228c0950b4b4a91ace00986789c5c5aa30ff89e07c252fe7f21b0743 not found: ID does not exist" containerID="8d13a8b2228c0950b4b4a91ace00986789c5c5aa30ff89e07c252fe7f21b0743"
Sep 17 18:02:14.012313 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:14.012271    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d13a8b2228c0950b4b4a91ace00986789c5c5aa30ff89e07c252fe7f21b0743"} err="failed to get container status \"8d13a8b2228c0950b4b4a91ace00986789c5c5aa30ff89e07c252fe7f21b0743\": rpc error: code = NotFound desc = could not find container \"8d13a8b2228c0950b4b4a91ace00986789c5c5aa30ff89e07c252fe7f21b0743\": container with ID starting with 8d13a8b2228c0950b4b4a91ace00986789c5c5aa30ff89e07c252fe7f21b0743 not found: ID does not exist"
Sep 17 18:02:14.014580 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:14.014565    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86e0f7c8-3f94-4ab8-9fd6-463dca55b391-kserve-provision-location\") pod \"86e0f7c8-3f94-4ab8-9fd6-463dca55b391\" (UID: \"86e0f7c8-3f94-4ab8-9fd6-463dca55b391\") "
Sep 17 18:02:14.014638 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:14.014622    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2l7jk\" (UniqueName: \"kubernetes.io/projected/86e0f7c8-3f94-4ab8-9fd6-463dca55b391-kube-api-access-2l7jk\") pod \"86e0f7c8-3f94-4ab8-9fd6-463dca55b391\" (UID: \"86e0f7c8-3f94-4ab8-9fd6-463dca55b391\") "
Sep 17 18:02:14.014675 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:14.014645    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86e0f7c8-3f94-4ab8-9fd6-463dca55b391-kube-rbac-proxy-sar-config\") pod \"86e0f7c8-3f94-4ab8-9fd6-463dca55b391\" (UID: \"86e0f7c8-3f94-4ab8-9fd6-463dca55b391\") "
Sep 17 18:02:14.014708 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:14.014682    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86e0f7c8-3f94-4ab8-9fd6-463dca55b391-cabundle-cert\") pod \"86e0f7c8-3f94-4ab8-9fd6-463dca55b391\" (UID: \"86e0f7c8-3f94-4ab8-9fd6-463dca55b391\") "
Sep 17 18:02:14.014747 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:14.014710    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86e0f7c8-3f94-4ab8-9fd6-463dca55b391-proxy-tls\") pod \"86e0f7c8-3f94-4ab8-9fd6-463dca55b391\" (UID: \"86e0f7c8-3f94-4ab8-9fd6-463dca55b391\") "
Sep 17 18:02:14.015044 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:14.015019    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86e0f7c8-3f94-4ab8-9fd6-463dca55b391-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "86e0f7c8-3f94-4ab8-9fd6-463dca55b391" (UID: "86e0f7c8-3f94-4ab8-9fd6-463dca55b391"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:02:14.015108 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:14.015034    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86e0f7c8-3f94-4ab8-9fd6-463dca55b391-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "86e0f7c8-3f94-4ab8-9fd6-463dca55b391" (UID: "86e0f7c8-3f94-4ab8-9fd6-463dca55b391"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:02:14.015601 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:14.015574    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e0f7c8-3f94-4ab8-9fd6-463dca55b391-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "86e0f7c8-3f94-4ab8-9fd6-463dca55b391" (UID: "86e0f7c8-3f94-4ab8-9fd6-463dca55b391"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:02:14.016743 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:14.016715    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86e0f7c8-3f94-4ab8-9fd6-463dca55b391-kube-api-access-2l7jk" (OuterVolumeSpecName: "kube-api-access-2l7jk") pod "86e0f7c8-3f94-4ab8-9fd6-463dca55b391" (UID: "86e0f7c8-3f94-4ab8-9fd6-463dca55b391"). InnerVolumeSpecName "kube-api-access-2l7jk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:02:14.016816 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:14.016790    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86e0f7c8-3f94-4ab8-9fd6-463dca55b391-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "86e0f7c8-3f94-4ab8-9fd6-463dca55b391" (UID: "86e0f7c8-3f94-4ab8-9fd6-463dca55b391"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:02:14.115266 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:14.115179    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l7jk\" (UniqueName: \"kubernetes.io/projected/86e0f7c8-3f94-4ab8-9fd6-463dca55b391-kube-api-access-2l7jk\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:02:14.115266 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:14.115221    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86e0f7c8-3f94-4ab8-9fd6-463dca55b391-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:02:14.115266 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:14.115236    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86e0f7c8-3f94-4ab8-9fd6-463dca55b391-cabundle-cert\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:02:14.115266 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:14.115249    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86e0f7c8-3f94-4ab8-9fd6-463dca55b391-proxy-tls\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:02:14.115266 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:14.115260    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86e0f7c8-3f94-4ab8-9fd6-463dca55b391-kserve-provision-location\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:02:14.315826 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:14.315793    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk"]
Sep 17 18:02:14.324673 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:14.324644    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk"]
Sep 17 18:02:14.873365 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:14.873308    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-f2zxk" podUID="86e0f7c8-3f94-4ab8-9fd6-463dca55b391" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 17 18:02:16.104049 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:16.104019    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86e0f7c8-3f94-4ab8-9fd6-463dca55b391" path="/var/lib/kubelet/pods/86e0f7c8-3f94-4ab8-9fd6-463dca55b391/volumes"
Sep 17 18:02:17.460341 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:17.460307    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:02:17.494313 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:17.494281    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz"]
Sep 17 18:02:17.494580 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:17.494566    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86e0f7c8-3f94-4ab8-9fd6-463dca55b391" containerName="kube-rbac-proxy"
Sep 17 18:02:17.494626 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:17.494581    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="86e0f7c8-3f94-4ab8-9fd6-463dca55b391" containerName="kube-rbac-proxy"
Sep 17 18:02:17.494626 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:17.494597    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86e0f7c8-3f94-4ab8-9fd6-463dca55b391" containerName="storage-initializer"
Sep 17 18:02:17.494626 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:17.494603    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="86e0f7c8-3f94-4ab8-9fd6-463dca55b391" containerName="storage-initializer"
Sep 17 18:02:17.494626 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:17.494609    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86e0f7c8-3f94-4ab8-9fd6-463dca55b391" containerName="kserve-container"
Sep 17 18:02:17.494626 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:17.494615    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="86e0f7c8-3f94-4ab8-9fd6-463dca55b391" containerName="kserve-container"
Sep 17 18:02:17.494774 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:17.494656    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="86e0f7c8-3f94-4ab8-9fd6-463dca55b391" containerName="kserve-container"
Sep 17 18:02:17.494774 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:17.494662    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="86e0f7c8-3f94-4ab8-9fd6-463dca55b391" containerName="kube-rbac-proxy"
Sep 17 18:02:17.499651 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:17.499633    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz"
Sep 17 18:02:17.502651 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:17.502633    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lps5w\""
Sep 17 18:02:17.502778 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:17.502637    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:02:17.502778 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:17.502669    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-predictor-serving-cert\""
Sep 17 18:02:17.503670 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:17.503653    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:02:17.503774 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:17.503654    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:02:17.503774 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:17.503714    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:02:17.503774 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:17.503771    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-kube-rbac-proxy-sar-config\""
Sep 17 18:02:17.517943 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:17.517924    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz"]
Sep 17 18:02:17.637092 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:17.637050    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e36b13eb-0f7e-4d04-8c59-701f19c318a0-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz\" (UID: \"e36b13eb-0f7e-4d04-8c59-701f19c318a0\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz"
Sep 17 18:02:17.637092 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:17.637088    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e36b13eb-0f7e-4d04-8c59-701f19c318a0-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz\" (UID: \"e36b13eb-0f7e-4d04-8c59-701f19c318a0\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz"
Sep 17 18:02:17.637304 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:17.637106    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e36b13eb-0f7e-4d04-8c59-701f19c318a0-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz\" (UID: \"e36b13eb-0f7e-4d04-8c59-701f19c318a0\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz"
Sep 17 18:02:17.637304 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:17.637176    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e36b13eb-0f7e-4d04-8c59-701f19c318a0-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz\" (UID: \"e36b13eb-0f7e-4d04-8c59-701f19c318a0\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz"
Sep 17 18:02:17.637304 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:17.637208    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4sw9\" (UniqueName: \"kubernetes.io/projected/e36b13eb-0f7e-4d04-8c59-701f19c318a0-kube-api-access-x4sw9\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz\" (UID: \"e36b13eb-0f7e-4d04-8c59-701f19c318a0\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz"
Sep 17 18:02:17.737962 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:17.737873    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e36b13eb-0f7e-4d04-8c59-701f19c318a0-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz\" (UID: \"e36b13eb-0f7e-4d04-8c59-701f19c318a0\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz"
Sep 17 18:02:17.737962 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:17.737914    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4sw9\" (UniqueName: \"kubernetes.io/projected/e36b13eb-0f7e-4d04-8c59-701f19c318a0-kube-api-access-x4sw9\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz\" (UID: \"e36b13eb-0f7e-4d04-8c59-701f19c318a0\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz"
Sep 17 18:02:17.737962 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:17.737957    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e36b13eb-0f7e-4d04-8c59-701f19c318a0-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz\" (UID: \"e36b13eb-0f7e-4d04-8c59-701f19c318a0\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz"
Sep 17 18:02:17.738242 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:17.737978    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e36b13eb-0f7e-4d04-8c59-701f19c318a0-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz\" (UID: \"e36b13eb-0f7e-4d04-8c59-701f19c318a0\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz"
Sep 17 18:02:17.738242 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:17.737993    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e36b13eb-0f7e-4d04-8c59-701f19c318a0-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz\" (UID: \"e36b13eb-0f7e-4d04-8c59-701f19c318a0\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz"
Sep 17 18:02:17.738242 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:02:17.738041    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-serving-cert: secret "isvc-autogluon-ts-v1-noslash-predictor-serving-cert" not found
Sep 17 18:02:17.738242 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:02:17.738121    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e36b13eb-0f7e-4d04-8c59-701f19c318a0-proxy-tls podName:e36b13eb-0f7e-4d04-8c59-701f19c318a0 nodeName:}" failed. No retries permitted until 2026-09-17 18:02:18.238098444 +0000 UTC m=+1674.651888863 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e36b13eb-0f7e-4d04-8c59-701f19c318a0-proxy-tls") pod "isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz" (UID: "e36b13eb-0f7e-4d04-8c59-701f19c318a0") : secret "isvc-autogluon-ts-v1-noslash-predictor-serving-cert" not found
Sep 17 18:02:17.738447 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:17.738409    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e36b13eb-0f7e-4d04-8c59-701f19c318a0-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz\" (UID: \"e36b13eb-0f7e-4d04-8c59-701f19c318a0\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz"
Sep 17 18:02:17.738667 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:17.738650    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e36b13eb-0f7e-4d04-8c59-701f19c318a0-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz\" (UID: \"e36b13eb-0f7e-4d04-8c59-701f19c318a0\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz"
Sep 17 18:02:17.738715 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:17.738670    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e36b13eb-0f7e-4d04-8c59-701f19c318a0-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz\" (UID: \"e36b13eb-0f7e-4d04-8c59-701f19c318a0\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz"
Sep 17 18:02:17.746884 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:17.746866    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4sw9\" (UniqueName: \"kubernetes.io/projected/e36b13eb-0f7e-4d04-8c59-701f19c318a0-kube-api-access-x4sw9\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz\" (UID: \"e36b13eb-0f7e-4d04-8c59-701f19c318a0\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz"
Sep 17 18:02:18.241562 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:18.241508    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e36b13eb-0f7e-4d04-8c59-701f19c318a0-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz\" (UID: \"e36b13eb-0f7e-4d04-8c59-701f19c318a0\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz"
Sep 17 18:02:18.244078 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:18.244048    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e36b13eb-0f7e-4d04-8c59-701f19c318a0-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz\" (UID: \"e36b13eb-0f7e-4d04-8c59-701f19c318a0\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz"
Sep 17 18:02:18.409226 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:18.409182    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz"
Sep 17 18:02:18.683170 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:18.683147    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz"]
Sep 17 18:02:18.685706 ip-10-0-133-139 kubenswrapper[2585]: W0917 18:02:18.685679    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode36b13eb_0f7e_4d04_8c59_701f19c318a0.slice/crio-81087da69b3f501034e07fb526cb8a34f0d0098b84e5e8d120088a3669b2dbb2 WatchSource:0}: Error finding container 81087da69b3f501034e07fb526cb8a34f0d0098b84e5e8d120088a3669b2dbb2: Status 404 returned error can't find the container with id 81087da69b3f501034e07fb526cb8a34f0d0098b84e5e8d120088a3669b2dbb2
Sep 17 18:02:18.858591 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:18.858559    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:02:19.008449 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:19.008417    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz" event={"ID":"e36b13eb-0f7e-4d04-8c59-701f19c318a0","Type":"ContainerStarted","Data":"5b2eee6cd01ea7ff4621b11d3ad5a16561926637e7160eeef3432ed8b61dba21"}
Sep 17 18:02:19.008625 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:19.008478    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz" event={"ID":"e36b13eb-0f7e-4d04-8c59-701f19c318a0","Type":"ContainerStarted","Data":"81087da69b3f501034e07fb526cb8a34f0d0098b84e5e8d120088a3669b2dbb2"}
Sep 17 18:02:25.032038 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:25.032006    2585 generic.go:356] "Generic (PLEG): container finished" podID="e36b13eb-0f7e-4d04-8c59-701f19c318a0" containerID="5b2eee6cd01ea7ff4621b11d3ad5a16561926637e7160eeef3432ed8b61dba21" exitCode=0
Sep 17 18:02:25.032373 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:25.032078    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz" event={"ID":"e36b13eb-0f7e-4d04-8c59-701f19c318a0","Type":"ContainerDied","Data":"5b2eee6cd01ea7ff4621b11d3ad5a16561926637e7160eeef3432ed8b61dba21"}
Sep 17 18:02:26.037757 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:26.037719    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz" event={"ID":"e36b13eb-0f7e-4d04-8c59-701f19c318a0","Type":"ContainerStarted","Data":"097fa72424d8c89e17fd1f79e73a1af4d8e2953171231f700fc369cda9763027"}
Sep 17 18:02:26.038228 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:26.037795    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz" event={"ID":"e36b13eb-0f7e-4d04-8c59-701f19c318a0","Type":"ContainerStarted","Data":"fe7fa9bc80ea2ae9291e9f364add6fe24f70d3b1eab39f6726495d030345a538"}
Sep 17 18:02:26.038228 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:26.038013    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz"
Sep 17 18:02:26.063449 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:26.063399    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz" podStartSLOduration=3.185379238 podStartE2EDuration="9.063384102s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.878004864s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:02:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:02:26.062155287 +0000 UTC m=+1682.475945727" watchObservedRunningTime="2026-09-17 18:02:26.063384102 +0000 UTC m=+1682.477174540"
Sep 17 18:02:27.041814 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:27.041779    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz"
Sep 17 18:02:27.043306 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:27.043271    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz" podUID="e36b13eb-0f7e-4d04-8c59-701f19c318a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 18:02:28.046602 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:28.046558    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz" podUID="e36b13eb-0f7e-4d04-8c59-701f19c318a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 18:02:33.051056 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:33.051032    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz"
Sep 17 18:02:33.051589 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:33.051563    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz" podUID="e36b13eb-0f7e-4d04-8c59-701f19c318a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 18:02:43.053304 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:43.053276    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz"
Sep 17 18:02:47.150299 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:47.150218    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:02:52.400365 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:52.400329    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:02:52.589220 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:52.589184    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:02:52.920161 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:52.920132    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz"]
Sep 17 18:02:52.920385 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:52.920362    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz" podUID="e36b13eb-0f7e-4d04-8c59-701f19c318a0" containerName="kserve-container" containerID="cri-o://fe7fa9bc80ea2ae9291e9f364add6fe24f70d3b1eab39f6726495d030345a538" gracePeriod=30
Sep 17 18:02:52.920475 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:52.920441    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz" podUID="e36b13eb-0f7e-4d04-8c59-701f19c318a0" containerName="kube-rbac-proxy" containerID="cri-o://097fa72424d8c89e17fd1f79e73a1af4d8e2953171231f700fc369cda9763027" gracePeriod=30
Sep 17 18:02:52.990928 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:52.990896    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:02:53.047389 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:53.047357    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz" podUID="e36b13eb-0f7e-4d04-8c59-701f19c318a0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Sep 17 18:02:53.051654 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:53.051624    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz" podUID="e36b13eb-0f7e-4d04-8c59-701f19c318a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 18:02:53.139820 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:53.139784    2585 generic.go:356] "Generic (PLEG): container finished" podID="e36b13eb-0f7e-4d04-8c59-701f19c318a0" containerID="097fa72424d8c89e17fd1f79e73a1af4d8e2953171231f700fc369cda9763027" exitCode=2
Sep 17 18:02:53.139978 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:53.139858    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz" event={"ID":"e36b13eb-0f7e-4d04-8c59-701f19c318a0","Type":"ContainerDied","Data":"097fa72424d8c89e17fd1f79e73a1af4d8e2953171231f700fc369cda9763027"}
Sep 17 18:02:54.356516 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:54.356477    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz"
Sep 17 18:02:54.519329 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:54.519296    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e36b13eb-0f7e-4d04-8c59-701f19c318a0-kube-rbac-proxy-sar-config\") pod \"e36b13eb-0f7e-4d04-8c59-701f19c318a0\" (UID: \"e36b13eb-0f7e-4d04-8c59-701f19c318a0\") "
Sep 17 18:02:54.519541 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:54.519343    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x4sw9\" (UniqueName: \"kubernetes.io/projected/e36b13eb-0f7e-4d04-8c59-701f19c318a0-kube-api-access-x4sw9\") pod \"e36b13eb-0f7e-4d04-8c59-701f19c318a0\" (UID: \"e36b13eb-0f7e-4d04-8c59-701f19c318a0\") "
Sep 17 18:02:54.519541 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:54.519369    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e36b13eb-0f7e-4d04-8c59-701f19c318a0-proxy-tls\") pod \"e36b13eb-0f7e-4d04-8c59-701f19c318a0\" (UID: \"e36b13eb-0f7e-4d04-8c59-701f19c318a0\") "
Sep 17 18:02:54.519541 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:54.519436    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e36b13eb-0f7e-4d04-8c59-701f19c318a0-kserve-provision-location\") pod \"e36b13eb-0f7e-4d04-8c59-701f19c318a0\" (UID: \"e36b13eb-0f7e-4d04-8c59-701f19c318a0\") "
Sep 17 18:02:54.519541 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:54.519451    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e36b13eb-0f7e-4d04-8c59-701f19c318a0-cabundle-cert\") pod \"e36b13eb-0f7e-4d04-8c59-701f19c318a0\" (UID: \"e36b13eb-0f7e-4d04-8c59-701f19c318a0\") "
Sep 17 18:02:54.519764 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:54.519741    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e36b13eb-0f7e-4d04-8c59-701f19c318a0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e36b13eb-0f7e-4d04-8c59-701f19c318a0" (UID: "e36b13eb-0f7e-4d04-8c59-701f19c318a0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:02:54.519955 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:54.519924    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e36b13eb-0f7e-4d04-8c59-701f19c318a0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e36b13eb-0f7e-4d04-8c59-701f19c318a0" (UID: "e36b13eb-0f7e-4d04-8c59-701f19c318a0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:02:54.520419 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:54.520401    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e36b13eb-0f7e-4d04-8c59-701f19c318a0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e36b13eb-0f7e-4d04-8c59-701f19c318a0" (UID: "e36b13eb-0f7e-4d04-8c59-701f19c318a0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:02:54.521574 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:54.521548    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e36b13eb-0f7e-4d04-8c59-701f19c318a0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e36b13eb-0f7e-4d04-8c59-701f19c318a0" (UID: "e36b13eb-0f7e-4d04-8c59-701f19c318a0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:02:54.521672 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:54.521640    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e36b13eb-0f7e-4d04-8c59-701f19c318a0-kube-api-access-x4sw9" (OuterVolumeSpecName: "kube-api-access-x4sw9") pod "e36b13eb-0f7e-4d04-8c59-701f19c318a0" (UID: "e36b13eb-0f7e-4d04-8c59-701f19c318a0"). InnerVolumeSpecName "kube-api-access-x4sw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:02:54.620890 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:54.620852    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e36b13eb-0f7e-4d04-8c59-701f19c318a0-kserve-provision-location\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:02:54.620890 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:54.620880    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e36b13eb-0f7e-4d04-8c59-701f19c318a0-cabundle-cert\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:02:54.620890 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:54.620890    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e36b13eb-0f7e-4d04-8c59-701f19c318a0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:02:54.620890 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:54.620899    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4sw9\" (UniqueName: \"kubernetes.io/projected/e36b13eb-0f7e-4d04-8c59-701f19c318a0-kube-api-access-x4sw9\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:02:54.621152 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:54.620910    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e36b13eb-0f7e-4d04-8c59-701f19c318a0-proxy-tls\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:02:55.149099 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:55.149067    2585 generic.go:356] "Generic (PLEG): container finished" podID="e36b13eb-0f7e-4d04-8c59-701f19c318a0" containerID="fe7fa9bc80ea2ae9291e9f364add6fe24f70d3b1eab39f6726495d030345a538" exitCode=0
Sep 17 18:02:55.149308 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:55.149118    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz" event={"ID":"e36b13eb-0f7e-4d04-8c59-701f19c318a0","Type":"ContainerDied","Data":"fe7fa9bc80ea2ae9291e9f364add6fe24f70d3b1eab39f6726495d030345a538"}
Sep 17 18:02:55.149308 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:55.149133    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz"
Sep 17 18:02:55.149308 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:55.149163    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz" event={"ID":"e36b13eb-0f7e-4d04-8c59-701f19c318a0","Type":"ContainerDied","Data":"81087da69b3f501034e07fb526cb8a34f0d0098b84e5e8d120088a3669b2dbb2"}
Sep 17 18:02:55.149308 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:55.149189    2585 scope.go:117] "RemoveContainer" containerID="097fa72424d8c89e17fd1f79e73a1af4d8e2953171231f700fc369cda9763027"
Sep 17 18:02:55.157286 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:55.157270    2585 scope.go:117] "RemoveContainer" containerID="fe7fa9bc80ea2ae9291e9f364add6fe24f70d3b1eab39f6726495d030345a538"
Sep 17 18:02:55.164388 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:55.164371    2585 scope.go:117] "RemoveContainer" containerID="5b2eee6cd01ea7ff4621b11d3ad5a16561926637e7160eeef3432ed8b61dba21"
Sep 17 18:02:55.170866 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:55.170849    2585 scope.go:117] "RemoveContainer" containerID="097fa72424d8c89e17fd1f79e73a1af4d8e2953171231f700fc369cda9763027"
Sep 17 18:02:55.171106 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:02:55.171088    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"097fa72424d8c89e17fd1f79e73a1af4d8e2953171231f700fc369cda9763027\": container with ID starting with 097fa72424d8c89e17fd1f79e73a1af4d8e2953171231f700fc369cda9763027 not found: ID does not exist" containerID="097fa72424d8c89e17fd1f79e73a1af4d8e2953171231f700fc369cda9763027"
Sep 17 18:02:55.171157 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:55.171114    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"097fa72424d8c89e17fd1f79e73a1af4d8e2953171231f700fc369cda9763027"} err="failed to get container status \"097fa72424d8c89e17fd1f79e73a1af4d8e2953171231f700fc369cda9763027\": rpc error: code = NotFound desc = could not find container \"097fa72424d8c89e17fd1f79e73a1af4d8e2953171231f700fc369cda9763027\": container with ID starting with 097fa72424d8c89e17fd1f79e73a1af4d8e2953171231f700fc369cda9763027 not found: ID does not exist"
Sep 17 18:02:55.171157 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:55.171131    2585 scope.go:117] "RemoveContainer" containerID="fe7fa9bc80ea2ae9291e9f364add6fe24f70d3b1eab39f6726495d030345a538"
Sep 17 18:02:55.171341 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:02:55.171323    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe7fa9bc80ea2ae9291e9f364add6fe24f70d3b1eab39f6726495d030345a538\": container with ID starting with fe7fa9bc80ea2ae9291e9f364add6fe24f70d3b1eab39f6726495d030345a538 not found: ID does not exist" containerID="fe7fa9bc80ea2ae9291e9f364add6fe24f70d3b1eab39f6726495d030345a538"
Sep 17 18:02:55.171384 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:55.171348    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe7fa9bc80ea2ae9291e9f364add6fe24f70d3b1eab39f6726495d030345a538"} err="failed to get container status \"fe7fa9bc80ea2ae9291e9f364add6fe24f70d3b1eab39f6726495d030345a538\": rpc error: code = NotFound desc = could not find container \"fe7fa9bc80ea2ae9291e9f364add6fe24f70d3b1eab39f6726495d030345a538\": container with ID starting with fe7fa9bc80ea2ae9291e9f364add6fe24f70d3b1eab39f6726495d030345a538 not found: ID does not exist"
Sep 17 18:02:55.171384 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:55.171365    2585 scope.go:117] "RemoveContainer" containerID="5b2eee6cd01ea7ff4621b11d3ad5a16561926637e7160eeef3432ed8b61dba21"
Sep 17 18:02:55.171597 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:02:55.171580    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b2eee6cd01ea7ff4621b11d3ad5a16561926637e7160eeef3432ed8b61dba21\": container with ID starting with 5b2eee6cd01ea7ff4621b11d3ad5a16561926637e7160eeef3432ed8b61dba21 not found: ID does not exist" containerID="5b2eee6cd01ea7ff4621b11d3ad5a16561926637e7160eeef3432ed8b61dba21"
Sep 17 18:02:55.171649 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:55.171601    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b2eee6cd01ea7ff4621b11d3ad5a16561926637e7160eeef3432ed8b61dba21"} err="failed to get container status \"5b2eee6cd01ea7ff4621b11d3ad5a16561926637e7160eeef3432ed8b61dba21\": rpc error: code = NotFound desc = could not find container \"5b2eee6cd01ea7ff4621b11d3ad5a16561926637e7160eeef3432ed8b61dba21\": container with ID starting with 5b2eee6cd01ea7ff4621b11d3ad5a16561926637e7160eeef3432ed8b61dba21 not found: ID does not exist"
Sep 17 18:02:55.176101 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:55.176081    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz"]
Sep 17 18:02:55.177122 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:55.177104    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-h42mz"]
Sep 17 18:02:56.107169 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:56.107137    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e36b13eb-0f7e-4d04-8c59-701f19c318a0" path="/var/lib/kubelet/pods/e36b13eb-0f7e-4d04-8c59-701f19c318a0/volumes"
Sep 17 18:02:58.179411 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:58.179379    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qq26k"]
Sep 17 18:02:58.179795 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:58.179625    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e36b13eb-0f7e-4d04-8c59-701f19c318a0" containerName="storage-initializer"
Sep 17 18:02:58.179795 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:58.179636    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e36b13eb-0f7e-4d04-8c59-701f19c318a0" containerName="storage-initializer"
Sep 17 18:02:58.179795 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:58.179652    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e36b13eb-0f7e-4d04-8c59-701f19c318a0" containerName="kserve-container"
Sep 17 18:02:58.179795 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:58.179657    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e36b13eb-0f7e-4d04-8c59-701f19c318a0" containerName="kserve-container"
Sep 17 18:02:58.179795 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:58.179665    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e36b13eb-0f7e-4d04-8c59-701f19c318a0" containerName="kube-rbac-proxy"
Sep 17 18:02:58.179795 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:58.179671    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e36b13eb-0f7e-4d04-8c59-701f19c318a0" containerName="kube-rbac-proxy"
Sep 17 18:02:58.179795 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:58.179705    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="e36b13eb-0f7e-4d04-8c59-701f19c318a0" containerName="kserve-container"
Sep 17 18:02:58.179795 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:58.179714    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="e36b13eb-0f7e-4d04-8c59-701f19c318a0" containerName="kube-rbac-proxy"
Sep 17 18:02:58.182448 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:58.182430    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qq26k"
Sep 17 18:02:58.191388 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:58.191365    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Sep 17 18:02:58.191535 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:58.191412    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:02:58.191535 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:58.191373    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 17 18:02:58.191535 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:58.191368    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:02:58.191689 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:58.191579    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lps5w\""
Sep 17 18:02:58.192766 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:58.192749    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:02:58.192880 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:58.192806    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:02:58.194160 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:58.194138    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:02:58.203210 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:58.203186    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qq26k"]
Sep 17 18:02:58.228601 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:58.228557    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-t4pxj"]
Sep 17 18:02:58.231822 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:58.231804    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-t4pxj"
Sep 17 18:02:58.235306 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:58.235280    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 17 18:02:58.255682 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:58.255654    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-t4pxj"]
Sep 17 18:02:58.323681 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:58.323647    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:02:58.345679 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:58.345647    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c25567c-e8a8-46fa-845a-34f9aa333b1f-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-96f77bdbd-qq26k\" (UID: \"5c25567c-e8a8-46fa-845a-34f9aa333b1f\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qq26k"
Sep 17 18:02:58.345822 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:58.345683    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq8h7\" (UniqueName: \"kubernetes.io/projected/5c25567c-e8a8-46fa-845a-34f9aa333b1f-kube-api-access-wq8h7\") pod \"isvc-canary-create-v2-predictor-96f77bdbd-qq26k\" (UID: \"5c25567c-e8a8-46fa-845a-34f9aa333b1f\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qq26k"
Sep 17 18:02:58.345822 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:58.345714    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/abee0c45-b6de-49cb-b16f-28957a9926bf-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-6474b875b9-t4pxj\" (UID: \"abee0c45-b6de-49cb-b16f-28957a9926bf\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-t4pxj"
Sep 17 18:02:58.345822 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:58.345736    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/abee0c45-b6de-49cb-b16f-28957a9926bf-cabundle-cert\") pod \"isvc-canary-create-predictor-6474b875b9-t4pxj\" (UID: \"abee0c45-b6de-49cb-b16f-28957a9926bf\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-t4pxj"
Sep 17 18:02:58.345822 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:58.345777    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/abee0c45-b6de-49cb-b16f-28957a9926bf-kserve-provision-location\") pod \"isvc-canary-create-predictor-6474b875b9-t4pxj\" (UID: \"abee0c45-b6de-49cb-b16f-28957a9926bf\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-t4pxj"
Sep 17 18:02:58.346013 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:58.345839    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abee0c45-b6de-49cb-b16f-28957a9926bf-proxy-tls\") pod \"isvc-canary-create-predictor-6474b875b9-t4pxj\" (UID: \"abee0c45-b6de-49cb-b16f-28957a9926bf\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-t4pxj"
Sep 17 18:02:58.346013 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:58.345897    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c25567c-e8a8-46fa-845a-34f9aa333b1f-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-96f77bdbd-qq26k\" (UID: \"5c25567c-e8a8-46fa-845a-34f9aa333b1f\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qq26k"
Sep 17 18:02:58.346013 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:58.345923    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c25567c-e8a8-46fa-845a-34f9aa333b1f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-96f77bdbd-qq26k\" (UID: \"5c25567c-e8a8-46fa-845a-34f9aa333b1f\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qq26k"
Sep 17 18:02:58.346013 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:58.345943    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c25567c-e8a8-46fa-845a-34f9aa333b1f-proxy-tls\") pod \"isvc-canary-create-v2-predictor-96f77bdbd-qq26k\" (UID: \"5c25567c-e8a8-46fa-845a-34f9aa333b1f\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qq26k"
Sep 17 18:02:58.346013 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:58.345964    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxfgr\" (UniqueName: \"kubernetes.io/projected/abee0c45-b6de-49cb-b16f-28957a9926bf-kube-api-access-sxfgr\") pod \"isvc-canary-create-predictor-6474b875b9-t4pxj\" (UID: \"abee0c45-b6de-49cb-b16f-28957a9926bf\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-t4pxj"
Sep 17 18:02:58.447047 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:58.446964    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c25567c-e8a8-46fa-845a-34f9aa333b1f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-96f77bdbd-qq26k\" (UID: \"5c25567c-e8a8-46fa-845a-34f9aa333b1f\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qq26k"
Sep 17 18:02:58.447047 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:58.447005    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c25567c-e8a8-46fa-845a-34f9aa333b1f-proxy-tls\") pod \"isvc-canary-create-v2-predictor-96f77bdbd-qq26k\" (UID: \"5c25567c-e8a8-46fa-845a-34f9aa333b1f\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qq26k"
Sep 17 18:02:58.447047 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:58.447028    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxfgr\" (UniqueName: \"kubernetes.io/projected/abee0c45-b6de-49cb-b16f-28957a9926bf-kube-api-access-sxfgr\") pod \"isvc-canary-create-predictor-6474b875b9-t4pxj\" (UID: \"abee0c45-b6de-49cb-b16f-28957a9926bf\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-t4pxj"
Sep 17 18:02:58.447353 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:58.447053    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c25567c-e8a8-46fa-845a-34f9aa333b1f-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-96f77bdbd-qq26k\" (UID: \"5c25567c-e8a8-46fa-845a-34f9aa333b1f\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qq26k"
Sep 17 18:02:58.447353 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:58.447072    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq8h7\" (UniqueName: \"kubernetes.io/projected/5c25567c-e8a8-46fa-845a-34f9aa333b1f-kube-api-access-wq8h7\") pod \"isvc-canary-create-v2-predictor-96f77bdbd-qq26k\" (UID: \"5c25567c-e8a8-46fa-845a-34f9aa333b1f\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qq26k"
Sep 17 18:02:58.447353 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:58.447221    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/abee0c45-b6de-49cb-b16f-28957a9926bf-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-6474b875b9-t4pxj\" (UID: \"abee0c45-b6de-49cb-b16f-28957a9926bf\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-t4pxj"
Sep 17 18:02:58.447353 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:58.447252    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/abee0c45-b6de-49cb-b16f-28957a9926bf-cabundle-cert\") pod \"isvc-canary-create-predictor-6474b875b9-t4pxj\" (UID: \"abee0c45-b6de-49cb-b16f-28957a9926bf\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-t4pxj"
Sep 17 18:02:58.447353 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:58.447307    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/abee0c45-b6de-49cb-b16f-28957a9926bf-kserve-provision-location\") pod \"isvc-canary-create-predictor-6474b875b9-t4pxj\" (UID: \"abee0c45-b6de-49cb-b16f-28957a9926bf\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-t4pxj"
Sep 17 18:02:58.447353 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:58.447345    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abee0c45-b6de-49cb-b16f-28957a9926bf-proxy-tls\") pod \"isvc-canary-create-predictor-6474b875b9-t4pxj\" (UID: \"abee0c45-b6de-49cb-b16f-28957a9926bf\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-t4pxj"
Sep 17 18:02:58.447708 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:58.447377    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c25567c-e8a8-46fa-845a-34f9aa333b1f-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-96f77bdbd-qq26k\" (UID: \"5c25567c-e8a8-46fa-845a-34f9aa333b1f\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qq26k"
Sep 17 18:02:58.447708 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:58.447427    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c25567c-e8a8-46fa-845a-34f9aa333b1f-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-96f77bdbd-qq26k\" (UID: \"5c25567c-e8a8-46fa-845a-34f9aa333b1f\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qq26k"
Sep 17 18:02:58.447816 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:58.447759    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/abee0c45-b6de-49cb-b16f-28957a9926bf-kserve-provision-location\") pod \"isvc-canary-create-predictor-6474b875b9-t4pxj\" (UID: \"abee0c45-b6de-49cb-b16f-28957a9926bf\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-t4pxj"
Sep 17 18:02:58.448029 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:58.447980    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/abee0c45-b6de-49cb-b16f-28957a9926bf-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-6474b875b9-t4pxj\" (UID: \"abee0c45-b6de-49cb-b16f-28957a9926bf\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-t4pxj"
Sep 17 18:02:58.448121 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:58.448058    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/abee0c45-b6de-49cb-b16f-28957a9926bf-cabundle-cert\") pod \"isvc-canary-create-predictor-6474b875b9-t4pxj\" (UID: \"abee0c45-b6de-49cb-b16f-28957a9926bf\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-t4pxj"
Sep 17 18:02:58.448121 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:58.448080    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c25567c-e8a8-46fa-845a-34f9aa333b1f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-96f77bdbd-qq26k\" (UID: \"5c25567c-e8a8-46fa-845a-34f9aa333b1f\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qq26k"
Sep 17 18:02:58.448239 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:58.448217    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c25567c-e8a8-46fa-845a-34f9aa333b1f-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-96f77bdbd-qq26k\" (UID: \"5c25567c-e8a8-46fa-845a-34f9aa333b1f\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qq26k"
Sep 17 18:02:58.449778 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:58.449759    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c25567c-e8a8-46fa-845a-34f9aa333b1f-proxy-tls\") pod \"isvc-canary-create-v2-predictor-96f77bdbd-qq26k\" (UID: \"5c25567c-e8a8-46fa-845a-34f9aa333b1f\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qq26k"
Sep 17 18:02:58.449860 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:58.449775    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abee0c45-b6de-49cb-b16f-28957a9926bf-proxy-tls\") pod \"isvc-canary-create-predictor-6474b875b9-t4pxj\" (UID: \"abee0c45-b6de-49cb-b16f-28957a9926bf\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-t4pxj"
Sep 17 18:02:58.459000 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:58.458979    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxfgr\" (UniqueName: \"kubernetes.io/projected/abee0c45-b6de-49cb-b16f-28957a9926bf-kube-api-access-sxfgr\") pod \"isvc-canary-create-predictor-6474b875b9-t4pxj\" (UID: \"abee0c45-b6de-49cb-b16f-28957a9926bf\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-t4pxj"
Sep 17 18:02:58.459624 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:58.459601    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq8h7\" (UniqueName: \"kubernetes.io/projected/5c25567c-e8a8-46fa-845a-34f9aa333b1f-kube-api-access-wq8h7\") pod \"isvc-canary-create-v2-predictor-96f77bdbd-qq26k\" (UID: \"5c25567c-e8a8-46fa-845a-34f9aa333b1f\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qq26k"
Sep 17 18:02:58.460606 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:58.460581    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:02:58.492329 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:58.492305    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qq26k"
Sep 17 18:02:58.541980 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:58.541937    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-t4pxj"
Sep 17 18:02:58.795925 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:58.795790    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qq26k"]
Sep 17 18:02:58.798369 ip-10-0-133-139 kubenswrapper[2585]: W0917 18:02:58.798345    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c25567c_e8a8_46fa_845a_34f9aa333b1f.slice/crio-d9dae7dd04e5b4236b2d577ea3281f30eff5eeef043ce43ce5d5b8a478c1e3c2 WatchSource:0}: Error finding container d9dae7dd04e5b4236b2d577ea3281f30eff5eeef043ce43ce5d5b8a478c1e3c2: Status 404 returned error can't find the container with id d9dae7dd04e5b4236b2d577ea3281f30eff5eeef043ce43ce5d5b8a478c1e3c2
Sep 17 18:02:58.833470 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:58.833416    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-t4pxj"]
Sep 17 18:02:58.835540 ip-10-0-133-139 kubenswrapper[2585]: W0917 18:02:58.835515    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabee0c45_b6de_49cb_b16f_28957a9926bf.slice/crio-390b46d5c1879ff4cae50d000901064701982fac530cfb0e6c8dd9f262922e11 WatchSource:0}: Error finding container 390b46d5c1879ff4cae50d000901064701982fac530cfb0e6c8dd9f262922e11: Status 404 returned error can't find the container with id 390b46d5c1879ff4cae50d000901064701982fac530cfb0e6c8dd9f262922e11
Sep 17 18:02:59.168013 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:59.167909    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qq26k" event={"ID":"5c25567c-e8a8-46fa-845a-34f9aa333b1f","Type":"ContainerStarted","Data":"4c92e70fc5e7a2514e233b251464697390125b72b9f50d9aa27640410ede660e"}
Sep 17 18:02:59.168013 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:59.167996    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qq26k" event={"ID":"5c25567c-e8a8-46fa-845a-34f9aa333b1f","Type":"ContainerStarted","Data":"d9dae7dd04e5b4236b2d577ea3281f30eff5eeef043ce43ce5d5b8a478c1e3c2"}
Sep 17 18:02:59.169658 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:59.169627    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-t4pxj" event={"ID":"abee0c45-b6de-49cb-b16f-28957a9926bf","Type":"ContainerStarted","Data":"93fff7722912ca513270a2b3973817e2c7cf553e04ba95848140cdd753381f76"}
Sep 17 18:02:59.169812 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:02:59.169685    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-t4pxj" event={"ID":"abee0c45-b6de-49cb-b16f-28957a9926bf","Type":"ContainerStarted","Data":"390b46d5c1879ff4cae50d000901064701982fac530cfb0e6c8dd9f262922e11"}
Sep 17 18:03:03.187776 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:03:03.187739    2585 generic.go:356] "Generic (PLEG): container finished" podID="5c25567c-e8a8-46fa-845a-34f9aa333b1f" containerID="4c92e70fc5e7a2514e233b251464697390125b72b9f50d9aa27640410ede660e" exitCode=0
Sep 17 18:03:03.188268 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:03:03.187817    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qq26k" event={"ID":"5c25567c-e8a8-46fa-845a-34f9aa333b1f","Type":"ContainerDied","Data":"4c92e70fc5e7a2514e233b251464697390125b72b9f50d9aa27640410ede660e"}
Sep 17 18:03:03.189109 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:03:03.189092    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:03:03.189653 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:03:03.189602    2585 generic.go:356] "Generic (PLEG): container finished" podID="abee0c45-b6de-49cb-b16f-28957a9926bf" containerID="93fff7722912ca513270a2b3973817e2c7cf553e04ba95848140cdd753381f76" exitCode=0
Sep 17 18:03:03.189722 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:03:03.189667    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-t4pxj" event={"ID":"abee0c45-b6de-49cb-b16f-28957a9926bf","Type":"ContainerDied","Data":"93fff7722912ca513270a2b3973817e2c7cf553e04ba95848140cdd753381f76"}
Sep 17 18:03:20.286440 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:03:20.286398    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-t4pxj" event={"ID":"abee0c45-b6de-49cb-b16f-28957a9926bf","Type":"ContainerStarted","Data":"69eb562c4ae3e36b18932d3faefad6e96ed42936326518e8e0e87bc38f17a3bd"}
Sep 17 18:03:20.286948 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:03:20.286504    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-t4pxj" event={"ID":"abee0c45-b6de-49cb-b16f-28957a9926bf","Type":"ContainerStarted","Data":"410358c980b16f9bd9e032afb3ac8769099e91137a4eff05f99807223d38e6ad"}
Sep 17 18:03:20.286948 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:03:20.286713    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-t4pxj"
Sep 17 18:03:20.288571 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:03:20.288541    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qq26k" event={"ID":"5c25567c-e8a8-46fa-845a-34f9aa333b1f","Type":"ContainerStarted","Data":"43cfc498ed0e39060b34d0bdb8a535053b089170b1c72acbb1aafa9847c0f0a1"}
Sep 17 18:03:20.288680 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:03:20.288599    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qq26k" event={"ID":"5c25567c-e8a8-46fa-845a-34f9aa333b1f","Type":"ContainerStarted","Data":"0866e0abb7a24efb22532a62a4647db6fdd5288221da7b514aa563a6dd2d7bcc"}
Sep 17 18:03:20.288784 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:03:20.288770    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qq26k"
Sep 17 18:03:20.310625 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:03:20.310581    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-t4pxj" podStartSLOduration=2.477799972 podStartE2EDuration="22.310568731s" totalImagesPullingTime="16.127454592s" totalInitContainerRuntime="3.705314167s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:02:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:20.309046392 +0000 UTC m=+1736.722836832" watchObservedRunningTime="2026-09-17 18:03:20.310568731 +0000 UTC m=+1736.724359171"
Sep 17 18:03:20.338056 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:03:20.338012    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qq26k" podStartSLOduration=2.468970023 podStartE2EDuration="22.337999266s" totalImagesPullingTime="16.112977001s" totalInitContainerRuntime="3.756052242s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:02:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:20.336951134 +0000 UTC m=+1736.750741574" watchObservedRunningTime="2026-09-17 18:03:20.337999266 +0000 UTC m=+1736.751789788"
Sep 17 18:03:21.292721 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:03:21.292693    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qq26k"
Sep 17 18:03:21.293169 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:03:21.292745    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-t4pxj"
Sep 17 18:03:21.293772 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:03:21.293748    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-t4pxj" podUID="abee0c45-b6de-49cb-b16f-28957a9926bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 18:03:21.293896 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:03:21.293768    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qq26k" podUID="5c25567c-e8a8-46fa-845a-34f9aa333b1f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 18:03:22.296280 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:03:22.296241    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qq26k" podUID="5c25567c-e8a8-46fa-845a-34f9aa333b1f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 18:03:22.296737 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:03:22.296344    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-t4pxj" podUID="abee0c45-b6de-49cb-b16f-28957a9926bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 18:03:24.306955 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:03:24.306906    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:27.301392 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:03:27.301362    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qq26k"
Sep 17 18:03:27.302001 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:03:27.301976    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-t4pxj"
Sep 17 18:03:27.302114 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:03:27.301978    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qq26k" podUID="5c25567c-e8a8-46fa-845a-34f9aa333b1f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 18:03:27.302546 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:03:27.302525    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-t4pxj" podUID="abee0c45-b6de-49cb-b16f-28957a9926bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 18:03:30.576441 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:03:30.576406    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:30.675428 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:03:30.675395    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:37.302622 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:03:37.302583    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-t4pxj" podUID="abee0c45-b6de-49cb-b16f-28957a9926bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 18:03:37.303053 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:03:37.302583    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qq26k" podUID="5c25567c-e8a8-46fa-845a-34f9aa333b1f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 18:03:47.301984 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:03:47.301944    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qq26k" podUID="5c25567c-e8a8-46fa-845a-34f9aa333b1f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 18:03:47.302443 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:03:47.302416    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-t4pxj" podUID="abee0c45-b6de-49cb-b16f-28957a9926bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 18:03:57.302725 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:03:57.302683    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-t4pxj" podUID="abee0c45-b6de-49cb-b16f-28957a9926bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 18:03:57.303204 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:03:57.302696    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qq26k" podUID="5c25567c-e8a8-46fa-845a-34f9aa333b1f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 18:03:58.534549 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:03:58.534452    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:07.302074 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:07.302035    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qq26k" podUID="5c25567c-e8a8-46fa-845a-34f9aa333b1f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 18:04:07.302554 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:07.302529    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-t4pxj" podUID="abee0c45-b6de-49cb-b16f-28957a9926bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 18:04:17.301984 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:17.301893    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qq26k" podUID="5c25567c-e8a8-46fa-845a-34f9aa333b1f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 18:04:17.302522 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:17.302479    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-t4pxj" podUID="abee0c45-b6de-49cb-b16f-28957a9926bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 18:04:27.301888 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:27.301843    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qq26k" podUID="5c25567c-e8a8-46fa-845a-34f9aa333b1f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 18:04:27.303070 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:27.303038    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-t4pxj" podUID="abee0c45-b6de-49cb-b16f-28957a9926bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 18:04:28.421291 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:28.421261    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rrf79_beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5/ovn-acl-logging/0.log"
Sep 17 18:04:28.422548 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:28.422529    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rrf79_beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5/ovn-acl-logging/0.log"
Sep 17 18:04:29.562634 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:29.562597    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:34.739896 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:34.739863    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:34.958574 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:34.958537    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:36.104213 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:36.104184    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-t4pxj"
Sep 17 18:04:37.303234 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:37.303204    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qq26k"
Sep 17 18:04:37.463208 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:37.463174    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-t4pxj"]
Sep 17 18:04:37.566903 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:37.566818    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-t4pxj" podUID="abee0c45-b6de-49cb-b16f-28957a9926bf" containerName="kserve-container" containerID="cri-o://410358c980b16f9bd9e032afb3ac8769099e91137a4eff05f99807223d38e6ad" gracePeriod=30
Sep 17 18:04:37.566903 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:37.566854    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-t4pxj" podUID="abee0c45-b6de-49cb-b16f-28957a9926bf" containerName="kube-rbac-proxy" containerID="cri-o://69eb562c4ae3e36b18932d3faefad6e96ed42936326518e8e0e87bc38f17a3bd" gracePeriod=30
Sep 17 18:04:38.324671 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:38.324636    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qq26k"]
Sep 17 18:04:38.325126 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:38.324963    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qq26k" podUID="5c25567c-e8a8-46fa-845a-34f9aa333b1f" containerName="kserve-container" containerID="cri-o://0866e0abb7a24efb22532a62a4647db6fdd5288221da7b514aa563a6dd2d7bcc" gracePeriod=30
Sep 17 18:04:38.325126 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:38.325036    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qq26k" podUID="5c25567c-e8a8-46fa-845a-34f9aa333b1f" containerName="kube-rbac-proxy" containerID="cri-o://43cfc498ed0e39060b34d0bdb8a535053b089170b1c72acbb1aafa9847c0f0a1" gracePeriod=30
Sep 17 18:04:38.380141 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:38.380105    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:38.493261 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:38.493228    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:38.576236 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:38.576153    2585 generic.go:356] "Generic (PLEG): container finished" podID="abee0c45-b6de-49cb-b16f-28957a9926bf" containerID="69eb562c4ae3e36b18932d3faefad6e96ed42936326518e8e0e87bc38f17a3bd" exitCode=2
Sep 17 18:04:38.576236 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:38.576227    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-t4pxj" event={"ID":"abee0c45-b6de-49cb-b16f-28957a9926bf","Type":"ContainerDied","Data":"69eb562c4ae3e36b18932d3faefad6e96ed42936326518e8e0e87bc38f17a3bd"}
Sep 17 18:04:38.578252 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:38.578230    2585 generic.go:356] "Generic (PLEG): container finished" podID="5c25567c-e8a8-46fa-845a-34f9aa333b1f" containerID="43cfc498ed0e39060b34d0bdb8a535053b089170b1c72acbb1aafa9847c0f0a1" exitCode=2
Sep 17 18:04:38.578367 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:38.578258    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qq26k" event={"ID":"5c25567c-e8a8-46fa-845a-34f9aa333b1f","Type":"ContainerDied","Data":"43cfc498ed0e39060b34d0bdb8a535053b089170b1c72acbb1aafa9847c0f0a1"}
Sep 17 18:04:42.296767 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:42.296726    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qq26k" podUID="5c25567c-e8a8-46fa-845a-34f9aa333b1f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Sep 17 18:04:42.297164 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:42.296726    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-t4pxj" podUID="abee0c45-b6de-49cb-b16f-28957a9926bf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Sep 17 18:04:42.510795 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:42.510767    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-t4pxj"
Sep 17 18:04:42.529893 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:42.529870    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxfgr\" (UniqueName: \"kubernetes.io/projected/abee0c45-b6de-49cb-b16f-28957a9926bf-kube-api-access-sxfgr\") pod \"abee0c45-b6de-49cb-b16f-28957a9926bf\" (UID: \"abee0c45-b6de-49cb-b16f-28957a9926bf\") "
Sep 17 18:04:42.530020 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:42.529905    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/abee0c45-b6de-49cb-b16f-28957a9926bf-kube-rbac-proxy-sar-config\") pod \"abee0c45-b6de-49cb-b16f-28957a9926bf\" (UID: \"abee0c45-b6de-49cb-b16f-28957a9926bf\") "
Sep 17 18:04:42.530020 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:42.529950    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abee0c45-b6de-49cb-b16f-28957a9926bf-proxy-tls\") pod \"abee0c45-b6de-49cb-b16f-28957a9926bf\" (UID: \"abee0c45-b6de-49cb-b16f-28957a9926bf\") "
Sep 17 18:04:42.530020 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:42.529985    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/abee0c45-b6de-49cb-b16f-28957a9926bf-cabundle-cert\") pod \"abee0c45-b6de-49cb-b16f-28957a9926bf\" (UID: \"abee0c45-b6de-49cb-b16f-28957a9926bf\") "
Sep 17 18:04:42.530159 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:42.530022    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/abee0c45-b6de-49cb-b16f-28957a9926bf-kserve-provision-location\") pod \"abee0c45-b6de-49cb-b16f-28957a9926bf\" (UID: \"abee0c45-b6de-49cb-b16f-28957a9926bf\") "
Sep 17 18:04:42.530371 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:42.530344    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abee0c45-b6de-49cb-b16f-28957a9926bf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "abee0c45-b6de-49cb-b16f-28957a9926bf" (UID: "abee0c45-b6de-49cb-b16f-28957a9926bf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:04:42.530371 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:42.530343    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abee0c45-b6de-49cb-b16f-28957a9926bf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "abee0c45-b6de-49cb-b16f-28957a9926bf" (UID: "abee0c45-b6de-49cb-b16f-28957a9926bf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:04:42.530556 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:42.530349    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abee0c45-b6de-49cb-b16f-28957a9926bf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "abee0c45-b6de-49cb-b16f-28957a9926bf" (UID: "abee0c45-b6de-49cb-b16f-28957a9926bf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:04:42.531982 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:42.531959    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abee0c45-b6de-49cb-b16f-28957a9926bf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "abee0c45-b6de-49cb-b16f-28957a9926bf" (UID: "abee0c45-b6de-49cb-b16f-28957a9926bf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:04:42.532038 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:42.532007    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abee0c45-b6de-49cb-b16f-28957a9926bf-kube-api-access-sxfgr" (OuterVolumeSpecName: "kube-api-access-sxfgr") pod "abee0c45-b6de-49cb-b16f-28957a9926bf" (UID: "abee0c45-b6de-49cb-b16f-28957a9926bf"). InnerVolumeSpecName "kube-api-access-sxfgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:04:42.594419 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:42.594326    2585 generic.go:356] "Generic (PLEG): container finished" podID="abee0c45-b6de-49cb-b16f-28957a9926bf" containerID="410358c980b16f9bd9e032afb3ac8769099e91137a4eff05f99807223d38e6ad" exitCode=0
Sep 17 18:04:42.594419 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:42.594388    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-t4pxj"
Sep 17 18:04:42.594419 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:42.594407    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-t4pxj" event={"ID":"abee0c45-b6de-49cb-b16f-28957a9926bf","Type":"ContainerDied","Data":"410358c980b16f9bd9e032afb3ac8769099e91137a4eff05f99807223d38e6ad"}
Sep 17 18:04:42.594670 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:42.594470    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-t4pxj" event={"ID":"abee0c45-b6de-49cb-b16f-28957a9926bf","Type":"ContainerDied","Data":"390b46d5c1879ff4cae50d000901064701982fac530cfb0e6c8dd9f262922e11"}
Sep 17 18:04:42.594670 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:42.594516    2585 scope.go:117] "RemoveContainer" containerID="69eb562c4ae3e36b18932d3faefad6e96ed42936326518e8e0e87bc38f17a3bd"
Sep 17 18:04:42.603071 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:42.603051    2585 scope.go:117] "RemoveContainer" containerID="410358c980b16f9bd9e032afb3ac8769099e91137a4eff05f99807223d38e6ad"
Sep 17 18:04:42.610084 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:42.610069    2585 scope.go:117] "RemoveContainer" containerID="93fff7722912ca513270a2b3973817e2c7cf553e04ba95848140cdd753381f76"
Sep 17 18:04:42.617758 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:42.617740    2585 scope.go:117] "RemoveContainer" containerID="69eb562c4ae3e36b18932d3faefad6e96ed42936326518e8e0e87bc38f17a3bd"
Sep 17 18:04:42.618011 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:04:42.617991    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69eb562c4ae3e36b18932d3faefad6e96ed42936326518e8e0e87bc38f17a3bd\": container with ID starting with 69eb562c4ae3e36b18932d3faefad6e96ed42936326518e8e0e87bc38f17a3bd not found: ID does not exist" containerID="69eb562c4ae3e36b18932d3faefad6e96ed42936326518e8e0e87bc38f17a3bd"
Sep 17 18:04:42.618073 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:42.618019    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69eb562c4ae3e36b18932d3faefad6e96ed42936326518e8e0e87bc38f17a3bd"} err="failed to get container status \"69eb562c4ae3e36b18932d3faefad6e96ed42936326518e8e0e87bc38f17a3bd\": rpc error: code = NotFound desc = could not find container \"69eb562c4ae3e36b18932d3faefad6e96ed42936326518e8e0e87bc38f17a3bd\": container with ID starting with 69eb562c4ae3e36b18932d3faefad6e96ed42936326518e8e0e87bc38f17a3bd not found: ID does not exist"
Sep 17 18:04:42.618073 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:42.618036    2585 scope.go:117] "RemoveContainer" containerID="410358c980b16f9bd9e032afb3ac8769099e91137a4eff05f99807223d38e6ad"
Sep 17 18:04:42.618295 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:04:42.618275    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"410358c980b16f9bd9e032afb3ac8769099e91137a4eff05f99807223d38e6ad\": container with ID starting with 410358c980b16f9bd9e032afb3ac8769099e91137a4eff05f99807223d38e6ad not found: ID does not exist" containerID="410358c980b16f9bd9e032afb3ac8769099e91137a4eff05f99807223d38e6ad"
Sep 17 18:04:42.618339 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:42.618305    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"410358c980b16f9bd9e032afb3ac8769099e91137a4eff05f99807223d38e6ad"} err="failed to get container status \"410358c980b16f9bd9e032afb3ac8769099e91137a4eff05f99807223d38e6ad\": rpc error: code = NotFound desc = could not find container \"410358c980b16f9bd9e032afb3ac8769099e91137a4eff05f99807223d38e6ad\": container with ID starting with 410358c980b16f9bd9e032afb3ac8769099e91137a4eff05f99807223d38e6ad not found: ID does not exist"
Sep 17 18:04:42.618339 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:42.618327    2585 scope.go:117] "RemoveContainer" containerID="93fff7722912ca513270a2b3973817e2c7cf553e04ba95848140cdd753381f76"
Sep 17 18:04:42.618596 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:42.618579    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-t4pxj"]
Sep 17 18:04:42.618673 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:04:42.618577    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93fff7722912ca513270a2b3973817e2c7cf553e04ba95848140cdd753381f76\": container with ID starting with 93fff7722912ca513270a2b3973817e2c7cf553e04ba95848140cdd753381f76 not found: ID does not exist" containerID="93fff7722912ca513270a2b3973817e2c7cf553e04ba95848140cdd753381f76"
Sep 17 18:04:42.618673 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:42.618626    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93fff7722912ca513270a2b3973817e2c7cf553e04ba95848140cdd753381f76"} err="failed to get container status \"93fff7722912ca513270a2b3973817e2c7cf553e04ba95848140cdd753381f76\": rpc error: code = NotFound desc = could not find container \"93fff7722912ca513270a2b3973817e2c7cf553e04ba95848140cdd753381f76\": container with ID starting with 93fff7722912ca513270a2b3973817e2c7cf553e04ba95848140cdd753381f76 not found: ID does not exist"
Sep 17 18:04:42.623173 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:42.623152    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6474b875b9-t4pxj"]
Sep 17 18:04:42.631086 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:42.631072    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/abee0c45-b6de-49cb-b16f-28957a9926bf-cabundle-cert\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:04:42.631142 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:42.631089    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/abee0c45-b6de-49cb-b16f-28957a9926bf-kserve-provision-location\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:04:42.631142 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:42.631099    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxfgr\" (UniqueName: \"kubernetes.io/projected/abee0c45-b6de-49cb-b16f-28957a9926bf-kube-api-access-sxfgr\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:04:42.631142 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:42.631108    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/abee0c45-b6de-49cb-b16f-28957a9926bf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:04:42.631142 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:42.631118    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abee0c45-b6de-49cb-b16f-28957a9926bf-proxy-tls\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:04:43.258500 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.258456    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qq26k"
Sep 17 18:04:43.335450 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.335420    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c25567c-e8a8-46fa-845a-34f9aa333b1f-proxy-tls\") pod \"5c25567c-e8a8-46fa-845a-34f9aa333b1f\" (UID: \"5c25567c-e8a8-46fa-845a-34f9aa333b1f\") "
Sep 17 18:04:43.335887 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.335471    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c25567c-e8a8-46fa-845a-34f9aa333b1f-kserve-provision-location\") pod \"5c25567c-e8a8-46fa-845a-34f9aa333b1f\" (UID: \"5c25567c-e8a8-46fa-845a-34f9aa333b1f\") "
Sep 17 18:04:43.335887 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.335520    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c25567c-e8a8-46fa-845a-34f9aa333b1f-kube-rbac-proxy-sar-config\") pod \"5c25567c-e8a8-46fa-845a-34f9aa333b1f\" (UID: \"5c25567c-e8a8-46fa-845a-34f9aa333b1f\") "
Sep 17 18:04:43.335887 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.335537    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c25567c-e8a8-46fa-845a-34f9aa333b1f-cabundle-cert\") pod \"5c25567c-e8a8-46fa-845a-34f9aa333b1f\" (UID: \"5c25567c-e8a8-46fa-845a-34f9aa333b1f\") "
Sep 17 18:04:43.335887 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.335592    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wq8h7\" (UniqueName: \"kubernetes.io/projected/5c25567c-e8a8-46fa-845a-34f9aa333b1f-kube-api-access-wq8h7\") pod \"5c25567c-e8a8-46fa-845a-34f9aa333b1f\" (UID: \"5c25567c-e8a8-46fa-845a-34f9aa333b1f\") "
Sep 17 18:04:43.335887 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.335871    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c25567c-e8a8-46fa-845a-34f9aa333b1f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5c25567c-e8a8-46fa-845a-34f9aa333b1f" (UID: "5c25567c-e8a8-46fa-845a-34f9aa333b1f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:04:43.336154 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.335919    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c25567c-e8a8-46fa-845a-34f9aa333b1f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5c25567c-e8a8-46fa-845a-34f9aa333b1f" (UID: "5c25567c-e8a8-46fa-845a-34f9aa333b1f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:04:43.336154 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.335966    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c25567c-e8a8-46fa-845a-34f9aa333b1f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5c25567c-e8a8-46fa-845a-34f9aa333b1f" (UID: "5c25567c-e8a8-46fa-845a-34f9aa333b1f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:04:43.338202 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.338178    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c25567c-e8a8-46fa-845a-34f9aa333b1f-kube-api-access-wq8h7" (OuterVolumeSpecName: "kube-api-access-wq8h7") pod "5c25567c-e8a8-46fa-845a-34f9aa333b1f" (UID: "5c25567c-e8a8-46fa-845a-34f9aa333b1f"). InnerVolumeSpecName "kube-api-access-wq8h7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:04:43.338324 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.338222    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c25567c-e8a8-46fa-845a-34f9aa333b1f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5c25567c-e8a8-46fa-845a-34f9aa333b1f" (UID: "5c25567c-e8a8-46fa-845a-34f9aa333b1f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:04:43.407486 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.407409    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:43.430198 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.430166    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6"]
Sep 17 18:04:43.430414 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.430403    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abee0c45-b6de-49cb-b16f-28957a9926bf" containerName="kube-rbac-proxy"
Sep 17 18:04:43.430455 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.430416    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="abee0c45-b6de-49cb-b16f-28957a9926bf" containerName="kube-rbac-proxy"
Sep 17 18:04:43.430455 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.430426    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c25567c-e8a8-46fa-845a-34f9aa333b1f" containerName="kube-rbac-proxy"
Sep 17 18:04:43.430455 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.430433    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c25567c-e8a8-46fa-845a-34f9aa333b1f" containerName="kube-rbac-proxy"
Sep 17 18:04:43.430455 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.430441    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c25567c-e8a8-46fa-845a-34f9aa333b1f" containerName="storage-initializer"
Sep 17 18:04:43.430455 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.430450    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c25567c-e8a8-46fa-845a-34f9aa333b1f" containerName="storage-initializer"
Sep 17 18:04:43.430632 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.430460    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abee0c45-b6de-49cb-b16f-28957a9926bf" containerName="storage-initializer"
Sep 17 18:04:43.430632 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.430468    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="abee0c45-b6de-49cb-b16f-28957a9926bf" containerName="storage-initializer"
Sep 17 18:04:43.430632 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.430477    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c25567c-e8a8-46fa-845a-34f9aa333b1f" containerName="kserve-container"
Sep 17 18:04:43.430632 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.430485    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c25567c-e8a8-46fa-845a-34f9aa333b1f" containerName="kserve-container"
Sep 17 18:04:43.430632 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.430510    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abee0c45-b6de-49cb-b16f-28957a9926bf" containerName="kserve-container"
Sep 17 18:04:43.430632 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.430519    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="abee0c45-b6de-49cb-b16f-28957a9926bf" containerName="kserve-container"
Sep 17 18:04:43.430632 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.430577    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="abee0c45-b6de-49cb-b16f-28957a9926bf" containerName="kube-rbac-proxy"
Sep 17 18:04:43.430632 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.430586    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c25567c-e8a8-46fa-845a-34f9aa333b1f" containerName="kserve-container"
Sep 17 18:04:43.430632 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.430592    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="abee0c45-b6de-49cb-b16f-28957a9926bf" containerName="kserve-container"
Sep 17 18:04:43.430632 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.430598    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c25567c-e8a8-46fa-845a-34f9aa333b1f" containerName="kube-rbac-proxy"
Sep 17 18:04:43.433697 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.433679    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6"
Sep 17 18:04:43.436271 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.436251    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 17 18:04:43.436390 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.436289    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 17 18:04:43.436446 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.436387    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c25567c-e8a8-46fa-845a-34f9aa333b1f-kserve-provision-location\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:04:43.436446 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.436404    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c25567c-e8a8-46fa-845a-34f9aa333b1f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:04:43.436446 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.436419    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c25567c-e8a8-46fa-845a-34f9aa333b1f-cabundle-cert\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:04:43.436446 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.436429    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wq8h7\" (UniqueName: \"kubernetes.io/projected/5c25567c-e8a8-46fa-845a-34f9aa333b1f-kube-api-access-wq8h7\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:04:43.436446 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.436437    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c25567c-e8a8-46fa-845a-34f9aa333b1f-proxy-tls\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:04:43.445888 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.445868    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6"]
Sep 17 18:04:43.469425 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.469395    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jzwv9"]
Sep 17 18:04:43.472521 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.472485    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jzwv9"
Sep 17 18:04:43.475045 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.475026    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 17 18:04:43.489018 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.488985    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jzwv9"]
Sep 17 18:04:43.503541 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.503512    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:43.537284 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.537244    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c655034f-3e59-40d4-8a90-3aa731dd4f26-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6\" (UID: \"c655034f-3e59-40d4-8a90-3aa731dd4f26\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6"
Sep 17 18:04:43.537420 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.537304    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d6fde28-25c1-4489-8ec2-1158d1efe4e4-kserve-provision-location\") pod \"isvc-canary-promote-predictor-7cf78f884f-jzwv9\" (UID: \"6d6fde28-25c1-4489-8ec2-1158d1efe4e4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jzwv9"
Sep 17 18:04:43.537420 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.537336    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d6fde28-25c1-4489-8ec2-1158d1efe4e4-cabundle-cert\") pod \"isvc-canary-promote-predictor-7cf78f884f-jzwv9\" (UID: \"6d6fde28-25c1-4489-8ec2-1158d1efe4e4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jzwv9"
Sep 17 18:04:43.537420 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.537358    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctf66\" (UniqueName: \"kubernetes.io/projected/c655034f-3e59-40d4-8a90-3aa731dd4f26-kube-api-access-ctf66\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6\" (UID: \"c655034f-3e59-40d4-8a90-3aa731dd4f26\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6"
Sep 17 18:04:43.537420 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.537396    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c655034f-3e59-40d4-8a90-3aa731dd4f26-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6\" (UID: \"c655034f-3e59-40d4-8a90-3aa731dd4f26\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6"
Sep 17 18:04:43.537626 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.537428    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d6fde28-25c1-4489-8ec2-1158d1efe4e4-proxy-tls\") pod \"isvc-canary-promote-predictor-7cf78f884f-jzwv9\" (UID: \"6d6fde28-25c1-4489-8ec2-1158d1efe4e4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jzwv9"
Sep 17 18:04:43.537626 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.537459    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqpjc\" (UniqueName: \"kubernetes.io/projected/6d6fde28-25c1-4489-8ec2-1158d1efe4e4-kube-api-access-rqpjc\") pod \"isvc-canary-promote-predictor-7cf78f884f-jzwv9\" (UID: \"6d6fde28-25c1-4489-8ec2-1158d1efe4e4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jzwv9"
Sep 17 18:04:43.537626 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.537543    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c655034f-3e59-40d4-8a90-3aa731dd4f26-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6\" (UID: \"c655034f-3e59-40d4-8a90-3aa731dd4f26\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6"
Sep 17 18:04:43.537626 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.537575    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d6fde28-25c1-4489-8ec2-1158d1efe4e4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-7cf78f884f-jzwv9\" (UID: \"6d6fde28-25c1-4489-8ec2-1158d1efe4e4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jzwv9"
Sep 17 18:04:43.537626 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.537604    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c655034f-3e59-40d4-8a90-3aa731dd4f26-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6\" (UID: \"c655034f-3e59-40d4-8a90-3aa731dd4f26\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6"
Sep 17 18:04:43.600464 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.600427    2585 generic.go:356] "Generic (PLEG): container finished" podID="5c25567c-e8a8-46fa-845a-34f9aa333b1f" containerID="0866e0abb7a24efb22532a62a4647db6fdd5288221da7b514aa563a6dd2d7bcc" exitCode=0
Sep 17 18:04:43.600646 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.600539    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qq26k"
Sep 17 18:04:43.600646 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.600536    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qq26k" event={"ID":"5c25567c-e8a8-46fa-845a-34f9aa333b1f","Type":"ContainerDied","Data":"0866e0abb7a24efb22532a62a4647db6fdd5288221da7b514aa563a6dd2d7bcc"}
Sep 17 18:04:43.600646 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.600612    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qq26k" event={"ID":"5c25567c-e8a8-46fa-845a-34f9aa333b1f","Type":"ContainerDied","Data":"d9dae7dd04e5b4236b2d577ea3281f30eff5eeef043ce43ce5d5b8a478c1e3c2"}
Sep 17 18:04:43.600803 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.600648    2585 scope.go:117] "RemoveContainer" containerID="43cfc498ed0e39060b34d0bdb8a535053b089170b1c72acbb1aafa9847c0f0a1"
Sep 17 18:04:43.608745 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.608638    2585 scope.go:117] "RemoveContainer" containerID="0866e0abb7a24efb22532a62a4647db6fdd5288221da7b514aa563a6dd2d7bcc"
Sep 17 18:04:43.615924 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.615908    2585 scope.go:117] "RemoveContainer" containerID="4c92e70fc5e7a2514e233b251464697390125b72b9f50d9aa27640410ede660e"
Sep 17 18:04:43.622773 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.622753    2585 scope.go:117] "RemoveContainer" containerID="43cfc498ed0e39060b34d0bdb8a535053b089170b1c72acbb1aafa9847c0f0a1"
Sep 17 18:04:43.623015 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:04:43.622994    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43cfc498ed0e39060b34d0bdb8a535053b089170b1c72acbb1aafa9847c0f0a1\": container with ID starting with 43cfc498ed0e39060b34d0bdb8a535053b089170b1c72acbb1aafa9847c0f0a1 not found: ID does not exist" containerID="43cfc498ed0e39060b34d0bdb8a535053b089170b1c72acbb1aafa9847c0f0a1"
Sep 17 18:04:43.623075 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.623023    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43cfc498ed0e39060b34d0bdb8a535053b089170b1c72acbb1aafa9847c0f0a1"} err="failed to get container status \"43cfc498ed0e39060b34d0bdb8a535053b089170b1c72acbb1aafa9847c0f0a1\": rpc error: code = NotFound desc = could not find container \"43cfc498ed0e39060b34d0bdb8a535053b089170b1c72acbb1aafa9847c0f0a1\": container with ID starting with 43cfc498ed0e39060b34d0bdb8a535053b089170b1c72acbb1aafa9847c0f0a1 not found: ID does not exist"
Sep 17 18:04:43.623075 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.623040    2585 scope.go:117] "RemoveContainer" containerID="0866e0abb7a24efb22532a62a4647db6fdd5288221da7b514aa563a6dd2d7bcc"
Sep 17 18:04:43.623288 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:04:43.623270    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0866e0abb7a24efb22532a62a4647db6fdd5288221da7b514aa563a6dd2d7bcc\": container with ID starting with 0866e0abb7a24efb22532a62a4647db6fdd5288221da7b514aa563a6dd2d7bcc not found: ID does not exist" containerID="0866e0abb7a24efb22532a62a4647db6fdd5288221da7b514aa563a6dd2d7bcc"
Sep 17 18:04:43.623369 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.623303    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0866e0abb7a24efb22532a62a4647db6fdd5288221da7b514aa563a6dd2d7bcc"} err="failed to get container status \"0866e0abb7a24efb22532a62a4647db6fdd5288221da7b514aa563a6dd2d7bcc\": rpc error: code = NotFound desc = could not find container \"0866e0abb7a24efb22532a62a4647db6fdd5288221da7b514aa563a6dd2d7bcc\": container with ID starting with 0866e0abb7a24efb22532a62a4647db6fdd5288221da7b514aa563a6dd2d7bcc not found: ID does not exist"
Sep 17 18:04:43.623369 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.623326    2585 scope.go:117] "RemoveContainer" containerID="4c92e70fc5e7a2514e233b251464697390125b72b9f50d9aa27640410ede660e"
Sep 17 18:04:43.623666 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:04:43.623649    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c92e70fc5e7a2514e233b251464697390125b72b9f50d9aa27640410ede660e\": container with ID starting with 4c92e70fc5e7a2514e233b251464697390125b72b9f50d9aa27640410ede660e not found: ID does not exist" containerID="4c92e70fc5e7a2514e233b251464697390125b72b9f50d9aa27640410ede660e"
Sep 17 18:04:43.623712 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.623671    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c92e70fc5e7a2514e233b251464697390125b72b9f50d9aa27640410ede660e"} err="failed to get container status \"4c92e70fc5e7a2514e233b251464697390125b72b9f50d9aa27640410ede660e\": rpc error: code = NotFound desc = could not find container \"4c92e70fc5e7a2514e233b251464697390125b72b9f50d9aa27640410ede660e\": container with ID starting with 4c92e70fc5e7a2514e233b251464697390125b72b9f50d9aa27640410ede660e not found: ID does not exist"
Sep 17 18:04:43.627942 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.627921    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qq26k"]
Sep 17 18:04:43.630071 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.630041    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:43.633558 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.633537    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-96f77bdbd-qq26k"]
Sep 17 18:04:43.638234 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.638216    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c655034f-3e59-40d4-8a90-3aa731dd4f26-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6\" (UID: \"c655034f-3e59-40d4-8a90-3aa731dd4f26\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6"
Sep 17 18:04:43.638298 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.638253    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d6fde28-25c1-4489-8ec2-1158d1efe4e4-kserve-provision-location\") pod \"isvc-canary-promote-predictor-7cf78f884f-jzwv9\" (UID: \"6d6fde28-25c1-4489-8ec2-1158d1efe4e4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jzwv9"
Sep 17 18:04:43.638298 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.638275    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d6fde28-25c1-4489-8ec2-1158d1efe4e4-cabundle-cert\") pod \"isvc-canary-promote-predictor-7cf78f884f-jzwv9\" (UID: \"6d6fde28-25c1-4489-8ec2-1158d1efe4e4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jzwv9"
Sep 17 18:04:43.638298 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.638289    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctf66\" (UniqueName: \"kubernetes.io/projected/c655034f-3e59-40d4-8a90-3aa731dd4f26-kube-api-access-ctf66\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6\" (UID: \"c655034f-3e59-40d4-8a90-3aa731dd4f26\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6"
Sep 17 18:04:43.638454 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.638321    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c655034f-3e59-40d4-8a90-3aa731dd4f26-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6\" (UID: \"c655034f-3e59-40d4-8a90-3aa731dd4f26\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6"
Sep 17 18:04:43.638454 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.638352    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d6fde28-25c1-4489-8ec2-1158d1efe4e4-proxy-tls\") pod \"isvc-canary-promote-predictor-7cf78f884f-jzwv9\" (UID: \"6d6fde28-25c1-4489-8ec2-1158d1efe4e4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jzwv9"
Sep 17 18:04:43.638454 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.638385    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqpjc\" (UniqueName: \"kubernetes.io/projected/6d6fde28-25c1-4489-8ec2-1158d1efe4e4-kube-api-access-rqpjc\") pod \"isvc-canary-promote-predictor-7cf78f884f-jzwv9\" (UID: \"6d6fde28-25c1-4489-8ec2-1158d1efe4e4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jzwv9"
Sep 17 18:04:43.638454 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.638418    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c655034f-3e59-40d4-8a90-3aa731dd4f26-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6\" (UID: \"c655034f-3e59-40d4-8a90-3aa731dd4f26\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6"
Sep 17 18:04:43.638454 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.638443    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d6fde28-25c1-4489-8ec2-1158d1efe4e4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-7cf78f884f-jzwv9\" (UID: \"6d6fde28-25c1-4489-8ec2-1158d1efe4e4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jzwv9"
Sep 17 18:04:43.638741 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.638481    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c655034f-3e59-40d4-8a90-3aa731dd4f26-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6\" (UID: \"c655034f-3e59-40d4-8a90-3aa731dd4f26\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6"
Sep 17 18:04:43.638741 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:04:43.638547    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 17 18:04:43.638741 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:04:43.638625    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d6fde28-25c1-4489-8ec2-1158d1efe4e4-proxy-tls podName:6d6fde28-25c1-4489-8ec2-1158d1efe4e4 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:44.138605222 +0000 UTC m=+1820.552395651 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6d6fde28-25c1-4489-8ec2-1158d1efe4e4-proxy-tls") pod "isvc-canary-promote-predictor-7cf78f884f-jzwv9" (UID: "6d6fde28-25c1-4489-8ec2-1158d1efe4e4") : secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 17 18:04:43.638741 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.638693    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d6fde28-25c1-4489-8ec2-1158d1efe4e4-kserve-provision-location\") pod \"isvc-canary-promote-predictor-7cf78f884f-jzwv9\" (UID: \"6d6fde28-25c1-4489-8ec2-1158d1efe4e4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jzwv9"
Sep 17 18:04:43.638949 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.638741    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c655034f-3e59-40d4-8a90-3aa731dd4f26-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6\" (UID: \"c655034f-3e59-40d4-8a90-3aa731dd4f26\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6"
Sep 17 18:04:43.639008 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.638993    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c655034f-3e59-40d4-8a90-3aa731dd4f26-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6\" (UID: \"c655034f-3e59-40d4-8a90-3aa731dd4f26\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6"
Sep 17 18:04:43.639079 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.639058    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d6fde28-25c1-4489-8ec2-1158d1efe4e4-cabundle-cert\") pod \"isvc-canary-promote-predictor-7cf78f884f-jzwv9\" (UID: \"6d6fde28-25c1-4489-8ec2-1158d1efe4e4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jzwv9"
Sep 17 18:04:43.639161 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.639142    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c655034f-3e59-40d4-8a90-3aa731dd4f26-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6\" (UID: \"c655034f-3e59-40d4-8a90-3aa731dd4f26\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6"
Sep 17 18:04:43.639255 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.639240    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d6fde28-25c1-4489-8ec2-1158d1efe4e4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-7cf78f884f-jzwv9\" (UID: \"6d6fde28-25c1-4489-8ec2-1158d1efe4e4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jzwv9"
Sep 17 18:04:43.640902 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.640887    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c655034f-3e59-40d4-8a90-3aa731dd4f26-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6\" (UID: \"c655034f-3e59-40d4-8a90-3aa731dd4f26\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6"
Sep 17 18:04:43.650061 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.650034    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctf66\" (UniqueName: \"kubernetes.io/projected/c655034f-3e59-40d4-8a90-3aa731dd4f26-kube-api-access-ctf66\") pod \"isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6\" (UID: \"c655034f-3e59-40d4-8a90-3aa731dd4f26\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6"
Sep 17 18:04:43.651698 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.651675    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqpjc\" (UniqueName: \"kubernetes.io/projected/6d6fde28-25c1-4489-8ec2-1158d1efe4e4-kube-api-access-rqpjc\") pod \"isvc-canary-promote-predictor-7cf78f884f-jzwv9\" (UID: \"6d6fde28-25c1-4489-8ec2-1158d1efe4e4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jzwv9"
Sep 17 18:04:43.744664 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.744631    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6"
Sep 17 18:04:43.883311 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:43.883273    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:44.024977 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:44.024904    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6"]
Sep 17 18:04:44.027931 ip-10-0-133-139 kubenswrapper[2585]: W0917 18:04:44.027907    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc655034f_3e59_40d4_8a90_3aa731dd4f26.slice/crio-a9ec6e54b7aa9eb22f9b3959b40bae788751f05c1af7e1bdd2bb7383c9ecd47d WatchSource:0}: Error finding container a9ec6e54b7aa9eb22f9b3959b40bae788751f05c1af7e1bdd2bb7383c9ecd47d: Status 404 returned error can't find the container with id a9ec6e54b7aa9eb22f9b3959b40bae788751f05c1af7e1bdd2bb7383c9ecd47d
Sep 17 18:04:44.103316 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:44.103274    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c25567c-e8a8-46fa-845a-34f9aa333b1f" path="/var/lib/kubelet/pods/5c25567c-e8a8-46fa-845a-34f9aa333b1f/volumes"
Sep 17 18:04:44.103850 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:44.103831    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abee0c45-b6de-49cb-b16f-28957a9926bf" path="/var/lib/kubelet/pods/abee0c45-b6de-49cb-b16f-28957a9926bf/volumes"
Sep 17 18:04:44.141967 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:44.141934    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d6fde28-25c1-4489-8ec2-1158d1efe4e4-proxy-tls\") pod \"isvc-canary-promote-predictor-7cf78f884f-jzwv9\" (UID: \"6d6fde28-25c1-4489-8ec2-1158d1efe4e4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jzwv9"
Sep 17 18:04:44.144253 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:44.144231    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d6fde28-25c1-4489-8ec2-1158d1efe4e4-proxy-tls\") pod \"isvc-canary-promote-predictor-7cf78f884f-jzwv9\" (UID: \"6d6fde28-25c1-4489-8ec2-1158d1efe4e4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jzwv9"
Sep 17 18:04:44.382280 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:44.382190    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jzwv9"
Sep 17 18:04:44.607944 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:44.607908    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6" event={"ID":"c655034f-3e59-40d4-8a90-3aa731dd4f26","Type":"ContainerStarted","Data":"f09b80ad54abaf0e971bf91303775b41129e89c76205265fc5f2a0b4d7a7719e"}
Sep 17 18:04:44.608117 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:44.607976    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6" event={"ID":"c655034f-3e59-40d4-8a90-3aa731dd4f26","Type":"ContainerStarted","Data":"a9ec6e54b7aa9eb22f9b3959b40bae788751f05c1af7e1bdd2bb7383c9ecd47d"}
Sep 17 18:04:44.665371 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:44.665349    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jzwv9"]
Sep 17 18:04:44.667943 ip-10-0-133-139 kubenswrapper[2585]: W0917 18:04:44.667911    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d6fde28_25c1_4489_8ec2_1158d1efe4e4.slice/crio-b99fa62e30522c634da12100a518eb8c35fb3ee963830c31bd7abd5ad2c94114 WatchSource:0}: Error finding container b99fa62e30522c634da12100a518eb8c35fb3ee963830c31bd7abd5ad2c94114: Status 404 returned error can't find the container with id b99fa62e30522c634da12100a518eb8c35fb3ee963830c31bd7abd5ad2c94114
Sep 17 18:04:45.613311 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:45.613271    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jzwv9" event={"ID":"6d6fde28-25c1-4489-8ec2-1158d1efe4e4","Type":"ContainerStarted","Data":"7579236f0ca30b022e6e57fadc7cca3c90fb43a2c313fdc1b3b0da188b0b5135"}
Sep 17 18:04:45.613693 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:45.613336    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jzwv9" event={"ID":"6d6fde28-25c1-4489-8ec2-1158d1efe4e4","Type":"ContainerStarted","Data":"b99fa62e30522c634da12100a518eb8c35fb3ee963830c31bd7abd5ad2c94114"}
Sep 17 18:04:48.625842 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:48.625808    2585 generic.go:356] "Generic (PLEG): container finished" podID="6d6fde28-25c1-4489-8ec2-1158d1efe4e4" containerID="7579236f0ca30b022e6e57fadc7cca3c90fb43a2c313fdc1b3b0da188b0b5135" exitCode=0
Sep 17 18:04:48.626272 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:48.625880    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jzwv9" event={"ID":"6d6fde28-25c1-4489-8ec2-1158d1efe4e4","Type":"ContainerDied","Data":"7579236f0ca30b022e6e57fadc7cca3c90fb43a2c313fdc1b3b0da188b0b5135"}
Sep 17 18:04:48.627635 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:48.627612    2585 generic.go:356] "Generic (PLEG): container finished" podID="c655034f-3e59-40d4-8a90-3aa731dd4f26" containerID="f09b80ad54abaf0e971bf91303775b41129e89c76205265fc5f2a0b4d7a7719e" exitCode=0
Sep 17 18:04:48.627779 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:48.627688    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6" event={"ID":"c655034f-3e59-40d4-8a90-3aa731dd4f26","Type":"ContainerDied","Data":"f09b80ad54abaf0e971bf91303775b41129e89c76205265fc5f2a0b4d7a7719e"}
Sep 17 18:04:49.633459 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:49.633423    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6" event={"ID":"c655034f-3e59-40d4-8a90-3aa731dd4f26","Type":"ContainerStarted","Data":"444f8e92d793c44c32b3fb58a60ec4aa791864b20f251188e5badb062e3bbcf0"}
Sep 17 18:04:49.633895 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:49.633485    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6" event={"ID":"c655034f-3e59-40d4-8a90-3aa731dd4f26","Type":"ContainerStarted","Data":"432061c7980078b212864beebd65df318e0bf7f8cd8fc850158cb31500b51f1a"}
Sep 17 18:04:49.633895 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:49.633804    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6"
Sep 17 18:04:49.634012 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:49.633978    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6"
Sep 17 18:04:49.635589 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:49.635563    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6" podUID="c655034f-3e59-40d4-8a90-3aa731dd4f26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 18:04:49.635700 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:49.635672    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jzwv9" event={"ID":"6d6fde28-25c1-4489-8ec2-1158d1efe4e4","Type":"ContainerStarted","Data":"523e60b0b0bd27459c54f52d21bc84a48e796c09605329f1b421abc39d7ce5f0"}
Sep 17 18:04:49.635751 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:49.635714    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jzwv9" event={"ID":"6d6fde28-25c1-4489-8ec2-1158d1efe4e4","Type":"ContainerStarted","Data":"9a4a73b42a376333513a1534ae2f75e1aa7e37d8c0784b00f129b40136350caf"}
Sep 17 18:04:49.635902 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:49.635889    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jzwv9"
Sep 17 18:04:49.635939 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:49.635925    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jzwv9"
Sep 17 18:04:49.637018 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:49.636996    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jzwv9" podUID="6d6fde28-25c1-4489-8ec2-1158d1efe4e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 18:04:49.662795 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:49.662748    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6" podStartSLOduration=2.999153702 podStartE2EDuration="6.66273606s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.663582358s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:04:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:04:49.660988422 +0000 UTC m=+1826.074778863" watchObservedRunningTime="2026-09-17 18:04:49.66273606 +0000 UTC m=+1826.076526498"
Sep 17 18:04:49.684256 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:49.684212    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jzwv9" podStartSLOduration=3.030238173 podStartE2EDuration="6.684200435s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.653962262s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:04:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:04:49.682126309 +0000 UTC m=+1826.095916748" watchObservedRunningTime="2026-09-17 18:04:49.684200435 +0000 UTC m=+1826.097990877"
Sep 17 18:04:50.640010 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:50.639969    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jzwv9" podUID="6d6fde28-25c1-4489-8ec2-1158d1efe4e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 18:04:50.640362 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:50.640080    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6" podUID="c655034f-3e59-40d4-8a90-3aa731dd4f26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 18:04:55.644879 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:55.644846    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6"
Sep 17 18:04:55.645385 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:55.645364    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jzwv9"
Sep 17 18:04:55.645431 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:55.645383    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6" podUID="c655034f-3e59-40d4-8a90-3aa731dd4f26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 18:04:55.645890 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:04:55.645866    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jzwv9" podUID="6d6fde28-25c1-4489-8ec2-1158d1efe4e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 18:05:05.645985 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:05:05.645936    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jzwv9" podUID="6d6fde28-25c1-4489-8ec2-1158d1efe4e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 18:05:05.646577 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:05:05.645936    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6" podUID="c655034f-3e59-40d4-8a90-3aa731dd4f26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 18:05:15.645529 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:05:15.645476    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6" podUID="c655034f-3e59-40d4-8a90-3aa731dd4f26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 18:05:15.647864 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:05:15.645742    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jzwv9" podUID="6d6fde28-25c1-4489-8ec2-1158d1efe4e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 18:05:25.646301 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:05:25.646263    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6" podUID="c655034f-3e59-40d4-8a90-3aa731dd4f26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 18:05:25.646692 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:05:25.646262    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jzwv9" podUID="6d6fde28-25c1-4489-8ec2-1158d1efe4e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 18:05:35.645910 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:05:35.645873    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jzwv9" podUID="6d6fde28-25c1-4489-8ec2-1158d1efe4e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 18:05:35.646273 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:05:35.645885    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6" podUID="c655034f-3e59-40d4-8a90-3aa731dd4f26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 18:05:45.645565 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:05:45.645522    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6" podUID="c655034f-3e59-40d4-8a90-3aa731dd4f26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 18:05:45.646033 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:05:45.645815    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jzwv9" podUID="6d6fde28-25c1-4489-8ec2-1158d1efe4e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 18:05:55.646095 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:05:55.646012    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jzwv9" podUID="6d6fde28-25c1-4489-8ec2-1158d1efe4e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 18:05:55.646458 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:05:55.646012    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6" podUID="c655034f-3e59-40d4-8a90-3aa731dd4f26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 18:06:01.099898 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:06:01.099848    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6" podUID="c655034f-3e59-40d4-8a90-3aa731dd4f26" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 18:06:05.037777 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:06:05.037741    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:05.647265 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:06:05.647239    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jzwv9"
Sep 17 18:06:10.076357 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:06:10.076318    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:10.463483 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:06:10.463409    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:11.100699 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:06:11.100671    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6"
Sep 17 18:06:13.512605 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:06:13.512572    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jzwv9"]
Sep 17 18:06:13.513049 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:06:13.512862    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jzwv9" podUID="6d6fde28-25c1-4489-8ec2-1158d1efe4e4" containerName="kserve-container" containerID="cri-o://9a4a73b42a376333513a1534ae2f75e1aa7e37d8c0784b00f129b40136350caf" gracePeriod=30
Sep 17 18:06:13.513049 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:06:13.512915    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jzwv9" podUID="6d6fde28-25c1-4489-8ec2-1158d1efe4e4" containerName="kube-rbac-proxy" containerID="cri-o://523e60b0b0bd27459c54f52d21bc84a48e796c09605329f1b421abc39d7ce5f0" gracePeriod=30
Sep 17 18:06:13.546645 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:06:13.546616    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:13.950237 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:06:13.950159    2585 generic.go:356] "Generic (PLEG): container finished" podID="6d6fde28-25c1-4489-8ec2-1158d1efe4e4" containerID="523e60b0b0bd27459c54f52d21bc84a48e796c09605329f1b421abc39d7ce5f0" exitCode=2
Sep 17 18:06:13.950237 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:06:13.950195    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jzwv9" event={"ID":"6d6fde28-25c1-4489-8ec2-1158d1efe4e4","Type":"ContainerDied","Data":"523e60b0b0bd27459c54f52d21bc84a48e796c09605329f1b421abc39d7ce5f0"}
Sep 17 18:06:15.640950 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:06:15.640899    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jzwv9" podUID="6d6fde28-25c1-4489-8ec2-1158d1efe4e4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Sep 17 18:06:15.646246 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:06:15.646217    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jzwv9" podUID="6d6fde28-25c1-4489-8ec2-1158d1efe4e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Sep 17 18:06:18.151930 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:06:18.151902    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jzwv9"
Sep 17 18:06:18.223721 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:06:18.223691    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d6fde28-25c1-4489-8ec2-1158d1efe4e4-cabundle-cert\") pod \"6d6fde28-25c1-4489-8ec2-1158d1efe4e4\" (UID: \"6d6fde28-25c1-4489-8ec2-1158d1efe4e4\") "
Sep 17 18:06:18.223859 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:06:18.223758    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d6fde28-25c1-4489-8ec2-1158d1efe4e4-kserve-provision-location\") pod \"6d6fde28-25c1-4489-8ec2-1158d1efe4e4\" (UID: \"6d6fde28-25c1-4489-8ec2-1158d1efe4e4\") "
Sep 17 18:06:18.223859 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:06:18.223778    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d6fde28-25c1-4489-8ec2-1158d1efe4e4-proxy-tls\") pod \"6d6fde28-25c1-4489-8ec2-1158d1efe4e4\" (UID: \"6d6fde28-25c1-4489-8ec2-1158d1efe4e4\") "
Sep 17 18:06:18.223859 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:06:18.223800    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d6fde28-25c1-4489-8ec2-1158d1efe4e4-kube-rbac-proxy-sar-config\") pod \"6d6fde28-25c1-4489-8ec2-1158d1efe4e4\" (UID: \"6d6fde28-25c1-4489-8ec2-1158d1efe4e4\") "
Sep 17 18:06:18.223859 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:06:18.223828    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqpjc\" (UniqueName: \"kubernetes.io/projected/6d6fde28-25c1-4489-8ec2-1158d1efe4e4-kube-api-access-rqpjc\") pod \"6d6fde28-25c1-4489-8ec2-1158d1efe4e4\" (UID: \"6d6fde28-25c1-4489-8ec2-1158d1efe4e4\") "
Sep 17 18:06:18.224109 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:06:18.224080    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d6fde28-25c1-4489-8ec2-1158d1efe4e4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6d6fde28-25c1-4489-8ec2-1158d1efe4e4" (UID: "6d6fde28-25c1-4489-8ec2-1158d1efe4e4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:06:18.224186 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:06:18.224110    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d6fde28-25c1-4489-8ec2-1158d1efe4e4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6d6fde28-25c1-4489-8ec2-1158d1efe4e4" (UID: "6d6fde28-25c1-4489-8ec2-1158d1efe4e4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:06:18.224186 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:06:18.224151    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d6fde28-25c1-4489-8ec2-1158d1efe4e4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6d6fde28-25c1-4489-8ec2-1158d1efe4e4" (UID: "6d6fde28-25c1-4489-8ec2-1158d1efe4e4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:06:18.225874 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:06:18.225840    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d6fde28-25c1-4489-8ec2-1158d1efe4e4-kube-api-access-rqpjc" (OuterVolumeSpecName: "kube-api-access-rqpjc") pod "6d6fde28-25c1-4489-8ec2-1158d1efe4e4" (UID: "6d6fde28-25c1-4489-8ec2-1158d1efe4e4"). InnerVolumeSpecName "kube-api-access-rqpjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:06:18.225974 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:06:18.225895    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d6fde28-25c1-4489-8ec2-1158d1efe4e4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6d6fde28-25c1-4489-8ec2-1158d1efe4e4" (UID: "6d6fde28-25c1-4489-8ec2-1158d1efe4e4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:06:18.325056 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:06:18.324994    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6d6fde28-25c1-4489-8ec2-1158d1efe4e4-kserve-provision-location\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:06:18.325056 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:06:18.325015    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d6fde28-25c1-4489-8ec2-1158d1efe4e4-proxy-tls\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:06:18.325056 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:06:18.325024    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d6fde28-25c1-4489-8ec2-1158d1efe4e4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:06:18.325056 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:06:18.325033    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqpjc\" (UniqueName: \"kubernetes.io/projected/6d6fde28-25c1-4489-8ec2-1158d1efe4e4-kube-api-access-rqpjc\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:06:18.325056 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:06:18.325043    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6d6fde28-25c1-4489-8ec2-1158d1efe4e4-cabundle-cert\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:06:18.969608 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:06:18.969573    2585 generic.go:356] "Generic (PLEG): container finished" podID="6d6fde28-25c1-4489-8ec2-1158d1efe4e4" containerID="9a4a73b42a376333513a1534ae2f75e1aa7e37d8c0784b00f129b40136350caf" exitCode=0
Sep 17 18:06:18.969772 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:06:18.969627    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jzwv9"
Sep 17 18:06:18.969772 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:06:18.969654    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jzwv9" event={"ID":"6d6fde28-25c1-4489-8ec2-1158d1efe4e4","Type":"ContainerDied","Data":"9a4a73b42a376333513a1534ae2f75e1aa7e37d8c0784b00f129b40136350caf"}
Sep 17 18:06:18.969772 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:06:18.969710    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jzwv9" event={"ID":"6d6fde28-25c1-4489-8ec2-1158d1efe4e4","Type":"ContainerDied","Data":"b99fa62e30522c634da12100a518eb8c35fb3ee963830c31bd7abd5ad2c94114"}
Sep 17 18:06:18.969772 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:06:18.969740    2585 scope.go:117] "RemoveContainer" containerID="523e60b0b0bd27459c54f52d21bc84a48e796c09605329f1b421abc39d7ce5f0"
Sep 17 18:06:18.979579 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:06:18.979533    2585 scope.go:117] "RemoveContainer" containerID="9a4a73b42a376333513a1534ae2f75e1aa7e37d8c0784b00f129b40136350caf"
Sep 17 18:06:18.986630 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:06:18.986603    2585 scope.go:117] "RemoveContainer" containerID="7579236f0ca30b022e6e57fadc7cca3c90fb43a2c313fdc1b3b0da188b0b5135"
Sep 17 18:06:18.994096 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:06:18.994076    2585 scope.go:117] "RemoveContainer" containerID="523e60b0b0bd27459c54f52d21bc84a48e796c09605329f1b421abc39d7ce5f0"
Sep 17 18:06:18.994415 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:06:18.994395    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"523e60b0b0bd27459c54f52d21bc84a48e796c09605329f1b421abc39d7ce5f0\": container with ID starting with 523e60b0b0bd27459c54f52d21bc84a48e796c09605329f1b421abc39d7ce5f0 not found: ID does not exist" containerID="523e60b0b0bd27459c54f52d21bc84a48e796c09605329f1b421abc39d7ce5f0"
Sep 17 18:06:18.994539 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:06:18.994428    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"523e60b0b0bd27459c54f52d21bc84a48e796c09605329f1b421abc39d7ce5f0"} err="failed to get container status \"523e60b0b0bd27459c54f52d21bc84a48e796c09605329f1b421abc39d7ce5f0\": rpc error: code = NotFound desc = could not find container \"523e60b0b0bd27459c54f52d21bc84a48e796c09605329f1b421abc39d7ce5f0\": container with ID starting with 523e60b0b0bd27459c54f52d21bc84a48e796c09605329f1b421abc39d7ce5f0 not found: ID does not exist"
Sep 17 18:06:18.994539 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:06:18.994450    2585 scope.go:117] "RemoveContainer" containerID="9a4a73b42a376333513a1534ae2f75e1aa7e37d8c0784b00f129b40136350caf"
Sep 17 18:06:18.994812 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:06:18.994794    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a4a73b42a376333513a1534ae2f75e1aa7e37d8c0784b00f129b40136350caf\": container with ID starting with 9a4a73b42a376333513a1534ae2f75e1aa7e37d8c0784b00f129b40136350caf not found: ID does not exist" containerID="9a4a73b42a376333513a1534ae2f75e1aa7e37d8c0784b00f129b40136350caf"
Sep 17 18:06:18.994889 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:06:18.994805    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jzwv9"]
Sep 17 18:06:18.994889 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:06:18.994820    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a4a73b42a376333513a1534ae2f75e1aa7e37d8c0784b00f129b40136350caf"} err="failed to get container status \"9a4a73b42a376333513a1534ae2f75e1aa7e37d8c0784b00f129b40136350caf\": rpc error: code = NotFound desc = could not find container \"9a4a73b42a376333513a1534ae2f75e1aa7e37d8c0784b00f129b40136350caf\": container with ID starting with 9a4a73b42a376333513a1534ae2f75e1aa7e37d8c0784b00f129b40136350caf not found: ID does not exist"
Sep 17 18:06:18.994889 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:06:18.994841    2585 scope.go:117] "RemoveContainer" containerID="7579236f0ca30b022e6e57fadc7cca3c90fb43a2c313fdc1b3b0da188b0b5135"
Sep 17 18:06:18.995161 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:06:18.995141    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7579236f0ca30b022e6e57fadc7cca3c90fb43a2c313fdc1b3b0da188b0b5135\": container with ID starting with 7579236f0ca30b022e6e57fadc7cca3c90fb43a2c313fdc1b3b0da188b0b5135 not found: ID does not exist" containerID="7579236f0ca30b022e6e57fadc7cca3c90fb43a2c313fdc1b3b0da188b0b5135"
Sep 17 18:06:18.995279 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:06:18.995164    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7579236f0ca30b022e6e57fadc7cca3c90fb43a2c313fdc1b3b0da188b0b5135"} err="failed to get container status \"7579236f0ca30b022e6e57fadc7cca3c90fb43a2c313fdc1b3b0da188b0b5135\": rpc error: code = NotFound desc = could not find container \"7579236f0ca30b022e6e57fadc7cca3c90fb43a2c313fdc1b3b0da188b0b5135\": container with ID starting with 7579236f0ca30b022e6e57fadc7cca3c90fb43a2c313fdc1b3b0da188b0b5135 not found: ID does not exist"
Sep 17 18:06:18.999605 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:06:18.999584    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-7cf78f884f-jzwv9"]
Sep 17 18:06:20.104251 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:06:20.104221    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d6fde28-25c1-4489-8ec2-1158d1efe4e4" path="/var/lib/kubelet/pods/6d6fde28-25c1-4489-8ec2-1158d1efe4e4/volumes"
Sep 17 18:07:16.934271 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:16.934188    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:40.353370 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:40.353321    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:43.871169 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:43.871088    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6"]
Sep 17 18:07:43.871589 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:43.871363    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6" podUID="c655034f-3e59-40d4-8a90-3aa731dd4f26" containerName="kserve-container" containerID="cri-o://432061c7980078b212864beebd65df318e0bf7f8cd8fc850158cb31500b51f1a" gracePeriod=30
Sep 17 18:07:43.871589 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:43.871391    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6" podUID="c655034f-3e59-40d4-8a90-3aa731dd4f26" containerName="kube-rbac-proxy" containerID="cri-o://444f8e92d793c44c32b3fb58a60ec4aa791864b20f251188e5badb062e3bbcf0" gracePeriod=30
Sep 17 18:07:43.931689 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:43.931651    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:44.274632 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:44.274598    2585 generic.go:356] "Generic (PLEG): container finished" podID="c655034f-3e59-40d4-8a90-3aa731dd4f26" containerID="444f8e92d793c44c32b3fb58a60ec4aa791864b20f251188e5badb062e3bbcf0" exitCode=2
Sep 17 18:07:44.274798 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:44.274661    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6" event={"ID":"c655034f-3e59-40d4-8a90-3aa731dd4f26","Type":"ContainerDied","Data":"444f8e92d793c44c32b3fb58a60ec4aa791864b20f251188e5badb062e3bbcf0"}
Sep 17 18:07:45.540971 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:45.540935    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:45.640983 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:45.640945    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6" podUID="c655034f-3e59-40d4-8a90-3aa731dd4f26" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Sep 17 18:07:46.862126 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:46.862094    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:48.608796 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:48.608771    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6"
Sep 17 18:07:48.682277 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:48.682206    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c655034f-3e59-40d4-8a90-3aa731dd4f26-proxy-tls\") pod \"c655034f-3e59-40d4-8a90-3aa731dd4f26\" (UID: \"c655034f-3e59-40d4-8a90-3aa731dd4f26\") "
Sep 17 18:07:48.682277 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:48.682256    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ctf66\" (UniqueName: \"kubernetes.io/projected/c655034f-3e59-40d4-8a90-3aa731dd4f26-kube-api-access-ctf66\") pod \"c655034f-3e59-40d4-8a90-3aa731dd4f26\" (UID: \"c655034f-3e59-40d4-8a90-3aa731dd4f26\") "
Sep 17 18:07:48.682472 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:48.682295    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c655034f-3e59-40d4-8a90-3aa731dd4f26-kserve-provision-location\") pod \"c655034f-3e59-40d4-8a90-3aa731dd4f26\" (UID: \"c655034f-3e59-40d4-8a90-3aa731dd4f26\") "
Sep 17 18:07:48.682472 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:48.682318    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c655034f-3e59-40d4-8a90-3aa731dd4f26-cabundle-cert\") pod \"c655034f-3e59-40d4-8a90-3aa731dd4f26\" (UID: \"c655034f-3e59-40d4-8a90-3aa731dd4f26\") "
Sep 17 18:07:48.682472 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:48.682343    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c655034f-3e59-40d4-8a90-3aa731dd4f26-kube-rbac-proxy-sar-config\") pod \"c655034f-3e59-40d4-8a90-3aa731dd4f26\" (UID: \"c655034f-3e59-40d4-8a90-3aa731dd4f26\") "
Sep 17 18:07:48.682728 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:48.682632    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c655034f-3e59-40d4-8a90-3aa731dd4f26-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c655034f-3e59-40d4-8a90-3aa731dd4f26" (UID: "c655034f-3e59-40d4-8a90-3aa731dd4f26"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:07:48.682785 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:48.682729    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c655034f-3e59-40d4-8a90-3aa731dd4f26-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c655034f-3e59-40d4-8a90-3aa731dd4f26" (UID: "c655034f-3e59-40d4-8a90-3aa731dd4f26"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:07:48.682785 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:48.682766    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c655034f-3e59-40d4-8a90-3aa731dd4f26-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c655034f-3e59-40d4-8a90-3aa731dd4f26" (UID: "c655034f-3e59-40d4-8a90-3aa731dd4f26"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:07:48.684336 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:48.684316    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c655034f-3e59-40d4-8a90-3aa731dd4f26-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c655034f-3e59-40d4-8a90-3aa731dd4f26" (UID: "c655034f-3e59-40d4-8a90-3aa731dd4f26"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:07:48.684416 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:48.684359    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c655034f-3e59-40d4-8a90-3aa731dd4f26-kube-api-access-ctf66" (OuterVolumeSpecName: "kube-api-access-ctf66") pod "c655034f-3e59-40d4-8a90-3aa731dd4f26" (UID: "c655034f-3e59-40d4-8a90-3aa731dd4f26"). InnerVolumeSpecName "kube-api-access-ctf66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:07:48.783663 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:48.783630    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c655034f-3e59-40d4-8a90-3aa731dd4f26-kserve-provision-location\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:07:48.783663 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:48.783656    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c655034f-3e59-40d4-8a90-3aa731dd4f26-cabundle-cert\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:07:48.783663 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:48.783666    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c655034f-3e59-40d4-8a90-3aa731dd4f26-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:07:48.783883 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:48.783677    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c655034f-3e59-40d4-8a90-3aa731dd4f26-proxy-tls\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:07:48.783883 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:48.783686    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctf66\" (UniqueName: \"kubernetes.io/projected/c655034f-3e59-40d4-8a90-3aa731dd4f26-kube-api-access-ctf66\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:07:49.117315 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.117279    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:49.180094 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.180058    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c"]
Sep 17 18:07:49.180310 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.180297    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d6fde28-25c1-4489-8ec2-1158d1efe4e4" containerName="storage-initializer"
Sep 17 18:07:49.180358 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.180312    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d6fde28-25c1-4489-8ec2-1158d1efe4e4" containerName="storage-initializer"
Sep 17 18:07:49.180358 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.180320    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d6fde28-25c1-4489-8ec2-1158d1efe4e4" containerName="kserve-container"
Sep 17 18:07:49.180358 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.180326    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d6fde28-25c1-4489-8ec2-1158d1efe4e4" containerName="kserve-container"
Sep 17 18:07:49.180358 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.180336    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d6fde28-25c1-4489-8ec2-1158d1efe4e4" containerName="kube-rbac-proxy"
Sep 17 18:07:49.180358 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.180342    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d6fde28-25c1-4489-8ec2-1158d1efe4e4" containerName="kube-rbac-proxy"
Sep 17 18:07:49.180358 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.180349    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c655034f-3e59-40d4-8a90-3aa731dd4f26" containerName="kserve-container"
Sep 17 18:07:49.180358 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.180355    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c655034f-3e59-40d4-8a90-3aa731dd4f26" containerName="kserve-container"
Sep 17 18:07:49.180632 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.180364    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c655034f-3e59-40d4-8a90-3aa731dd4f26" containerName="storage-initializer"
Sep 17 18:07:49.180632 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.180369    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c655034f-3e59-40d4-8a90-3aa731dd4f26" containerName="storage-initializer"
Sep 17 18:07:49.180632 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.180380    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c655034f-3e59-40d4-8a90-3aa731dd4f26" containerName="kube-rbac-proxy"
Sep 17 18:07:49.180632 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.180385    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c655034f-3e59-40d4-8a90-3aa731dd4f26" containerName="kube-rbac-proxy"
Sep 17 18:07:49.180632 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.180422    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d6fde28-25c1-4489-8ec2-1158d1efe4e4" containerName="kube-rbac-proxy"
Sep 17 18:07:49.180632 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.180429    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d6fde28-25c1-4489-8ec2-1158d1efe4e4" containerName="kserve-container"
Sep 17 18:07:49.180632 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.180437    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c655034f-3e59-40d4-8a90-3aa731dd4f26" containerName="kube-rbac-proxy"
Sep 17 18:07:49.180632 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.180444    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c655034f-3e59-40d4-8a90-3aa731dd4f26" containerName="kserve-container"
Sep 17 18:07:49.183582 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.183566    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c"
Sep 17 18:07:49.196737 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.196716    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 17 18:07:49.197169 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.197154    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Sep 17 18:07:49.223467 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.223439    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c"]
Sep 17 18:07:49.257407 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.257379    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:49.288534 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.288486    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f48bdd6-9b9d-448a-b625-fecb314969e6-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c\" (UID: \"6f48bdd6-9b9d-448a-b625-fecb314969e6\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c"
Sep 17 18:07:49.288653 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.288536    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f48bdd6-9b9d-448a-b625-fecb314969e6-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c\" (UID: \"6f48bdd6-9b9d-448a-b625-fecb314969e6\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c"
Sep 17 18:07:49.288653 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.288563    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5tvn\" (UniqueName: \"kubernetes.io/projected/6f48bdd6-9b9d-448a-b625-fecb314969e6-kube-api-access-l5tvn\") pod \"isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c\" (UID: \"6f48bdd6-9b9d-448a-b625-fecb314969e6\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c"
Sep 17 18:07:49.288653 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.288592    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f48bdd6-9b9d-448a-b625-fecb314969e6-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c\" (UID: \"6f48bdd6-9b9d-448a-b625-fecb314969e6\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c"
Sep 17 18:07:49.288653 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.288618    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f48bdd6-9b9d-448a-b625-fecb314969e6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c\" (UID: \"6f48bdd6-9b9d-448a-b625-fecb314969e6\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c"
Sep 17 18:07:49.293411 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.293386    2585 generic.go:356] "Generic (PLEG): container finished" podID="c655034f-3e59-40d4-8a90-3aa731dd4f26" containerID="432061c7980078b212864beebd65df318e0bf7f8cd8fc850158cb31500b51f1a" exitCode=0
Sep 17 18:07:49.293525 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.293421    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6" event={"ID":"c655034f-3e59-40d4-8a90-3aa731dd4f26","Type":"ContainerDied","Data":"432061c7980078b212864beebd65df318e0bf7f8cd8fc850158cb31500b51f1a"}
Sep 17 18:07:49.293525 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.293441    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6"
Sep 17 18:07:49.293525 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.293470    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6" event={"ID":"c655034f-3e59-40d4-8a90-3aa731dd4f26","Type":"ContainerDied","Data":"a9ec6e54b7aa9eb22f9b3959b40bae788751f05c1af7e1bdd2bb7383c9ecd47d"}
Sep 17 18:07:49.293525 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.293521    2585 scope.go:117] "RemoveContainer" containerID="444f8e92d793c44c32b3fb58a60ec4aa791864b20f251188e5badb062e3bbcf0"
Sep 17 18:07:49.301864 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.301836    2585 scope.go:117] "RemoveContainer" containerID="432061c7980078b212864beebd65df318e0bf7f8cd8fc850158cb31500b51f1a"
Sep 17 18:07:49.310798 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.310775    2585 scope.go:117] "RemoveContainer" containerID="f09b80ad54abaf0e971bf91303775b41129e89c76205265fc5f2a0b4d7a7719e"
Sep 17 18:07:49.318560 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.318542    2585 scope.go:117] "RemoveContainer" containerID="444f8e92d793c44c32b3fb58a60ec4aa791864b20f251188e5badb062e3bbcf0"
Sep 17 18:07:49.318830 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:07:49.318813    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"444f8e92d793c44c32b3fb58a60ec4aa791864b20f251188e5badb062e3bbcf0\": container with ID starting with 444f8e92d793c44c32b3fb58a60ec4aa791864b20f251188e5badb062e3bbcf0 not found: ID does not exist" containerID="444f8e92d793c44c32b3fb58a60ec4aa791864b20f251188e5badb062e3bbcf0"
Sep 17 18:07:49.318905 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.318840    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"444f8e92d793c44c32b3fb58a60ec4aa791864b20f251188e5badb062e3bbcf0"} err="failed to get container status \"444f8e92d793c44c32b3fb58a60ec4aa791864b20f251188e5badb062e3bbcf0\": rpc error: code = NotFound desc = could not find container \"444f8e92d793c44c32b3fb58a60ec4aa791864b20f251188e5badb062e3bbcf0\": container with ID starting with 444f8e92d793c44c32b3fb58a60ec4aa791864b20f251188e5badb062e3bbcf0 not found: ID does not exist"
Sep 17 18:07:49.318905 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.318863    2585 scope.go:117] "RemoveContainer" containerID="432061c7980078b212864beebd65df318e0bf7f8cd8fc850158cb31500b51f1a"
Sep 17 18:07:49.319122 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:07:49.319103    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"432061c7980078b212864beebd65df318e0bf7f8cd8fc850158cb31500b51f1a\": container with ID starting with 432061c7980078b212864beebd65df318e0bf7f8cd8fc850158cb31500b51f1a not found: ID does not exist" containerID="432061c7980078b212864beebd65df318e0bf7f8cd8fc850158cb31500b51f1a"
Sep 17 18:07:49.319167 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.319127    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"432061c7980078b212864beebd65df318e0bf7f8cd8fc850158cb31500b51f1a"} err="failed to get container status \"432061c7980078b212864beebd65df318e0bf7f8cd8fc850158cb31500b51f1a\": rpc error: code = NotFound desc = could not find container \"432061c7980078b212864beebd65df318e0bf7f8cd8fc850158cb31500b51f1a\": container with ID starting with 432061c7980078b212864beebd65df318e0bf7f8cd8fc850158cb31500b51f1a not found: ID does not exist"
Sep 17 18:07:49.319167 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.319142    2585 scope.go:117] "RemoveContainer" containerID="f09b80ad54abaf0e971bf91303775b41129e89c76205265fc5f2a0b4d7a7719e"
Sep 17 18:07:49.319389 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:07:49.319373    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f09b80ad54abaf0e971bf91303775b41129e89c76205265fc5f2a0b4d7a7719e\": container with ID starting with f09b80ad54abaf0e971bf91303775b41129e89c76205265fc5f2a0b4d7a7719e not found: ID does not exist" containerID="f09b80ad54abaf0e971bf91303775b41129e89c76205265fc5f2a0b4d7a7719e"
Sep 17 18:07:49.319448 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.319395    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f09b80ad54abaf0e971bf91303775b41129e89c76205265fc5f2a0b4d7a7719e"} err="failed to get container status \"f09b80ad54abaf0e971bf91303775b41129e89c76205265fc5f2a0b4d7a7719e\": rpc error: code = NotFound desc = could not find container \"f09b80ad54abaf0e971bf91303775b41129e89c76205265fc5f2a0b4d7a7719e\": container with ID starting with f09b80ad54abaf0e971bf91303775b41129e89c76205265fc5f2a0b4d7a7719e not found: ID does not exist"
Sep 17 18:07:49.324415 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.324396    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf"]
Sep 17 18:07:49.332324 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.332302    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf"
Sep 17 18:07:49.333525 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.333482    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6"]
Sep 17 18:07:49.335691 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.335674    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 18:07:49.339860 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.339841    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5b4b466b66-bwtr6"]
Sep 17 18:07:49.361150 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.361125    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf"]
Sep 17 18:07:49.389165 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.389116    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f48bdd6-9b9d-448a-b625-fecb314969e6-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c\" (UID: \"6f48bdd6-9b9d-448a-b625-fecb314969e6\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c"
Sep 17 18:07:49.389165 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.389149    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5tvn\" (UniqueName: \"kubernetes.io/projected/6f48bdd6-9b9d-448a-b625-fecb314969e6-kube-api-access-l5tvn\") pod \"isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c\" (UID: \"6f48bdd6-9b9d-448a-b625-fecb314969e6\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c"
Sep 17 18:07:49.389322 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.389171    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0f56b65-f575-4152-aad7-b4fcece232c1-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf\" (UID: \"e0f56b65-f575-4152-aad7-b4fcece232c1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf"
Sep 17 18:07:49.389322 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.389199    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f48bdd6-9b9d-448a-b625-fecb314969e6-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c\" (UID: \"6f48bdd6-9b9d-448a-b625-fecb314969e6\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c"
Sep 17 18:07:49.389322 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.389228    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f48bdd6-9b9d-448a-b625-fecb314969e6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c\" (UID: \"6f48bdd6-9b9d-448a-b625-fecb314969e6\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c"
Sep 17 18:07:49.389322 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.389273    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0f56b65-f575-4152-aad7-b4fcece232c1-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf\" (UID: \"e0f56b65-f575-4152-aad7-b4fcece232c1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf"
Sep 17 18:07:49.389322 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.389318    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0f56b65-f575-4152-aad7-b4fcece232c1-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf\" (UID: \"e0f56b65-f575-4152-aad7-b4fcece232c1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf"
Sep 17 18:07:49.389619 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.389363    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0f56b65-f575-4152-aad7-b4fcece232c1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf\" (UID: \"e0f56b65-f575-4152-aad7-b4fcece232c1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf"
Sep 17 18:07:49.389619 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.389420    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f48bdd6-9b9d-448a-b625-fecb314969e6-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c\" (UID: \"6f48bdd6-9b9d-448a-b625-fecb314969e6\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c"
Sep 17 18:07:49.389619 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.389449    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq9rj\" (UniqueName: \"kubernetes.io/projected/e0f56b65-f575-4152-aad7-b4fcece232c1-kube-api-access-bq9rj\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf\" (UID: \"e0f56b65-f575-4152-aad7-b4fcece232c1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf"
Sep 17 18:07:49.389758 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.389680    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f48bdd6-9b9d-448a-b625-fecb314969e6-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c\" (UID: \"6f48bdd6-9b9d-448a-b625-fecb314969e6\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c"
Sep 17 18:07:49.389813 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.389796    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f48bdd6-9b9d-448a-b625-fecb314969e6-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c\" (UID: \"6f48bdd6-9b9d-448a-b625-fecb314969e6\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c"
Sep 17 18:07:49.389980 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.389957    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f48bdd6-9b9d-448a-b625-fecb314969e6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c\" (UID: \"6f48bdd6-9b9d-448a-b625-fecb314969e6\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c"
Sep 17 18:07:49.391812 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.391798    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f48bdd6-9b9d-448a-b625-fecb314969e6-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c\" (UID: \"6f48bdd6-9b9d-448a-b625-fecb314969e6\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c"
Sep 17 18:07:49.419639 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.419616    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5tvn\" (UniqueName: \"kubernetes.io/projected/6f48bdd6-9b9d-448a-b625-fecb314969e6-kube-api-access-l5tvn\") pod \"isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c\" (UID: \"6f48bdd6-9b9d-448a-b625-fecb314969e6\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c"
Sep 17 18:07:49.490157 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.490128    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0f56b65-f575-4152-aad7-b4fcece232c1-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf\" (UID: \"e0f56b65-f575-4152-aad7-b4fcece232c1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf"
Sep 17 18:07:49.490315 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.490165    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0f56b65-f575-4152-aad7-b4fcece232c1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf\" (UID: \"e0f56b65-f575-4152-aad7-b4fcece232c1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf"
Sep 17 18:07:49.490315 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.490187    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq9rj\" (UniqueName: \"kubernetes.io/projected/e0f56b65-f575-4152-aad7-b4fcece232c1-kube-api-access-bq9rj\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf\" (UID: \"e0f56b65-f575-4152-aad7-b4fcece232c1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf"
Sep 17 18:07:49.490315 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.490225    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0f56b65-f575-4152-aad7-b4fcece232c1-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf\" (UID: \"e0f56b65-f575-4152-aad7-b4fcece232c1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf"
Sep 17 18:07:49.490315 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.490264    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0f56b65-f575-4152-aad7-b4fcece232c1-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf\" (UID: \"e0f56b65-f575-4152-aad7-b4fcece232c1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf"
Sep 17 18:07:49.490577 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:07:49.490370    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 17 18:07:49.490577 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:07:49.490433    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0f56b65-f575-4152-aad7-b4fcece232c1-proxy-tls podName:e0f56b65-f575-4152-aad7-b4fcece232c1 nodeName:}" failed. No retries permitted until 2026-09-17 18:07:49.990413597 +0000 UTC m=+2006.404204015 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e0f56b65-f575-4152-aad7-b4fcece232c1-proxy-tls") pod "isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf" (UID: "e0f56b65-f575-4152-aad7-b4fcece232c1") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 17 18:07:49.490669 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.490618    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0f56b65-f575-4152-aad7-b4fcece232c1-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf\" (UID: \"e0f56b65-f575-4152-aad7-b4fcece232c1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf"
Sep 17 18:07:49.490845 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.490828    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0f56b65-f575-4152-aad7-b4fcece232c1-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf\" (UID: \"e0f56b65-f575-4152-aad7-b4fcece232c1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf"
Sep 17 18:07:49.490909 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.490892    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0f56b65-f575-4152-aad7-b4fcece232c1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf\" (UID: \"e0f56b65-f575-4152-aad7-b4fcece232c1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf"
Sep 17 18:07:49.493133 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.493110    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c"
Sep 17 18:07:49.507585 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.507565    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq9rj\" (UniqueName: \"kubernetes.io/projected/e0f56b65-f575-4152-aad7-b4fcece232c1-kube-api-access-bq9rj\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf\" (UID: \"e0f56b65-f575-4152-aad7-b4fcece232c1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf"
Sep 17 18:07:49.792244 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.792206    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c"]
Sep 17 18:07:49.795433 ip-10-0-133-139 kubenswrapper[2585]: W0917 18:07:49.795408    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f48bdd6_9b9d_448a_b625_fecb314969e6.slice/crio-d7f04c36fab70d9d84ab8d80762bdb19ae2852a69f88f454070839cbf6b1e835 WatchSource:0}: Error finding container d7f04c36fab70d9d84ab8d80762bdb19ae2852a69f88f454070839cbf6b1e835: Status 404 returned error can't find the container with id d7f04c36fab70d9d84ab8d80762bdb19ae2852a69f88f454070839cbf6b1e835
Sep 17 18:07:49.830537 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.830510    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:49.994379 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.994346    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0f56b65-f575-4152-aad7-b4fcece232c1-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf\" (UID: \"e0f56b65-f575-4152-aad7-b4fcece232c1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf"
Sep 17 18:07:49.996696 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:49.996676    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0f56b65-f575-4152-aad7-b4fcece232c1-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf\" (UID: \"e0f56b65-f575-4152-aad7-b4fcece232c1\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf"
Sep 17 18:07:50.104814 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:50.104730    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c655034f-3e59-40d4-8a90-3aa731dd4f26" path="/var/lib/kubelet/pods/c655034f-3e59-40d4-8a90-3aa731dd4f26/volumes"
Sep 17 18:07:50.242928 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:50.242899    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf"
Sep 17 18:07:50.301298 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:50.301264    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c" event={"ID":"6f48bdd6-9b9d-448a-b625-fecb314969e6","Type":"ContainerStarted","Data":"ed353541f19a81d70cfae54e241fdf9c16e9f7e254debfd7d890302ad25b4aea"}
Sep 17 18:07:50.301417 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:50.301339    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c" event={"ID":"6f48bdd6-9b9d-448a-b625-fecb314969e6","Type":"ContainerStarted","Data":"d7f04c36fab70d9d84ab8d80762bdb19ae2852a69f88f454070839cbf6b1e835"}
Sep 17 18:07:50.533898 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:50.533649    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf"]
Sep 17 18:07:50.536177 ip-10-0-133-139 kubenswrapper[2585]: W0917 18:07:50.536152    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0f56b65_f575_4152_aad7_b4fcece232c1.slice/crio-05477dd6eda03aab9832930b16bd6d550225a6baf7f31a79d9683941b6e12e74 WatchSource:0}: Error finding container 05477dd6eda03aab9832930b16bd6d550225a6baf7f31a79d9683941b6e12e74: Status 404 returned error can't find the container with id 05477dd6eda03aab9832930b16bd6d550225a6baf7f31a79d9683941b6e12e74
Sep 17 18:07:51.307893 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:51.307849    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf" event={"ID":"e0f56b65-f575-4152-aad7-b4fcece232c1","Type":"ContainerStarted","Data":"9255985cf435544fa4c9d8f77e98a500489cca7a5a4da32281540250ee0c9764"}
Sep 17 18:07:51.308251 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:51.307913    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf" event={"ID":"e0f56b65-f575-4152-aad7-b4fcece232c1","Type":"ContainerStarted","Data":"05477dd6eda03aab9832930b16bd6d550225a6baf7f31a79d9683941b6e12e74"}
Sep 17 18:07:54.320444 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:54.320413    2585 generic.go:356] "Generic (PLEG): container finished" podID="6f48bdd6-9b9d-448a-b625-fecb314969e6" containerID="ed353541f19a81d70cfae54e241fdf9c16e9f7e254debfd7d890302ad25b4aea" exitCode=0
Sep 17 18:07:54.320814 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:54.320475    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c" event={"ID":"6f48bdd6-9b9d-448a-b625-fecb314969e6","Type":"ContainerDied","Data":"ed353541f19a81d70cfae54e241fdf9c16e9f7e254debfd7d890302ad25b4aea"}
Sep 17 18:07:54.410032 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:54.410005    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:54.535335 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:54.535295    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:55.325550 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:55.325519    2585 generic.go:356] "Generic (PLEG): container finished" podID="e0f56b65-f575-4152-aad7-b4fcece232c1" containerID="9255985cf435544fa4c9d8f77e98a500489cca7a5a4da32281540250ee0c9764" exitCode=0
Sep 17 18:07:55.325969 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:55.325608    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf" event={"ID":"e0f56b65-f575-4152-aad7-b4fcece232c1","Type":"ContainerDied","Data":"9255985cf435544fa4c9d8f77e98a500489cca7a5a4da32281540250ee0c9764"}
Sep 17 18:07:55.330563 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:55.330535    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c" event={"ID":"6f48bdd6-9b9d-448a-b625-fecb314969e6","Type":"ContainerStarted","Data":"dfeebeaed70e91cfd3258618d265419edb7f28887919b56f1f3137006ffd375f"}
Sep 17 18:07:55.330669 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:55.330586    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c" event={"ID":"6f48bdd6-9b9d-448a-b625-fecb314969e6","Type":"ContainerStarted","Data":"cc8a6c7b77d1ca427403ba315352684c02d18a9355f2e58e7e9976ced4354176"}
Sep 17 18:07:55.330790 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:55.330773    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c"
Sep 17 18:07:56.336982 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:56.336949    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf" event={"ID":"e0f56b65-f575-4152-aad7-b4fcece232c1","Type":"ContainerStarted","Data":"a7045ca15ccab35999a87af54e91bc00de1b903a6bdd7471634d2b244d8582af"}
Sep 17 18:07:56.337420 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:56.337006    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf" event={"ID":"e0f56b65-f575-4152-aad7-b4fcece232c1","Type":"ContainerStarted","Data":"3f5808858382ea10bc40f5b079fdf1281634a704190aa27f82fc3686dad3f2c2"}
Sep 17 18:07:56.337420 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:56.337250    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c"
Sep 17 18:07:56.337518 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:56.337462    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf"
Sep 17 18:07:56.338451 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:56.338427    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c" podUID="6f48bdd6-9b9d-448a-b625-fecb314969e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 18:07:56.381639 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:56.381598    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c" podStartSLOduration=3.56387788 podStartE2EDuration="7.381583801s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.817705921s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:07:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:07:55.403299599 +0000 UTC m=+2011.817090054" watchObservedRunningTime="2026-09-17 18:07:56.381583801 +0000 UTC m=+2012.795374281"
Sep 17 18:07:57.341485 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:57.341445    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf"
Sep 17 18:07:57.341886 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:57.341598    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c" podUID="6f48bdd6-9b9d-448a-b625-fecb314969e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 18:07:57.342532 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:57.342509    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf" podUID="e0f56b65-f575-4152-aad7-b4fcece232c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:07:58.345276 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:07:58.345238    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf" podUID="e0f56b65-f575-4152-aad7-b4fcece232c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:08:02.346319 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:08:02.346290    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c"
Sep 17 18:08:02.346801 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:08:02.346773    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c" podUID="6f48bdd6-9b9d-448a-b625-fecb314969e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 18:08:02.382314 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:08:02.382260    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf" podStartSLOduration=9.506282646 podStartE2EDuration="13.382244897s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.875962251s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:07:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:07:56.384863058 +0000 UTC m=+2012.798653496" watchObservedRunningTime="2026-09-17 18:08:02.382244897 +0000 UTC m=+2018.796035336"
Sep 17 18:08:03.349959 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:08:03.349929    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf"
Sep 17 18:08:03.350625 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:08:03.350595    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf" podUID="e0f56b65-f575-4152-aad7-b4fcece232c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:08:12.347737 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:08:12.347695    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c" podUID="6f48bdd6-9b9d-448a-b625-fecb314969e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 18:08:13.351087 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:08:13.351052    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf" podUID="e0f56b65-f575-4152-aad7-b4fcece232c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:08:22.346994 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:08:22.346955    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c" podUID="6f48bdd6-9b9d-448a-b625-fecb314969e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 18:08:23.351164 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:08:23.351121    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf" podUID="e0f56b65-f575-4152-aad7-b4fcece232c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:08:32.347671 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:08:32.347635    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c" podUID="6f48bdd6-9b9d-448a-b625-fecb314969e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 18:08:33.351426 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:08:33.351388    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf" podUID="e0f56b65-f575-4152-aad7-b4fcece232c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:08:36.462411 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:08:36.462370    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:41.591630 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:08:41.591592    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:41.623387 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:08:41.623353    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7"]
Sep 17 18:08:41.626736 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:08:41.626713    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7"
Sep 17 18:08:41.629774 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:08:41.629748    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\""
Sep 17 18:08:41.629774 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:08:41.629764    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-predictor-serving-cert\""
Sep 17 18:08:41.653653 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:08:41.653632    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7"]
Sep 17 18:08:41.785921 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:08:41.785887    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/536c3c62-e109-41c3-8b42-ec33ca5f524b-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7\" (UID: \"536c3c62-e109-41c3-8b42-ec33ca5f524b\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7"
Sep 17 18:08:41.785921 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:08:41.785925    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/536c3c62-e109-41c3-8b42-ec33ca5f524b-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7\" (UID: \"536c3c62-e109-41c3-8b42-ec33ca5f524b\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7"
Sep 17 18:08:41.786125 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:08:41.785946    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/536c3c62-e109-41c3-8b42-ec33ca5f524b-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7\" (UID: \"536c3c62-e109-41c3-8b42-ec33ca5f524b\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7"
Sep 17 18:08:41.786125 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:08:41.785969    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/536c3c62-e109-41c3-8b42-ec33ca5f524b-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7\" (UID: \"536c3c62-e109-41c3-8b42-ec33ca5f524b\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7"
Sep 17 18:08:41.786125 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:08:41.786041    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txt9l\" (UniqueName: \"kubernetes.io/projected/536c3c62-e109-41c3-8b42-ec33ca5f524b-kube-api-access-txt9l\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7\" (UID: \"536c3c62-e109-41c3-8b42-ec33ca5f524b\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7"
Sep 17 18:08:41.887265 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:08:41.887176    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txt9l\" (UniqueName: \"kubernetes.io/projected/536c3c62-e109-41c3-8b42-ec33ca5f524b-kube-api-access-txt9l\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7\" (UID: \"536c3c62-e109-41c3-8b42-ec33ca5f524b\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7"
Sep 17 18:08:41.887408 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:08:41.887271    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/536c3c62-e109-41c3-8b42-ec33ca5f524b-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7\" (UID: \"536c3c62-e109-41c3-8b42-ec33ca5f524b\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7"
Sep 17 18:08:41.887408 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:08:41.887301    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/536c3c62-e109-41c3-8b42-ec33ca5f524b-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7\" (UID: \"536c3c62-e109-41c3-8b42-ec33ca5f524b\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7"
Sep 17 18:08:41.887408 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:08:41.887337    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/536c3c62-e109-41c3-8b42-ec33ca5f524b-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7\" (UID: \"536c3c62-e109-41c3-8b42-ec33ca5f524b\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7"
Sep 17 18:08:41.887408 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:08:41.887365    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/536c3c62-e109-41c3-8b42-ec33ca5f524b-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7\" (UID: \"536c3c62-e109-41c3-8b42-ec33ca5f524b\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7"
Sep 17 18:08:41.887608 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:08:41.887436    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-serving-cert: secret "isvc-tensorflow-runtime-predictor-serving-cert" not found
Sep 17 18:08:41.887608 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:08:41.887533    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/536c3c62-e109-41c3-8b42-ec33ca5f524b-proxy-tls podName:536c3c62-e109-41c3-8b42-ec33ca5f524b nodeName:}" failed. No retries permitted until 2026-09-17 18:08:42.387510998 +0000 UTC m=+2058.801301430 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/536c3c62-e109-41c3-8b42-ec33ca5f524b-proxy-tls") pod "isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7" (UID: "536c3c62-e109-41c3-8b42-ec33ca5f524b") : secret "isvc-tensorflow-runtime-predictor-serving-cert" not found
Sep 17 18:08:41.887789 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:08:41.887772    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/536c3c62-e109-41c3-8b42-ec33ca5f524b-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7\" (UID: \"536c3c62-e109-41c3-8b42-ec33ca5f524b\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7"
Sep 17 18:08:41.888047 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:08:41.888030    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/536c3c62-e109-41c3-8b42-ec33ca5f524b-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7\" (UID: \"536c3c62-e109-41c3-8b42-ec33ca5f524b\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7"
Sep 17 18:08:41.888108 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:08:41.888091    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/536c3c62-e109-41c3-8b42-ec33ca5f524b-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7\" (UID: \"536c3c62-e109-41c3-8b42-ec33ca5f524b\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7"
Sep 17 18:08:41.898180 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:08:41.898155    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txt9l\" (UniqueName: \"kubernetes.io/projected/536c3c62-e109-41c3-8b42-ec33ca5f524b-kube-api-access-txt9l\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7\" (UID: \"536c3c62-e109-41c3-8b42-ec33ca5f524b\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7"
Sep 17 18:08:42.106841 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:08:42.106791    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:42.346712 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:08:42.346678    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c" podUID="6f48bdd6-9b9d-448a-b625-fecb314969e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 18:08:42.390220 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:08:42.390192    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/536c3c62-e109-41c3-8b42-ec33ca5f524b-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7\" (UID: \"536c3c62-e109-41c3-8b42-ec33ca5f524b\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7"
Sep 17 18:08:42.392630 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:08:42.392605    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/536c3c62-e109-41c3-8b42-ec33ca5f524b-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7\" (UID: \"536c3c62-e109-41c3-8b42-ec33ca5f524b\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7"
Sep 17 18:08:42.537144 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:08:42.537108    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7"
Sep 17 18:08:42.817707 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:08:42.817678    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7"]
Sep 17 18:08:42.819271 ip-10-0-133-139 kubenswrapper[2585]: W0917 18:08:42.819246    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod536c3c62_e109_41c3_8b42_ec33ca5f524b.slice/crio-d6b4cf9a2601d79b2d115c298096f5993ef15fbbd3672e7be50f4e84508b1170 WatchSource:0}: Error finding container d6b4cf9a2601d79b2d115c298096f5993ef15fbbd3672e7be50f4e84508b1170: Status 404 returned error can't find the container with id d6b4cf9a2601d79b2d115c298096f5993ef15fbbd3672e7be50f4e84508b1170
Sep 17 18:08:43.350625 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:08:43.350591    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf" podUID="e0f56b65-f575-4152-aad7-b4fcece232c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:08:43.516795 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:08:43.516759    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7" event={"ID":"536c3c62-e109-41c3-8b42-ec33ca5f524b","Type":"ContainerStarted","Data":"1bcc43839e4c12fa8a99ee921b61ecee37181efa124ceb013ce26221e72a563d"}
Sep 17 18:08:43.516985 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:08:43.516827    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7" event={"ID":"536c3c62-e109-41c3-8b42-ec33ca5f524b","Type":"ContainerStarted","Data":"d6b4cf9a2601d79b2d115c298096f5993ef15fbbd3672e7be50f4e84508b1170"}
Sep 17 18:08:47.531392 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:08:47.531303    2585 generic.go:356] "Generic (PLEG): container finished" podID="536c3c62-e109-41c3-8b42-ec33ca5f524b" containerID="1bcc43839e4c12fa8a99ee921b61ecee37181efa124ceb013ce26221e72a563d" exitCode=0
Sep 17 18:08:47.531392 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:08:47.531378    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7" event={"ID":"536c3c62-e109-41c3-8b42-ec33ca5f524b","Type":"ContainerDied","Data":"1bcc43839e4c12fa8a99ee921b61ecee37181efa124ceb013ce26221e72a563d"}
Sep 17 18:08:47.532567 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:08:47.532552    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:08:51.555554 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:08:51.555512    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7" event={"ID":"536c3c62-e109-41c3-8b42-ec33ca5f524b","Type":"ContainerStarted","Data":"f48b1bc5c1dd0dc5a0c3d9fe65ce5927baaad267b51deb64b9d95527dc29bf59"}
Sep 17 18:08:51.555938 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:08:51.555580    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7" event={"ID":"536c3c62-e109-41c3-8b42-ec33ca5f524b","Type":"ContainerStarted","Data":"51e8cf1333455424b069770eb8689c2ad3649ab3a93dcd3ff9abeec874725f37"}
Sep 17 18:08:51.555938 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:08:51.555872    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7"
Sep 17 18:08:51.556025 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:08:51.556014    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7"
Sep 17 18:08:51.557425 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:08:51.557395    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7" podUID="536c3c62-e109-41c3-8b42-ec33ca5f524b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 18:08:51.594158 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:08:51.594108    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7" podStartSLOduration=2.583185572 podStartE2EDuration="10.594091514s" totalImagesPullingTime="3.616909954s" totalInitContainerRuntime="4.393995988s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:08:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:08:51.59391628 +0000 UTC m=+2068.007706718" watchObservedRunningTime="2026-09-17 18:08:51.594091514 +0000 UTC m=+2068.007881953"
Sep 17 18:08:52.347700 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:08:52.347663    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c" podUID="6f48bdd6-9b9d-448a-b625-fecb314969e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 18:08:52.559672 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:08:52.559632    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7" podUID="536c3c62-e109-41c3-8b42-ec33ca5f524b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 18:08:53.351441 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:08:53.351398    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf" podUID="e0f56b65-f575-4152-aad7-b4fcece232c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:08:57.564177 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:08:57.564151    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7"
Sep 17 18:08:57.564675 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:08:57.564645    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7" podUID="536c3c62-e109-41c3-8b42-ec33ca5f524b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 18:09:02.347386 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:02.347348    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c" podUID="6f48bdd6-9b9d-448a-b625-fecb314969e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 18:09:03.351546 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:03.351486    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf" podUID="e0f56b65-f575-4152-aad7-b4fcece232c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 18:09:07.565954 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:07.565926    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7"
Sep 17 18:09:12.347656 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:12.347628    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c"
Sep 17 18:09:12.585019 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:12.584984    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf"]
Sep 17 18:09:12.585241 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:12.585220    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf" podUID="e0f56b65-f575-4152-aad7-b4fcece232c1" containerName="kserve-container" containerID="cri-o://3f5808858382ea10bc40f5b079fdf1281634a704190aa27f82fc3686dad3f2c2" gracePeriod=30
Sep 17 18:09:12.585376 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:12.585317    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf" podUID="e0f56b65-f575-4152-aad7-b4fcece232c1" containerName="kube-rbac-proxy" containerID="cri-o://a7045ca15ccab35999a87af54e91bc00de1b903a6bdd7471634d2b244d8582af" gracePeriod=30
Sep 17 18:09:12.586523 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:12.586485    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf"
Sep 17 18:09:13.345880 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:13.345839    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf" podUID="e0f56b65-f575-4152-aad7-b4fcece232c1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Sep 17 18:09:13.639606 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:13.639525    2585 generic.go:356] "Generic (PLEG): container finished" podID="e0f56b65-f575-4152-aad7-b4fcece232c1" containerID="a7045ca15ccab35999a87af54e91bc00de1b903a6bdd7471634d2b244d8582af" exitCode=2
Sep 17 18:09:13.639943 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:13.639601    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf" event={"ID":"e0f56b65-f575-4152-aad7-b4fcece232c1","Type":"ContainerDied","Data":"a7045ca15ccab35999a87af54e91bc00de1b903a6bdd7471634d2b244d8582af"}
Sep 17 18:09:17.229744 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:17.229722    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf"
Sep 17 18:09:17.254692 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:17.254667    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0f56b65-f575-4152-aad7-b4fcece232c1-proxy-tls\") pod \"e0f56b65-f575-4152-aad7-b4fcece232c1\" (UID: \"e0f56b65-f575-4152-aad7-b4fcece232c1\") "
Sep 17 18:09:17.254801 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:17.254700    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0f56b65-f575-4152-aad7-b4fcece232c1-kube-rbac-proxy-sar-config\") pod \"e0f56b65-f575-4152-aad7-b4fcece232c1\" (UID: \"e0f56b65-f575-4152-aad7-b4fcece232c1\") "
Sep 17 18:09:17.254801 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:17.254723    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0f56b65-f575-4152-aad7-b4fcece232c1-kserve-provision-location\") pod \"e0f56b65-f575-4152-aad7-b4fcece232c1\" (UID: \"e0f56b65-f575-4152-aad7-b4fcece232c1\") "
Sep 17 18:09:17.254876 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:17.254837    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0f56b65-f575-4152-aad7-b4fcece232c1-cabundle-cert\") pod \"e0f56b65-f575-4152-aad7-b4fcece232c1\" (UID: \"e0f56b65-f575-4152-aad7-b4fcece232c1\") "
Sep 17 18:09:17.254946 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:17.254902    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bq9rj\" (UniqueName: \"kubernetes.io/projected/e0f56b65-f575-4152-aad7-b4fcece232c1-kube-api-access-bq9rj\") pod \"e0f56b65-f575-4152-aad7-b4fcece232c1\" (UID: \"e0f56b65-f575-4152-aad7-b4fcece232c1\") "
Sep 17 18:09:17.255027 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:17.255003    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0f56b65-f575-4152-aad7-b4fcece232c1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e0f56b65-f575-4152-aad7-b4fcece232c1" (UID: "e0f56b65-f575-4152-aad7-b4fcece232c1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:09:17.255179 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:17.255149    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0f56b65-f575-4152-aad7-b4fcece232c1-kserve-provision-location\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:09:17.255179 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:17.255146    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0f56b65-f575-4152-aad7-b4fcece232c1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e0f56b65-f575-4152-aad7-b4fcece232c1" (UID: "e0f56b65-f575-4152-aad7-b4fcece232c1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:09:17.255338 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:17.255198    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0f56b65-f575-4152-aad7-b4fcece232c1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e0f56b65-f575-4152-aad7-b4fcece232c1" (UID: "e0f56b65-f575-4152-aad7-b4fcece232c1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:09:17.256782 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:17.256751    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0f56b65-f575-4152-aad7-b4fcece232c1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e0f56b65-f575-4152-aad7-b4fcece232c1" (UID: "e0f56b65-f575-4152-aad7-b4fcece232c1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:09:17.256893 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:17.256856    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0f56b65-f575-4152-aad7-b4fcece232c1-kube-api-access-bq9rj" (OuterVolumeSpecName: "kube-api-access-bq9rj") pod "e0f56b65-f575-4152-aad7-b4fcece232c1" (UID: "e0f56b65-f575-4152-aad7-b4fcece232c1"). InnerVolumeSpecName "kube-api-access-bq9rj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:09:17.356063 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:17.355979    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0f56b65-f575-4152-aad7-b4fcece232c1-cabundle-cert\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:09:17.356063 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:17.356013    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq9rj\" (UniqueName: \"kubernetes.io/projected/e0f56b65-f575-4152-aad7-b4fcece232c1-kube-api-access-bq9rj\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:09:17.356063 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:17.356028    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0f56b65-f575-4152-aad7-b4fcece232c1-proxy-tls\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:09:17.356063 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:17.356041    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0f56b65-f575-4152-aad7-b4fcece232c1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:09:17.656435 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:17.656353    2585 generic.go:356] "Generic (PLEG): container finished" podID="e0f56b65-f575-4152-aad7-b4fcece232c1" containerID="3f5808858382ea10bc40f5b079fdf1281634a704190aa27f82fc3686dad3f2c2" exitCode=0
Sep 17 18:09:17.656435 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:17.656422    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf"
Sep 17 18:09:17.656626 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:17.656438    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf" event={"ID":"e0f56b65-f575-4152-aad7-b4fcece232c1","Type":"ContainerDied","Data":"3f5808858382ea10bc40f5b079fdf1281634a704190aa27f82fc3686dad3f2c2"}
Sep 17 18:09:17.656626 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:17.656517    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf" event={"ID":"e0f56b65-f575-4152-aad7-b4fcece232c1","Type":"ContainerDied","Data":"05477dd6eda03aab9832930b16bd6d550225a6baf7f31a79d9683941b6e12e74"}
Sep 17 18:09:17.656626 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:17.656546    2585 scope.go:117] "RemoveContainer" containerID="a7045ca15ccab35999a87af54e91bc00de1b903a6bdd7471634d2b244d8582af"
Sep 17 18:09:17.665404 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:17.665389    2585 scope.go:117] "RemoveContainer" containerID="3f5808858382ea10bc40f5b079fdf1281634a704190aa27f82fc3686dad3f2c2"
Sep 17 18:09:17.672533 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:17.672516    2585 scope.go:117] "RemoveContainer" containerID="9255985cf435544fa4c9d8f77e98a500489cca7a5a4da32281540250ee0c9764"
Sep 17 18:09:17.679259 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:17.679239    2585 scope.go:117] "RemoveContainer" containerID="a7045ca15ccab35999a87af54e91bc00de1b903a6bdd7471634d2b244d8582af"
Sep 17 18:09:17.679543 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:09:17.679524    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7045ca15ccab35999a87af54e91bc00de1b903a6bdd7471634d2b244d8582af\": container with ID starting with a7045ca15ccab35999a87af54e91bc00de1b903a6bdd7471634d2b244d8582af not found: ID does not exist" containerID="a7045ca15ccab35999a87af54e91bc00de1b903a6bdd7471634d2b244d8582af"
Sep 17 18:09:17.679628 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:17.679549    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7045ca15ccab35999a87af54e91bc00de1b903a6bdd7471634d2b244d8582af"} err="failed to get container status \"a7045ca15ccab35999a87af54e91bc00de1b903a6bdd7471634d2b244d8582af\": rpc error: code = NotFound desc = could not find container \"a7045ca15ccab35999a87af54e91bc00de1b903a6bdd7471634d2b244d8582af\": container with ID starting with a7045ca15ccab35999a87af54e91bc00de1b903a6bdd7471634d2b244d8582af not found: ID does not exist"
Sep 17 18:09:17.679628 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:17.679565    2585 scope.go:117] "RemoveContainer" containerID="3f5808858382ea10bc40f5b079fdf1281634a704190aa27f82fc3686dad3f2c2"
Sep 17 18:09:17.679824 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:09:17.679807    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f5808858382ea10bc40f5b079fdf1281634a704190aa27f82fc3686dad3f2c2\": container with ID starting with 3f5808858382ea10bc40f5b079fdf1281634a704190aa27f82fc3686dad3f2c2 not found: ID does not exist" containerID="3f5808858382ea10bc40f5b079fdf1281634a704190aa27f82fc3686dad3f2c2"
Sep 17 18:09:17.679872 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:17.679830    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f5808858382ea10bc40f5b079fdf1281634a704190aa27f82fc3686dad3f2c2"} err="failed to get container status \"3f5808858382ea10bc40f5b079fdf1281634a704190aa27f82fc3686dad3f2c2\": rpc error: code = NotFound desc = could not find container \"3f5808858382ea10bc40f5b079fdf1281634a704190aa27f82fc3686dad3f2c2\": container with ID starting with 3f5808858382ea10bc40f5b079fdf1281634a704190aa27f82fc3686dad3f2c2 not found: ID does not exist"
Sep 17 18:09:17.679872 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:17.679846    2585 scope.go:117] "RemoveContainer" containerID="9255985cf435544fa4c9d8f77e98a500489cca7a5a4da32281540250ee0c9764"
Sep 17 18:09:17.680103 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:09:17.680084    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9255985cf435544fa4c9d8f77e98a500489cca7a5a4da32281540250ee0c9764\": container with ID starting with 9255985cf435544fa4c9d8f77e98a500489cca7a5a4da32281540250ee0c9764 not found: ID does not exist" containerID="9255985cf435544fa4c9d8f77e98a500489cca7a5a4da32281540250ee0c9764"
Sep 17 18:09:17.680159 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:17.680111    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9255985cf435544fa4c9d8f77e98a500489cca7a5a4da32281540250ee0c9764"} err="failed to get container status \"9255985cf435544fa4c9d8f77e98a500489cca7a5a4da32281540250ee0c9764\": rpc error: code = NotFound desc = could not find container \"9255985cf435544fa4c9d8f77e98a500489cca7a5a4da32281540250ee0c9764\": container with ID starting with 9255985cf435544fa4c9d8f77e98a500489cca7a5a4da32281540250ee0c9764 not found: ID does not exist"
Sep 17 18:09:17.684303 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:17.684280    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf"]
Sep 17 18:09:17.688032 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:17.688014    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-mx5cf"]
Sep 17 18:09:18.104290 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:18.104255    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0f56b65-f575-4152-aad7-b4fcece232c1" path="/var/lib/kubelet/pods/e0f56b65-f575-4152-aad7-b4fcece232c1/volumes"
Sep 17 18:09:19.071486 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:19.071452    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c"]
Sep 17 18:09:19.071905 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:19.071752    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c" podUID="6f48bdd6-9b9d-448a-b625-fecb314969e6" containerName="kserve-container" containerID="cri-o://cc8a6c7b77d1ca427403ba315352684c02d18a9355f2e58e7e9976ced4354176" gracePeriod=30
Sep 17 18:09:19.071905 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:19.071780    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c" podUID="6f48bdd6-9b9d-448a-b625-fecb314969e6" containerName="kube-rbac-proxy" containerID="cri-o://dfeebeaed70e91cfd3258618d265419edb7f28887919b56f1f3137006ffd375f" gracePeriod=30
Sep 17 18:09:19.132415 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:19.132387    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-4whhd"]
Sep 17 18:09:19.132664 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:19.132652    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0f56b65-f575-4152-aad7-b4fcece232c1" containerName="kserve-container"
Sep 17 18:09:19.132711 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:19.132666    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0f56b65-f575-4152-aad7-b4fcece232c1" containerName="kserve-container"
Sep 17 18:09:19.132711 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:19.132681    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0f56b65-f575-4152-aad7-b4fcece232c1" containerName="kube-rbac-proxy"
Sep 17 18:09:19.132711 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:19.132686    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0f56b65-f575-4152-aad7-b4fcece232c1" containerName="kube-rbac-proxy"
Sep 17 18:09:19.132711 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:19.132700    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0f56b65-f575-4152-aad7-b4fcece232c1" containerName="storage-initializer"
Sep 17 18:09:19.132711 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:19.132705    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0f56b65-f575-4152-aad7-b4fcece232c1" containerName="storage-initializer"
Sep 17 18:09:19.132866 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:19.132749    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0f56b65-f575-4152-aad7-b4fcece232c1" containerName="kserve-container"
Sep 17 18:09:19.132866 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:19.132758    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0f56b65-f575-4152-aad7-b4fcece232c1" containerName="kube-rbac-proxy"
Sep 17 18:09:19.137226 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:19.137210    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-4whhd"
Sep 17 18:09:19.140048 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:19.140030    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 18:09:19.153292 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:19.153271    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-4whhd"]
Sep 17 18:09:19.154029 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:19.154009    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:09:19.168957 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:19.168926    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5eec5a9b-2547-46c0-996d-7d7129ab1b52-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-4whhd\" (UID: \"5eec5a9b-2547-46c0-996d-7d7129ab1b52\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-4whhd"
Sep 17 18:09:19.169066 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:19.168967    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5eec5a9b-2547-46c0-996d-7d7129ab1b52-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-4whhd\" (UID: \"5eec5a9b-2547-46c0-996d-7d7129ab1b52\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-4whhd"
Sep 17 18:09:19.169066 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:19.169003    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5eec5a9b-2547-46c0-996d-7d7129ab1b52-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-4whhd\" (UID: \"5eec5a9b-2547-46c0-996d-7d7129ab1b52\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-4whhd"
Sep 17 18:09:19.169066 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:19.169028    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5eec5a9b-2547-46c0-996d-7d7129ab1b52-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-4whhd\" (UID: \"5eec5a9b-2547-46c0-996d-7d7129ab1b52\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-4whhd"
Sep 17 18:09:19.169222 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:19.169152    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln6dw\" (UniqueName: \"kubernetes.io/projected/5eec5a9b-2547-46c0-996d-7d7129ab1b52-kube-api-access-ln6dw\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-4whhd\" (UID: \"5eec5a9b-2547-46c0-996d-7d7129ab1b52\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-4whhd"
Sep 17 18:09:19.270198 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:19.270172    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln6dw\" (UniqueName: \"kubernetes.io/projected/5eec5a9b-2547-46c0-996d-7d7129ab1b52-kube-api-access-ln6dw\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-4whhd\" (UID: \"5eec5a9b-2547-46c0-996d-7d7129ab1b52\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-4whhd"
Sep 17 18:09:19.270314 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:19.270213    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5eec5a9b-2547-46c0-996d-7d7129ab1b52-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-4whhd\" (UID: \"5eec5a9b-2547-46c0-996d-7d7129ab1b52\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-4whhd"
Sep 17 18:09:19.270356 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:19.270318    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5eec5a9b-2547-46c0-996d-7d7129ab1b52-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-4whhd\" (UID: \"5eec5a9b-2547-46c0-996d-7d7129ab1b52\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-4whhd"
Sep 17 18:09:19.270400 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:19.270369    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5eec5a9b-2547-46c0-996d-7d7129ab1b52-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-4whhd\" (UID: \"5eec5a9b-2547-46c0-996d-7d7129ab1b52\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-4whhd"
Sep 17 18:09:19.270440 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:19.270398    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5eec5a9b-2547-46c0-996d-7d7129ab1b52-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-4whhd\" (UID: \"5eec5a9b-2547-46c0-996d-7d7129ab1b52\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-4whhd"
Sep 17 18:09:19.270735 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:19.270720    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5eec5a9b-2547-46c0-996d-7d7129ab1b52-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-4whhd\" (UID: \"5eec5a9b-2547-46c0-996d-7d7129ab1b52\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-4whhd"
Sep 17 18:09:19.270878 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:19.270856    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5eec5a9b-2547-46c0-996d-7d7129ab1b52-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-4whhd\" (UID: \"5eec5a9b-2547-46c0-996d-7d7129ab1b52\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-4whhd"
Sep 17 18:09:19.270969 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:19.270894    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5eec5a9b-2547-46c0-996d-7d7129ab1b52-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-4whhd\" (UID: \"5eec5a9b-2547-46c0-996d-7d7129ab1b52\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-4whhd"
Sep 17 18:09:19.272799 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:19.272778    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5eec5a9b-2547-46c0-996d-7d7129ab1b52-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-4whhd\" (UID: \"5eec5a9b-2547-46c0-996d-7d7129ab1b52\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-4whhd"
Sep 17 18:09:19.280144 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:19.280122    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln6dw\" (UniqueName: \"kubernetes.io/projected/5eec5a9b-2547-46c0-996d-7d7129ab1b52-kube-api-access-ln6dw\") pod \"isvc-canary-rollback-predictor-6b8c75c7b9-4whhd\" (UID: \"5eec5a9b-2547-46c0-996d-7d7129ab1b52\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-4whhd"
Sep 17 18:09:19.446533 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:19.446430    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-4whhd"
Sep 17 18:09:19.667333 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:19.667293    2585 generic.go:356] "Generic (PLEG): container finished" podID="6f48bdd6-9b9d-448a-b625-fecb314969e6" containerID="dfeebeaed70e91cfd3258618d265419edb7f28887919b56f1f3137006ffd375f" exitCode=2
Sep 17 18:09:19.667513 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:19.667357    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c" event={"ID":"6f48bdd6-9b9d-448a-b625-fecb314969e6","Type":"ContainerDied","Data":"dfeebeaed70e91cfd3258618d265419edb7f28887919b56f1f3137006ffd375f"}
Sep 17 18:09:19.734319 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:19.734296    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-4whhd"]
Sep 17 18:09:19.735811 ip-10-0-133-139 kubenswrapper[2585]: W0917 18:09:19.735788    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eec5a9b_2547_46c0_996d_7d7129ab1b52.slice/crio-5654312fb03453f63fe897493204711b251dcee95b297e897740a6b0cc296ce3 WatchSource:0}: Error finding container 5654312fb03453f63fe897493204711b251dcee95b297e897740a6b0cc296ce3: Status 404 returned error can't find the container with id 5654312fb03453f63fe897493204711b251dcee95b297e897740a6b0cc296ce3
Sep 17 18:09:20.673040 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:20.672999    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-4whhd" event={"ID":"5eec5a9b-2547-46c0-996d-7d7129ab1b52","Type":"ContainerStarted","Data":"f98fa40402a7c7225182147f2baa8eeb0bba8549f6d7a705c609354264ffdca0"}
Sep 17 18:09:20.673467 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:20.673089    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-4whhd" event={"ID":"5eec5a9b-2547-46c0-996d-7d7129ab1b52","Type":"ContainerStarted","Data":"5654312fb03453f63fe897493204711b251dcee95b297e897740a6b0cc296ce3"}
Sep 17 18:09:22.342541 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:22.342466    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c" podUID="6f48bdd6-9b9d-448a-b625-fecb314969e6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Sep 17 18:09:22.346833 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:22.346805    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c" podUID="6f48bdd6-9b9d-448a-b625-fecb314969e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 18:09:22.759236 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:22.759206    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7"]
Sep 17 18:09:22.759472 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:22.759445    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7" podUID="536c3c62-e109-41c3-8b42-ec33ca5f524b" containerName="kserve-container" containerID="cri-o://51e8cf1333455424b069770eb8689c2ad3649ab3a93dcd3ff9abeec874725f37" gracePeriod=30
Sep 17 18:09:22.759629 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:22.759485    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7" podUID="536c3c62-e109-41c3-8b42-ec33ca5f524b" containerName="kube-rbac-proxy" containerID="cri-o://f48b1bc5c1dd0dc5a0c3d9fe65ce5927baaad267b51deb64b9d95527dc29bf59" gracePeriod=30
Sep 17 18:09:22.793898 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:22.793875    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:09:23.686990 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:23.686954    2585 generic.go:356] "Generic (PLEG): container finished" podID="536c3c62-e109-41c3-8b42-ec33ca5f524b" containerID="f48b1bc5c1dd0dc5a0c3d9fe65ce5927baaad267b51deb64b9d95527dc29bf59" exitCode=2
Sep 17 18:09:23.687367 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:23.687003    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7" event={"ID":"536c3c62-e109-41c3-8b42-ec33ca5f524b","Type":"ContainerDied","Data":"f48b1bc5c1dd0dc5a0c3d9fe65ce5927baaad267b51deb64b9d95527dc29bf59"}
Sep 17 18:09:23.953293 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:23.953246    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c"
Sep 17 18:09:24.007152 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:24.007099    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f48bdd6-9b9d-448a-b625-fecb314969e6-proxy-tls\") pod \"6f48bdd6-9b9d-448a-b625-fecb314969e6\" (UID: \"6f48bdd6-9b9d-448a-b625-fecb314969e6\") "
Sep 17 18:09:24.007275 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:24.007170    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f48bdd6-9b9d-448a-b625-fecb314969e6-cabundle-cert\") pod \"6f48bdd6-9b9d-448a-b625-fecb314969e6\" (UID: \"6f48bdd6-9b9d-448a-b625-fecb314969e6\") "
Sep 17 18:09:24.007275 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:24.007262    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f48bdd6-9b9d-448a-b625-fecb314969e6-kube-rbac-proxy-sar-config\") pod \"6f48bdd6-9b9d-448a-b625-fecb314969e6\" (UID: \"6f48bdd6-9b9d-448a-b625-fecb314969e6\") "
Sep 17 18:09:24.007375 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:24.007317    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f48bdd6-9b9d-448a-b625-fecb314969e6-kserve-provision-location\") pod \"6f48bdd6-9b9d-448a-b625-fecb314969e6\" (UID: \"6f48bdd6-9b9d-448a-b625-fecb314969e6\") "
Sep 17 18:09:24.007411 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:24.007387    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l5tvn\" (UniqueName: \"kubernetes.io/projected/6f48bdd6-9b9d-448a-b625-fecb314969e6-kube-api-access-l5tvn\") pod \"6f48bdd6-9b9d-448a-b625-fecb314969e6\" (UID: \"6f48bdd6-9b9d-448a-b625-fecb314969e6\") "
Sep 17 18:09:24.007549 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:24.007524    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f48bdd6-9b9d-448a-b625-fecb314969e6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6f48bdd6-9b9d-448a-b625-fecb314969e6" (UID: "6f48bdd6-9b9d-448a-b625-fecb314969e6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:09:24.007683 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:24.007618    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f48bdd6-9b9d-448a-b625-fecb314969e6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6f48bdd6-9b9d-448a-b625-fecb314969e6" (UID: "6f48bdd6-9b9d-448a-b625-fecb314969e6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:09:24.007683 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:24.007632    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f48bdd6-9b9d-448a-b625-fecb314969e6-cabundle-cert\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:09:24.007683 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:24.007669    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f48bdd6-9b9d-448a-b625-fecb314969e6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6f48bdd6-9b9d-448a-b625-fecb314969e6" (UID: "6f48bdd6-9b9d-448a-b625-fecb314969e6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:09:24.009211 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:24.009194    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f48bdd6-9b9d-448a-b625-fecb314969e6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6f48bdd6-9b9d-448a-b625-fecb314969e6" (UID: "6f48bdd6-9b9d-448a-b625-fecb314969e6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:09:24.009385 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:24.009362    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f48bdd6-9b9d-448a-b625-fecb314969e6-kube-api-access-l5tvn" (OuterVolumeSpecName: "kube-api-access-l5tvn") pod "6f48bdd6-9b9d-448a-b625-fecb314969e6" (UID: "6f48bdd6-9b9d-448a-b625-fecb314969e6"). InnerVolumeSpecName "kube-api-access-l5tvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:09:24.107972 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:24.107941    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f48bdd6-9b9d-448a-b625-fecb314969e6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:09:24.107972 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:24.107962    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f48bdd6-9b9d-448a-b625-fecb314969e6-kserve-provision-location\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:09:24.107972 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:24.107972    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5tvn\" (UniqueName: \"kubernetes.io/projected/6f48bdd6-9b9d-448a-b625-fecb314969e6-kube-api-access-l5tvn\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:09:24.107972 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:24.107981    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f48bdd6-9b9d-448a-b625-fecb314969e6-proxy-tls\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:09:24.691977 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:24.691940    2585 generic.go:356] "Generic (PLEG): container finished" podID="6f48bdd6-9b9d-448a-b625-fecb314969e6" containerID="cc8a6c7b77d1ca427403ba315352684c02d18a9355f2e58e7e9976ced4354176" exitCode=0
Sep 17 18:09:24.692540 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:24.692001    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c"
Sep 17 18:09:24.692540 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:24.692025    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c" event={"ID":"6f48bdd6-9b9d-448a-b625-fecb314969e6","Type":"ContainerDied","Data":"cc8a6c7b77d1ca427403ba315352684c02d18a9355f2e58e7e9976ced4354176"}
Sep 17 18:09:24.692540 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:24.692105    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c" event={"ID":"6f48bdd6-9b9d-448a-b625-fecb314969e6","Type":"ContainerDied","Data":"d7f04c36fab70d9d84ab8d80762bdb19ae2852a69f88f454070839cbf6b1e835"}
Sep 17 18:09:24.692540 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:24.692165    2585 scope.go:117] "RemoveContainer" containerID="dfeebeaed70e91cfd3258618d265419edb7f28887919b56f1f3137006ffd375f"
Sep 17 18:09:24.693858 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:24.693840    2585 generic.go:356] "Generic (PLEG): container finished" podID="5eec5a9b-2547-46c0-996d-7d7129ab1b52" containerID="f98fa40402a7c7225182147f2baa8eeb0bba8549f6d7a705c609354264ffdca0" exitCode=0
Sep 17 18:09:24.693969 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:24.693891    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-4whhd" event={"ID":"5eec5a9b-2547-46c0-996d-7d7129ab1b52","Type":"ContainerDied","Data":"f98fa40402a7c7225182147f2baa8eeb0bba8549f6d7a705c609354264ffdca0"}
Sep 17 18:09:24.700262 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:24.700246    2585 scope.go:117] "RemoveContainer" containerID="cc8a6c7b77d1ca427403ba315352684c02d18a9355f2e58e7e9976ced4354176"
Sep 17 18:09:24.707379 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:24.707363    2585 scope.go:117] "RemoveContainer" containerID="ed353541f19a81d70cfae54e241fdf9c16e9f7e254debfd7d890302ad25b4aea"
Sep 17 18:09:24.719311 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:24.717174    2585 scope.go:117] "RemoveContainer" containerID="dfeebeaed70e91cfd3258618d265419edb7f28887919b56f1f3137006ffd375f"
Sep 17 18:09:24.719473 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:09:24.719450    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfeebeaed70e91cfd3258618d265419edb7f28887919b56f1f3137006ffd375f\": container with ID starting with dfeebeaed70e91cfd3258618d265419edb7f28887919b56f1f3137006ffd375f not found: ID does not exist" containerID="dfeebeaed70e91cfd3258618d265419edb7f28887919b56f1f3137006ffd375f"
Sep 17 18:09:24.719573 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:24.719480    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfeebeaed70e91cfd3258618d265419edb7f28887919b56f1f3137006ffd375f"} err="failed to get container status \"dfeebeaed70e91cfd3258618d265419edb7f28887919b56f1f3137006ffd375f\": rpc error: code = NotFound desc = could not find container \"dfeebeaed70e91cfd3258618d265419edb7f28887919b56f1f3137006ffd375f\": container with ID starting with dfeebeaed70e91cfd3258618d265419edb7f28887919b56f1f3137006ffd375f not found: ID does not exist"
Sep 17 18:09:24.719658 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:24.719520    2585 scope.go:117] "RemoveContainer" containerID="cc8a6c7b77d1ca427403ba315352684c02d18a9355f2e58e7e9976ced4354176"
Sep 17 18:09:24.719913 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:09:24.719892    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc8a6c7b77d1ca427403ba315352684c02d18a9355f2e58e7e9976ced4354176\": container with ID starting with cc8a6c7b77d1ca427403ba315352684c02d18a9355f2e58e7e9976ced4354176 not found: ID does not exist" containerID="cc8a6c7b77d1ca427403ba315352684c02d18a9355f2e58e7e9976ced4354176"
Sep 17 18:09:24.719968 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:24.719919    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc8a6c7b77d1ca427403ba315352684c02d18a9355f2e58e7e9976ced4354176"} err="failed to get container status \"cc8a6c7b77d1ca427403ba315352684c02d18a9355f2e58e7e9976ced4354176\": rpc error: code = NotFound desc = could not find container \"cc8a6c7b77d1ca427403ba315352684c02d18a9355f2e58e7e9976ced4354176\": container with ID starting with cc8a6c7b77d1ca427403ba315352684c02d18a9355f2e58e7e9976ced4354176 not found: ID does not exist"
Sep 17 18:09:24.719968 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:24.719937    2585 scope.go:117] "RemoveContainer" containerID="ed353541f19a81d70cfae54e241fdf9c16e9f7e254debfd7d890302ad25b4aea"
Sep 17 18:09:24.720243 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:09:24.720220    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed353541f19a81d70cfae54e241fdf9c16e9f7e254debfd7d890302ad25b4aea\": container with ID starting with ed353541f19a81d70cfae54e241fdf9c16e9f7e254debfd7d890302ad25b4aea not found: ID does not exist" containerID="ed353541f19a81d70cfae54e241fdf9c16e9f7e254debfd7d890302ad25b4aea"
Sep 17 18:09:24.720329 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:24.720253    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed353541f19a81d70cfae54e241fdf9c16e9f7e254debfd7d890302ad25b4aea"} err="failed to get container status \"ed353541f19a81d70cfae54e241fdf9c16e9f7e254debfd7d890302ad25b4aea\": rpc error: code = NotFound desc = could not find container \"ed353541f19a81d70cfae54e241fdf9c16e9f7e254debfd7d890302ad25b4aea\": container with ID starting with ed353541f19a81d70cfae54e241fdf9c16e9f7e254debfd7d890302ad25b4aea not found: ID does not exist"
Sep 17 18:09:24.734953 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:24.734931    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c"]
Sep 17 18:09:24.738909 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:24.738887    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7b8b964c7b-dt85c"]
Sep 17 18:09:25.699513 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:25.699462    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-4whhd" event={"ID":"5eec5a9b-2547-46c0-996d-7d7129ab1b52","Type":"ContainerStarted","Data":"e8ab983b1c8f5d6460ea5bfd0235d3df876d4931d95fc8f46f720a8dc321e3d1"}
Sep 17 18:09:25.699970 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:25.699584    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-4whhd" event={"ID":"5eec5a9b-2547-46c0-996d-7d7129ab1b52","Type":"ContainerStarted","Data":"8ea039dfacd1f265574d6a00beeb7e270a489d8c5aeb05767229fbdf1ee22dcc"}
Sep 17 18:09:25.699970 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:25.699803    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-4whhd"
Sep 17 18:09:25.730004 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:25.729958    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-4whhd" podStartSLOduration=2.623521113 podStartE2EDuration="6.729943388s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.106422275s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:09:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:09:25.729221877 +0000 UTC m=+2102.143012332" watchObservedRunningTime="2026-09-17 18:09:25.729943388 +0000 UTC m=+2102.143733872"
Sep 17 18:09:26.104870 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:26.104831    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f48bdd6-9b9d-448a-b625-fecb314969e6" path="/var/lib/kubelet/pods/6f48bdd6-9b9d-448a-b625-fecb314969e6/volumes"
Sep 17 18:09:26.705512 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:26.705469    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-4whhd"
Sep 17 18:09:26.706789 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:26.706764    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-4whhd" podUID="5eec5a9b-2547-46c0-996d-7d7129ab1b52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 18:09:27.560584 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:27.560542    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7" podUID="536c3c62-e109-41c3-8b42-ec33ca5f524b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Sep 17 18:09:27.708976 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:27.708942    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-4whhd" podUID="5eec5a9b-2547-46c0-996d-7d7129ab1b52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 18:09:27.954084 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:27.953999    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:09:28.362557 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:28.362523    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:09:28.444952 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:28.444929    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rrf79_beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5/ovn-acl-logging/0.log"
Sep 17 18:09:28.446849 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:28.446828    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rrf79_beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5/ovn-acl-logging/0.log"
Sep 17 18:09:32.560860 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:32.560815    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7" podUID="536c3c62-e109-41c3-8b42-ec33ca5f524b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Sep 17 18:09:32.714146 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:32.714118    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-4whhd"
Sep 17 18:09:32.714654 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:32.714623    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-4whhd" podUID="5eec5a9b-2547-46c0-996d-7d7129ab1b52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 18:09:37.560765 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:37.560721    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7" podUID="536c3c62-e109-41c3-8b42-ec33ca5f524b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Sep 17 18:09:37.561204 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:37.560843    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7"
Sep 17 18:09:42.560057 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:42.560014    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7" podUID="536c3c62-e109-41c3-8b42-ec33ca5f524b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Sep 17 18:09:42.714854 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:42.714816    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-4whhd" podUID="5eec5a9b-2547-46c0-996d-7d7129ab1b52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 18:09:47.561362 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:47.559868    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7" podUID="536c3c62-e109-41c3-8b42-ec33ca5f524b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Sep 17 18:09:52.560443 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:52.560398    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7" podUID="536c3c62-e109-41c3-8b42-ec33ca5f524b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Sep 17 18:09:52.714983 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:52.714946    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-4whhd" podUID="5eec5a9b-2547-46c0-996d-7d7129ab1b52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 18:09:53.395434 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:53.395409    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7"
Sep 17 18:09:53.521441 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:53.521405    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-txt9l\" (UniqueName: \"kubernetes.io/projected/536c3c62-e109-41c3-8b42-ec33ca5f524b-kube-api-access-txt9l\") pod \"536c3c62-e109-41c3-8b42-ec33ca5f524b\" (UID: \"536c3c62-e109-41c3-8b42-ec33ca5f524b\") "
Sep 17 18:09:53.521639 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:53.521457    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/536c3c62-e109-41c3-8b42-ec33ca5f524b-kserve-provision-location\") pod \"536c3c62-e109-41c3-8b42-ec33ca5f524b\" (UID: \"536c3c62-e109-41c3-8b42-ec33ca5f524b\") "
Sep 17 18:09:53.521639 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:53.521517    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/536c3c62-e109-41c3-8b42-ec33ca5f524b-kube-rbac-proxy-sar-config\") pod \"536c3c62-e109-41c3-8b42-ec33ca5f524b\" (UID: \"536c3c62-e109-41c3-8b42-ec33ca5f524b\") "
Sep 17 18:09:53.521639 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:53.521542    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/536c3c62-e109-41c3-8b42-ec33ca5f524b-cabundle-cert\") pod \"536c3c62-e109-41c3-8b42-ec33ca5f524b\" (UID: \"536c3c62-e109-41c3-8b42-ec33ca5f524b\") "
Sep 17 18:09:53.521639 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:53.521570    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/536c3c62-e109-41c3-8b42-ec33ca5f524b-proxy-tls\") pod \"536c3c62-e109-41c3-8b42-ec33ca5f524b\" (UID: \"536c3c62-e109-41c3-8b42-ec33ca5f524b\") "
Sep 17 18:09:53.521918 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:53.521883    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/536c3c62-e109-41c3-8b42-ec33ca5f524b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "536c3c62-e109-41c3-8b42-ec33ca5f524b" (UID: "536c3c62-e109-41c3-8b42-ec33ca5f524b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:09:53.522056 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:53.521886    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/536c3c62-e109-41c3-8b42-ec33ca5f524b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "536c3c62-e109-41c3-8b42-ec33ca5f524b" (UID: "536c3c62-e109-41c3-8b42-ec33ca5f524b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:09:53.523570 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:53.523543    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/536c3c62-e109-41c3-8b42-ec33ca5f524b-kube-api-access-txt9l" (OuterVolumeSpecName: "kube-api-access-txt9l") pod "536c3c62-e109-41c3-8b42-ec33ca5f524b" (UID: "536c3c62-e109-41c3-8b42-ec33ca5f524b"). InnerVolumeSpecName "kube-api-access-txt9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:09:53.523681 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:53.523647    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/536c3c62-e109-41c3-8b42-ec33ca5f524b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "536c3c62-e109-41c3-8b42-ec33ca5f524b" (UID: "536c3c62-e109-41c3-8b42-ec33ca5f524b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:09:53.531150 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:53.531120    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/536c3c62-e109-41c3-8b42-ec33ca5f524b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "536c3c62-e109-41c3-8b42-ec33ca5f524b" (UID: "536c3c62-e109-41c3-8b42-ec33ca5f524b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:09:53.622047 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:53.622000    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/536c3c62-e109-41c3-8b42-ec33ca5f524b-kserve-provision-location\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:09:53.622047 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:53.622039    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/536c3c62-e109-41c3-8b42-ec33ca5f524b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:09:53.622047 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:53.622051    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/536c3c62-e109-41c3-8b42-ec33ca5f524b-cabundle-cert\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:09:53.622047 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:53.622062    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/536c3c62-e109-41c3-8b42-ec33ca5f524b-proxy-tls\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:09:53.622539 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:53.622070    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txt9l\" (UniqueName: \"kubernetes.io/projected/536c3c62-e109-41c3-8b42-ec33ca5f524b-kube-api-access-txt9l\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:09:53.807090 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:53.806998    2585 generic.go:356] "Generic (PLEG): container finished" podID="536c3c62-e109-41c3-8b42-ec33ca5f524b" containerID="51e8cf1333455424b069770eb8689c2ad3649ab3a93dcd3ff9abeec874725f37" exitCode=137
Sep 17 18:09:53.807090 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:53.807042    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7" event={"ID":"536c3c62-e109-41c3-8b42-ec33ca5f524b","Type":"ContainerDied","Data":"51e8cf1333455424b069770eb8689c2ad3649ab3a93dcd3ff9abeec874725f37"}
Sep 17 18:09:53.807090 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:53.807066    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7"
Sep 17 18:09:53.807383 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:53.807112    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7" event={"ID":"536c3c62-e109-41c3-8b42-ec33ca5f524b","Type":"ContainerDied","Data":"d6b4cf9a2601d79b2d115c298096f5993ef15fbbd3672e7be50f4e84508b1170"}
Sep 17 18:09:53.807383 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:53.807139    2585 scope.go:117] "RemoveContainer" containerID="f48b1bc5c1dd0dc5a0c3d9fe65ce5927baaad267b51deb64b9d95527dc29bf59"
Sep 17 18:09:53.815724 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:53.815708    2585 scope.go:117] "RemoveContainer" containerID="51e8cf1333455424b069770eb8689c2ad3649ab3a93dcd3ff9abeec874725f37"
Sep 17 18:09:53.822928 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:53.822907    2585 scope.go:117] "RemoveContainer" containerID="1bcc43839e4c12fa8a99ee921b61ecee37181efa124ceb013ce26221e72a563d"
Sep 17 18:09:53.829707 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:53.829689    2585 scope.go:117] "RemoveContainer" containerID="f48b1bc5c1dd0dc5a0c3d9fe65ce5927baaad267b51deb64b9d95527dc29bf59"
Sep 17 18:09:53.829953 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:09:53.829933    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f48b1bc5c1dd0dc5a0c3d9fe65ce5927baaad267b51deb64b9d95527dc29bf59\": container with ID starting with f48b1bc5c1dd0dc5a0c3d9fe65ce5927baaad267b51deb64b9d95527dc29bf59 not found: ID does not exist" containerID="f48b1bc5c1dd0dc5a0c3d9fe65ce5927baaad267b51deb64b9d95527dc29bf59"
Sep 17 18:09:53.830027 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:53.829966    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f48b1bc5c1dd0dc5a0c3d9fe65ce5927baaad267b51deb64b9d95527dc29bf59"} err="failed to get container status \"f48b1bc5c1dd0dc5a0c3d9fe65ce5927baaad267b51deb64b9d95527dc29bf59\": rpc error: code = NotFound desc = could not find container \"f48b1bc5c1dd0dc5a0c3d9fe65ce5927baaad267b51deb64b9d95527dc29bf59\": container with ID starting with f48b1bc5c1dd0dc5a0c3d9fe65ce5927baaad267b51deb64b9d95527dc29bf59 not found: ID does not exist"
Sep 17 18:09:53.830027 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:53.829991    2585 scope.go:117] "RemoveContainer" containerID="51e8cf1333455424b069770eb8689c2ad3649ab3a93dcd3ff9abeec874725f37"
Sep 17 18:09:53.830258 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:09:53.830241    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51e8cf1333455424b069770eb8689c2ad3649ab3a93dcd3ff9abeec874725f37\": container with ID starting with 51e8cf1333455424b069770eb8689c2ad3649ab3a93dcd3ff9abeec874725f37 not found: ID does not exist" containerID="51e8cf1333455424b069770eb8689c2ad3649ab3a93dcd3ff9abeec874725f37"
Sep 17 18:09:53.830297 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:53.830264    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51e8cf1333455424b069770eb8689c2ad3649ab3a93dcd3ff9abeec874725f37"} err="failed to get container status \"51e8cf1333455424b069770eb8689c2ad3649ab3a93dcd3ff9abeec874725f37\": rpc error: code = NotFound desc = could not find container \"51e8cf1333455424b069770eb8689c2ad3649ab3a93dcd3ff9abeec874725f37\": container with ID starting with 51e8cf1333455424b069770eb8689c2ad3649ab3a93dcd3ff9abeec874725f37 not found: ID does not exist"
Sep 17 18:09:53.830297 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:53.830280    2585 scope.go:117] "RemoveContainer" containerID="1bcc43839e4c12fa8a99ee921b61ecee37181efa124ceb013ce26221e72a563d"
Sep 17 18:09:53.830517 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:09:53.830484    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bcc43839e4c12fa8a99ee921b61ecee37181efa124ceb013ce26221e72a563d\": container with ID starting with 1bcc43839e4c12fa8a99ee921b61ecee37181efa124ceb013ce26221e72a563d not found: ID does not exist" containerID="1bcc43839e4c12fa8a99ee921b61ecee37181efa124ceb013ce26221e72a563d"
Sep 17 18:09:53.830577 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:53.830524    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bcc43839e4c12fa8a99ee921b61ecee37181efa124ceb013ce26221e72a563d"} err="failed to get container status \"1bcc43839e4c12fa8a99ee921b61ecee37181efa124ceb013ce26221e72a563d\": rpc error: code = NotFound desc = could not find container \"1bcc43839e4c12fa8a99ee921b61ecee37181efa124ceb013ce26221e72a563d\": container with ID starting with 1bcc43839e4c12fa8a99ee921b61ecee37181efa124ceb013ce26221e72a563d not found: ID does not exist"
Sep 17 18:09:53.839509 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:53.839467    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7"]
Sep 17 18:09:53.844865 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:53.844845    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-qg5l7"]
Sep 17 18:09:54.105719 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:09:54.105636    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="536c3c62-e109-41c3-8b42-ec33ca5f524b" path="/var/lib/kubelet/pods/536c3c62-e109-41c3-8b42-ec33ca5f524b/volumes"
Sep 17 18:10:02.714612 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:02.714573    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-4whhd" podUID="5eec5a9b-2547-46c0-996d-7d7129ab1b52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 18:10:12.715155 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:12.715107    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-4whhd" podUID="5eec5a9b-2547-46c0-996d-7d7129ab1b52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 18:10:22.715056 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:22.714977    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-4whhd" podUID="5eec5a9b-2547-46c0-996d-7d7129ab1b52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 18:10:32.715065 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:32.715026    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-4whhd" podUID="5eec5a9b-2547-46c0-996d-7d7129ab1b52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 18:10:42.715282 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:42.715254    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-4whhd"
Sep 17 18:10:49.209778 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:49.209742    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-4whhd"]
Sep 17 18:10:49.210274 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:49.210020    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-4whhd" podUID="5eec5a9b-2547-46c0-996d-7d7129ab1b52" containerName="kserve-container" containerID="cri-o://8ea039dfacd1f265574d6a00beeb7e270a489d8c5aeb05767229fbdf1ee22dcc" gracePeriod=30
Sep 17 18:10:49.210274 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:49.210075    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-4whhd" podUID="5eec5a9b-2547-46c0-996d-7d7129ab1b52" containerName="kube-rbac-proxy" containerID="cri-o://e8ab983b1c8f5d6460ea5bfd0235d3df876d4931d95fc8f46f720a8dc321e3d1" gracePeriod=30
Sep 17 18:10:49.236716 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:49.236684    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:50.007294 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:50.007254    2585 generic.go:356] "Generic (PLEG): container finished" podID="5eec5a9b-2547-46c0-996d-7d7129ab1b52" containerID="e8ab983b1c8f5d6460ea5bfd0235d3df876d4931d95fc8f46f720a8dc321e3d1" exitCode=2
Sep 17 18:10:50.007466 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:50.007330    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-4whhd" event={"ID":"5eec5a9b-2547-46c0-996d-7d7129ab1b52","Type":"ContainerDied","Data":"e8ab983b1c8f5d6460ea5bfd0235d3df876d4931d95fc8f46f720a8dc321e3d1"}
Sep 17 18:10:52.710037 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:52.709992    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-4whhd" podUID="5eec5a9b-2547-46c0-996d-7d7129ab1b52" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Sep 17 18:10:52.715014 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:52.714991    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-4whhd" podUID="5eec5a9b-2547-46c0-996d-7d7129ab1b52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 18:10:53.545574 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:53.545551    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-4whhd"
Sep 17 18:10:53.549690 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:53.549671    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5eec5a9b-2547-46c0-996d-7d7129ab1b52-kserve-provision-location\") pod \"5eec5a9b-2547-46c0-996d-7d7129ab1b52\" (UID: \"5eec5a9b-2547-46c0-996d-7d7129ab1b52\") "
Sep 17 18:10:53.549800 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:53.549710    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5eec5a9b-2547-46c0-996d-7d7129ab1b52-cabundle-cert\") pod \"5eec5a9b-2547-46c0-996d-7d7129ab1b52\" (UID: \"5eec5a9b-2547-46c0-996d-7d7129ab1b52\") "
Sep 17 18:10:53.549800 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:53.549729    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5eec5a9b-2547-46c0-996d-7d7129ab1b52-proxy-tls\") pod \"5eec5a9b-2547-46c0-996d-7d7129ab1b52\" (UID: \"5eec5a9b-2547-46c0-996d-7d7129ab1b52\") "
Sep 17 18:10:53.549979 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:53.549957    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eec5a9b-2547-46c0-996d-7d7129ab1b52-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5eec5a9b-2547-46c0-996d-7d7129ab1b52" (UID: "5eec5a9b-2547-46c0-996d-7d7129ab1b52"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:10:53.550045 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:53.550008    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eec5a9b-2547-46c0-996d-7d7129ab1b52-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5eec5a9b-2547-46c0-996d-7d7129ab1b52" (UID: "5eec5a9b-2547-46c0-996d-7d7129ab1b52"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:10:53.551693 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:53.551669    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5eec5a9b-2547-46c0-996d-7d7129ab1b52-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5eec5a9b-2547-46c0-996d-7d7129ab1b52" (UID: "5eec5a9b-2547-46c0-996d-7d7129ab1b52"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:10:53.650483 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:53.650448    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5eec5a9b-2547-46c0-996d-7d7129ab1b52-kube-rbac-proxy-sar-config\") pod \"5eec5a9b-2547-46c0-996d-7d7129ab1b52\" (UID: \"5eec5a9b-2547-46c0-996d-7d7129ab1b52\") "
Sep 17 18:10:53.650689 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:53.650511    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ln6dw\" (UniqueName: \"kubernetes.io/projected/5eec5a9b-2547-46c0-996d-7d7129ab1b52-kube-api-access-ln6dw\") pod \"5eec5a9b-2547-46c0-996d-7d7129ab1b52\" (UID: \"5eec5a9b-2547-46c0-996d-7d7129ab1b52\") "
Sep 17 18:10:53.650757 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:53.650699    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5eec5a9b-2547-46c0-996d-7d7129ab1b52-kserve-provision-location\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:10:53.650757 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:53.650716    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5eec5a9b-2547-46c0-996d-7d7129ab1b52-cabundle-cert\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:10:53.650757 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:53.650731    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5eec5a9b-2547-46c0-996d-7d7129ab1b52-proxy-tls\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:10:53.650869 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:53.650799    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eec5a9b-2547-46c0-996d-7d7129ab1b52-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5eec5a9b-2547-46c0-996d-7d7129ab1b52" (UID: "5eec5a9b-2547-46c0-996d-7d7129ab1b52"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:10:53.652586 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:53.652563    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eec5a9b-2547-46c0-996d-7d7129ab1b52-kube-api-access-ln6dw" (OuterVolumeSpecName: "kube-api-access-ln6dw") pod "5eec5a9b-2547-46c0-996d-7d7129ab1b52" (UID: "5eec5a9b-2547-46c0-996d-7d7129ab1b52"). InnerVolumeSpecName "kube-api-access-ln6dw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:10:53.751720 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:53.751694    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5eec5a9b-2547-46c0-996d-7d7129ab1b52-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:10:53.751720 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:53.751718    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ln6dw\" (UniqueName: \"kubernetes.io/projected/5eec5a9b-2547-46c0-996d-7d7129ab1b52-kube-api-access-ln6dw\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:10:54.023796 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.023708    2585 generic.go:356] "Generic (PLEG): container finished" podID="5eec5a9b-2547-46c0-996d-7d7129ab1b52" containerID="8ea039dfacd1f265574d6a00beeb7e270a489d8c5aeb05767229fbdf1ee22dcc" exitCode=0
Sep 17 18:10:54.023796 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.023748    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-4whhd" event={"ID":"5eec5a9b-2547-46c0-996d-7d7129ab1b52","Type":"ContainerDied","Data":"8ea039dfacd1f265574d6a00beeb7e270a489d8c5aeb05767229fbdf1ee22dcc"}
Sep 17 18:10:54.023796 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.023762    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-4whhd"
Sep 17 18:10:54.024051 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.023808    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-4whhd" event={"ID":"5eec5a9b-2547-46c0-996d-7d7129ab1b52","Type":"ContainerDied","Data":"5654312fb03453f63fe897493204711b251dcee95b297e897740a6b0cc296ce3"}
Sep 17 18:10:54.024051 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.023842    2585 scope.go:117] "RemoveContainer" containerID="e8ab983b1c8f5d6460ea5bfd0235d3df876d4931d95fc8f46f720a8dc321e3d1"
Sep 17 18:10:54.031727 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.031528    2585 scope.go:117] "RemoveContainer" containerID="8ea039dfacd1f265574d6a00beeb7e270a489d8c5aeb05767229fbdf1ee22dcc"
Sep 17 18:10:54.038258 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.038243    2585 scope.go:117] "RemoveContainer" containerID="f98fa40402a7c7225182147f2baa8eeb0bba8549f6d7a705c609354264ffdca0"
Sep 17 18:10:54.045024 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.044731    2585 scope.go:117] "RemoveContainer" containerID="e8ab983b1c8f5d6460ea5bfd0235d3df876d4931d95fc8f46f720a8dc321e3d1"
Sep 17 18:10:54.045188 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:10:54.045164    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8ab983b1c8f5d6460ea5bfd0235d3df876d4931d95fc8f46f720a8dc321e3d1\": container with ID starting with e8ab983b1c8f5d6460ea5bfd0235d3df876d4931d95fc8f46f720a8dc321e3d1 not found: ID does not exist" containerID="e8ab983b1c8f5d6460ea5bfd0235d3df876d4931d95fc8f46f720a8dc321e3d1"
Sep 17 18:10:54.045235 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.045199    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8ab983b1c8f5d6460ea5bfd0235d3df876d4931d95fc8f46f720a8dc321e3d1"} err="failed to get container status \"e8ab983b1c8f5d6460ea5bfd0235d3df876d4931d95fc8f46f720a8dc321e3d1\": rpc error: code = NotFound desc = could not find container \"e8ab983b1c8f5d6460ea5bfd0235d3df876d4931d95fc8f46f720a8dc321e3d1\": container with ID starting with e8ab983b1c8f5d6460ea5bfd0235d3df876d4931d95fc8f46f720a8dc321e3d1 not found: ID does not exist"
Sep 17 18:10:54.045235 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.045223    2585 scope.go:117] "RemoveContainer" containerID="8ea039dfacd1f265574d6a00beeb7e270a489d8c5aeb05767229fbdf1ee22dcc"
Sep 17 18:10:54.045475 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:10:54.045459    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ea039dfacd1f265574d6a00beeb7e270a489d8c5aeb05767229fbdf1ee22dcc\": container with ID starting with 8ea039dfacd1f265574d6a00beeb7e270a489d8c5aeb05767229fbdf1ee22dcc not found: ID does not exist" containerID="8ea039dfacd1f265574d6a00beeb7e270a489d8c5aeb05767229fbdf1ee22dcc"
Sep 17 18:10:54.045579 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.045481    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ea039dfacd1f265574d6a00beeb7e270a489d8c5aeb05767229fbdf1ee22dcc"} err="failed to get container status \"8ea039dfacd1f265574d6a00beeb7e270a489d8c5aeb05767229fbdf1ee22dcc\": rpc error: code = NotFound desc = could not find container \"8ea039dfacd1f265574d6a00beeb7e270a489d8c5aeb05767229fbdf1ee22dcc\": container with ID starting with 8ea039dfacd1f265574d6a00beeb7e270a489d8c5aeb05767229fbdf1ee22dcc not found: ID does not exist"
Sep 17 18:10:54.045579 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.045523    2585 scope.go:117] "RemoveContainer" containerID="f98fa40402a7c7225182147f2baa8eeb0bba8549f6d7a705c609354264ffdca0"
Sep 17 18:10:54.045800 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:10:54.045781    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f98fa40402a7c7225182147f2baa8eeb0bba8549f6d7a705c609354264ffdca0\": container with ID starting with f98fa40402a7c7225182147f2baa8eeb0bba8549f6d7a705c609354264ffdca0 not found: ID does not exist" containerID="f98fa40402a7c7225182147f2baa8eeb0bba8549f6d7a705c609354264ffdca0"
Sep 17 18:10:54.045946 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.045806    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f98fa40402a7c7225182147f2baa8eeb0bba8549f6d7a705c609354264ffdca0"} err="failed to get container status \"f98fa40402a7c7225182147f2baa8eeb0bba8549f6d7a705c609354264ffdca0\": rpc error: code = NotFound desc = could not find container \"f98fa40402a7c7225182147f2baa8eeb0bba8549f6d7a705c609354264ffdca0\": container with ID starting with f98fa40402a7c7225182147f2baa8eeb0bba8549f6d7a705c609354264ffdca0 not found: ID does not exist"
Sep 17 18:10:54.046372 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.046356    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-4whhd"]
Sep 17 18:10:54.050606 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.050586    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b8c75c7b9-4whhd"]
Sep 17 18:10:54.104224 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.104197    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5eec5a9b-2547-46c0-996d-7d7129ab1b52" path="/var/lib/kubelet/pods/5eec5a9b-2547-46c0-996d-7d7129ab1b52/volumes"
Sep 17 18:10:54.387221 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.387142    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:54.437372 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.437340    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-vdf9d"]
Sep 17 18:10:54.441083 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.439094    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="536c3c62-e109-41c3-8b42-ec33ca5f524b" containerName="kserve-container"
Sep 17 18:10:54.441083 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.439122    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="536c3c62-e109-41c3-8b42-ec33ca5f524b" containerName="kserve-container"
Sep 17 18:10:54.441083 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.439154    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5eec5a9b-2547-46c0-996d-7d7129ab1b52" containerName="kserve-container"
Sep 17 18:10:54.441083 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.439163    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="5eec5a9b-2547-46c0-996d-7d7129ab1b52" containerName="kserve-container"
Sep 17 18:10:54.441083 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.439179    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5eec5a9b-2547-46c0-996d-7d7129ab1b52" containerName="kube-rbac-proxy"
Sep 17 18:10:54.441083 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.439187    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="5eec5a9b-2547-46c0-996d-7d7129ab1b52" containerName="kube-rbac-proxy"
Sep 17 18:10:54.441083 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.439204    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="536c3c62-e109-41c3-8b42-ec33ca5f524b" containerName="kube-rbac-proxy"
Sep 17 18:10:54.441083 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.439214    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="536c3c62-e109-41c3-8b42-ec33ca5f524b" containerName="kube-rbac-proxy"
Sep 17 18:10:54.441083 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.439225    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="536c3c62-e109-41c3-8b42-ec33ca5f524b" containerName="storage-initializer"
Sep 17 18:10:54.441083 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.439235    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="536c3c62-e109-41c3-8b42-ec33ca5f524b" containerName="storage-initializer"
Sep 17 18:10:54.441083 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.439252    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f48bdd6-9b9d-448a-b625-fecb314969e6" containerName="kube-rbac-proxy"
Sep 17 18:10:54.441083 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.439260    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f48bdd6-9b9d-448a-b625-fecb314969e6" containerName="kube-rbac-proxy"
Sep 17 18:10:54.441083 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.439285    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f48bdd6-9b9d-448a-b625-fecb314969e6" containerName="storage-initializer"
Sep 17 18:10:54.441083 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.439294    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f48bdd6-9b9d-448a-b625-fecb314969e6" containerName="storage-initializer"
Sep 17 18:10:54.441083 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.439303    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f48bdd6-9b9d-448a-b625-fecb314969e6" containerName="kserve-container"
Sep 17 18:10:54.441083 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.439318    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f48bdd6-9b9d-448a-b625-fecb314969e6" containerName="kserve-container"
Sep 17 18:10:54.441083 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.439335    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5eec5a9b-2547-46c0-996d-7d7129ab1b52" containerName="storage-initializer"
Sep 17 18:10:54.441083 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.439345    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="5eec5a9b-2547-46c0-996d-7d7129ab1b52" containerName="storage-initializer"
Sep 17 18:10:54.441083 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.439450    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="5eec5a9b-2547-46c0-996d-7d7129ab1b52" containerName="kube-rbac-proxy"
Sep 17 18:10:54.441083 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.439466    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="536c3c62-e109-41c3-8b42-ec33ca5f524b" containerName="kube-rbac-proxy"
Sep 17 18:10:54.441083 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.439477    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f48bdd6-9b9d-448a-b625-fecb314969e6" containerName="kube-rbac-proxy"
Sep 17 18:10:54.441083 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.439485    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="536c3c62-e109-41c3-8b42-ec33ca5f524b" containerName="kserve-container"
Sep 17 18:10:54.441083 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.439513    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="5eec5a9b-2547-46c0-996d-7d7129ab1b52" containerName="kserve-container"
Sep 17 18:10:54.441083 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.439522    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f48bdd6-9b9d-448a-b625-fecb314969e6" containerName="kserve-container"
Sep 17 18:10:54.445363 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.445339    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-vdf9d"
Sep 17 18:10:54.452242 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.452221    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:10:54.452357 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.452221    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:10:54.452795 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.452763    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:10:54.452795 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.452774    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:10:54.453167 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.452844    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Sep 17 18:10:54.453517 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.453480    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lps5w\""
Sep 17 18:10:54.453822 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.453718    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 17 18:10:54.455147 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.455110    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-km5ql"]
Sep 17 18:10:54.455624 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.455594    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4567fe39-89aa-494f-a1a6-5e3ea51e0f9f-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-596664c988-vdf9d\" (UID: \"4567fe39-89aa-494f-a1a6-5e3ea51e0f9f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-vdf9d"
Sep 17 18:10:54.455706 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.455643    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6tm6\" (UniqueName: \"kubernetes.io/projected/4567fe39-89aa-494f-a1a6-5e3ea51e0f9f-kube-api-access-x6tm6\") pod \"isvc-canary-stop-v2-predictor-596664c988-vdf9d\" (UID: \"4567fe39-89aa-494f-a1a6-5e3ea51e0f9f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-vdf9d"
Sep 17 18:10:54.455706 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.455686    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4567fe39-89aa-494f-a1a6-5e3ea51e0f9f-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-596664c988-vdf9d\" (UID: \"4567fe39-89aa-494f-a1a6-5e3ea51e0f9f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-vdf9d"
Sep 17 18:10:54.455819 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.455733    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4567fe39-89aa-494f-a1a6-5e3ea51e0f9f-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-596664c988-vdf9d\" (UID: \"4567fe39-89aa-494f-a1a6-5e3ea51e0f9f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-vdf9d"
Sep 17 18:10:54.455819 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.455786    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4567fe39-89aa-494f-a1a6-5e3ea51e0f9f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-596664c988-vdf9d\" (UID: \"4567fe39-89aa-494f-a1a6-5e3ea51e0f9f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-vdf9d"
Sep 17 18:10:54.458670 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.458652    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-vdf9d"]
Sep 17 18:10:54.458761 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.458747    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-km5ql"
Sep 17 18:10:54.460940 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.460921    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 17 18:10:54.474146 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.474122    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-km5ql"]
Sep 17 18:10:54.484243 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.484223    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:54.556047 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.556019    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4567fe39-89aa-494f-a1a6-5e3ea51e0f9f-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-596664c988-vdf9d\" (UID: \"4567fe39-89aa-494f-a1a6-5e3ea51e0f9f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-vdf9d"
Sep 17 18:10:54.556205 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.556055    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4567fe39-89aa-494f-a1a6-5e3ea51e0f9f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-596664c988-vdf9d\" (UID: \"4567fe39-89aa-494f-a1a6-5e3ea51e0f9f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-vdf9d"
Sep 17 18:10:54.556205 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.556089    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77003c43-ce6b-47fc-b59e-ed19e15c9059-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5cdcf6545-km5ql\" (UID: \"77003c43-ce6b-47fc-b59e-ed19e15c9059\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-km5ql"
Sep 17 18:10:54.556205 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.556123    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77003c43-ce6b-47fc-b59e-ed19e15c9059-proxy-tls\") pod \"isvc-canary-stop-predictor-5cdcf6545-km5ql\" (UID: \"77003c43-ce6b-47fc-b59e-ed19e15c9059\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-km5ql"
Sep 17 18:10:54.556205 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.556149    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77003c43-ce6b-47fc-b59e-ed19e15c9059-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5cdcf6545-km5ql\" (UID: \"77003c43-ce6b-47fc-b59e-ed19e15c9059\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-km5ql"
Sep 17 18:10:54.556205 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:10:54.556187    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-serving-cert: secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Sep 17 18:10:54.556445 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.556219    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4567fe39-89aa-494f-a1a6-5e3ea51e0f9f-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-596664c988-vdf9d\" (UID: \"4567fe39-89aa-494f-a1a6-5e3ea51e0f9f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-vdf9d"
Sep 17 18:10:54.556445 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:10:54.556292    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4567fe39-89aa-494f-a1a6-5e3ea51e0f9f-proxy-tls podName:4567fe39-89aa-494f-a1a6-5e3ea51e0f9f nodeName:}" failed. No retries permitted until 2026-09-17 18:10:55.056269014 +0000 UTC m=+2191.470059452 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4567fe39-89aa-494f-a1a6-5e3ea51e0f9f-proxy-tls") pod "isvc-canary-stop-v2-predictor-596664c988-vdf9d" (UID: "4567fe39-89aa-494f-a1a6-5e3ea51e0f9f") : secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Sep 17 18:10:54.556445 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.556341    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6tm6\" (UniqueName: \"kubernetes.io/projected/4567fe39-89aa-494f-a1a6-5e3ea51e0f9f-kube-api-access-x6tm6\") pod \"isvc-canary-stop-v2-predictor-596664c988-vdf9d\" (UID: \"4567fe39-89aa-494f-a1a6-5e3ea51e0f9f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-vdf9d"
Sep 17 18:10:54.556445 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.556373    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g85pm\" (UniqueName: \"kubernetes.io/projected/77003c43-ce6b-47fc-b59e-ed19e15c9059-kube-api-access-g85pm\") pod \"isvc-canary-stop-predictor-5cdcf6545-km5ql\" (UID: \"77003c43-ce6b-47fc-b59e-ed19e15c9059\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-km5ql"
Sep 17 18:10:54.556445 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.556422    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4567fe39-89aa-494f-a1a6-5e3ea51e0f9f-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-596664c988-vdf9d\" (UID: \"4567fe39-89aa-494f-a1a6-5e3ea51e0f9f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-vdf9d"
Sep 17 18:10:54.556722 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.556451    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77003c43-ce6b-47fc-b59e-ed19e15c9059-cabundle-cert\") pod \"isvc-canary-stop-predictor-5cdcf6545-km5ql\" (UID: \"77003c43-ce6b-47fc-b59e-ed19e15c9059\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-km5ql"
Sep 17 18:10:54.556722 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.556598    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4567fe39-89aa-494f-a1a6-5e3ea51e0f9f-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-596664c988-vdf9d\" (UID: \"4567fe39-89aa-494f-a1a6-5e3ea51e0f9f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-vdf9d"
Sep 17 18:10:54.556895 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.556875    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4567fe39-89aa-494f-a1a6-5e3ea51e0f9f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-596664c988-vdf9d\" (UID: \"4567fe39-89aa-494f-a1a6-5e3ea51e0f9f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-vdf9d"
Sep 17 18:10:54.557111 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.557092    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4567fe39-89aa-494f-a1a6-5e3ea51e0f9f-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-596664c988-vdf9d\" (UID: \"4567fe39-89aa-494f-a1a6-5e3ea51e0f9f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-vdf9d"
Sep 17 18:10:54.571593 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.571572    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6tm6\" (UniqueName: \"kubernetes.io/projected/4567fe39-89aa-494f-a1a6-5e3ea51e0f9f-kube-api-access-x6tm6\") pod \"isvc-canary-stop-v2-predictor-596664c988-vdf9d\" (UID: \"4567fe39-89aa-494f-a1a6-5e3ea51e0f9f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-vdf9d"
Sep 17 18:10:54.656904 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.656817    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77003c43-ce6b-47fc-b59e-ed19e15c9059-cabundle-cert\") pod \"isvc-canary-stop-predictor-5cdcf6545-km5ql\" (UID: \"77003c43-ce6b-47fc-b59e-ed19e15c9059\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-km5ql"
Sep 17 18:10:54.656904 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.656880    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77003c43-ce6b-47fc-b59e-ed19e15c9059-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5cdcf6545-km5ql\" (UID: \"77003c43-ce6b-47fc-b59e-ed19e15c9059\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-km5ql"
Sep 17 18:10:54.657116 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.656910    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77003c43-ce6b-47fc-b59e-ed19e15c9059-proxy-tls\") pod \"isvc-canary-stop-predictor-5cdcf6545-km5ql\" (UID: \"77003c43-ce6b-47fc-b59e-ed19e15c9059\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-km5ql"
Sep 17 18:10:54.657116 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.656929    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77003c43-ce6b-47fc-b59e-ed19e15c9059-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5cdcf6545-km5ql\" (UID: \"77003c43-ce6b-47fc-b59e-ed19e15c9059\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-km5ql"
Sep 17 18:10:54.657116 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.656952    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g85pm\" (UniqueName: \"kubernetes.io/projected/77003c43-ce6b-47fc-b59e-ed19e15c9059-kube-api-access-g85pm\") pod \"isvc-canary-stop-predictor-5cdcf6545-km5ql\" (UID: \"77003c43-ce6b-47fc-b59e-ed19e15c9059\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-km5ql"
Sep 17 18:10:54.657116 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:10:54.657052    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-predictor-serving-cert: secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 17 18:10:54.657340 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:10:54.657140    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77003c43-ce6b-47fc-b59e-ed19e15c9059-proxy-tls podName:77003c43-ce6b-47fc-b59e-ed19e15c9059 nodeName:}" failed. No retries permitted until 2026-09-17 18:10:55.157121344 +0000 UTC m=+2191.570911778 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/77003c43-ce6b-47fc-b59e-ed19e15c9059-proxy-tls") pod "isvc-canary-stop-predictor-5cdcf6545-km5ql" (UID: "77003c43-ce6b-47fc-b59e-ed19e15c9059") : secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 17 18:10:54.657434 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.657412    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77003c43-ce6b-47fc-b59e-ed19e15c9059-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5cdcf6545-km5ql\" (UID: \"77003c43-ce6b-47fc-b59e-ed19e15c9059\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-km5ql"
Sep 17 18:10:54.657613 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.657594    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77003c43-ce6b-47fc-b59e-ed19e15c9059-cabundle-cert\") pod \"isvc-canary-stop-predictor-5cdcf6545-km5ql\" (UID: \"77003c43-ce6b-47fc-b59e-ed19e15c9059\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-km5ql"
Sep 17 18:10:54.657665 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.657643    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77003c43-ce6b-47fc-b59e-ed19e15c9059-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5cdcf6545-km5ql\" (UID: \"77003c43-ce6b-47fc-b59e-ed19e15c9059\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-km5ql"
Sep 17 18:10:54.665793 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:54.665764    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g85pm\" (UniqueName: \"kubernetes.io/projected/77003c43-ce6b-47fc-b59e-ed19e15c9059-kube-api-access-g85pm\") pod \"isvc-canary-stop-predictor-5cdcf6545-km5ql\" (UID: \"77003c43-ce6b-47fc-b59e-ed19e15c9059\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-km5ql"
Sep 17 18:10:55.060269 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:55.060230    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4567fe39-89aa-494f-a1a6-5e3ea51e0f9f-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-596664c988-vdf9d\" (UID: \"4567fe39-89aa-494f-a1a6-5e3ea51e0f9f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-vdf9d"
Sep 17 18:10:55.060688 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:10:55.060373    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-serving-cert: secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Sep 17 18:10:55.060688 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:10:55.060443    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4567fe39-89aa-494f-a1a6-5e3ea51e0f9f-proxy-tls podName:4567fe39-89aa-494f-a1a6-5e3ea51e0f9f nodeName:}" failed. No retries permitted until 2026-09-17 18:10:56.060426001 +0000 UTC m=+2192.474216428 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4567fe39-89aa-494f-a1a6-5e3ea51e0f9f-proxy-tls") pod "isvc-canary-stop-v2-predictor-596664c988-vdf9d" (UID: "4567fe39-89aa-494f-a1a6-5e3ea51e0f9f") : secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Sep 17 18:10:55.161207 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:55.161174    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77003c43-ce6b-47fc-b59e-ed19e15c9059-proxy-tls\") pod \"isvc-canary-stop-predictor-5cdcf6545-km5ql\" (UID: \"77003c43-ce6b-47fc-b59e-ed19e15c9059\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-km5ql"
Sep 17 18:10:55.161381 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:10:55.161343    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-predictor-serving-cert: secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 17 18:10:55.161468 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:10:55.161414    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77003c43-ce6b-47fc-b59e-ed19e15c9059-proxy-tls podName:77003c43-ce6b-47fc-b59e-ed19e15c9059 nodeName:}" failed. No retries permitted until 2026-09-17 18:10:56.161395086 +0000 UTC m=+2192.575185521 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/77003c43-ce6b-47fc-b59e-ed19e15c9059-proxy-tls") pod "isvc-canary-stop-predictor-5cdcf6545-km5ql" (UID: "77003c43-ce6b-47fc-b59e-ed19e15c9059") : secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 17 18:10:55.228772 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:55.228722    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:55.554322 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:55.554289    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:56.067745 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:56.067714    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4567fe39-89aa-494f-a1a6-5e3ea51e0f9f-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-596664c988-vdf9d\" (UID: \"4567fe39-89aa-494f-a1a6-5e3ea51e0f9f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-vdf9d"
Sep 17 18:10:56.070094 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:56.070070    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4567fe39-89aa-494f-a1a6-5e3ea51e0f9f-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-596664c988-vdf9d\" (UID: \"4567fe39-89aa-494f-a1a6-5e3ea51e0f9f\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-vdf9d"
Sep 17 18:10:56.168345 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:56.168315    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77003c43-ce6b-47fc-b59e-ed19e15c9059-proxy-tls\") pod \"isvc-canary-stop-predictor-5cdcf6545-km5ql\" (UID: \"77003c43-ce6b-47fc-b59e-ed19e15c9059\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-km5ql"
Sep 17 18:10:56.170635 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:56.170613    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77003c43-ce6b-47fc-b59e-ed19e15c9059-proxy-tls\") pod \"isvc-canary-stop-predictor-5cdcf6545-km5ql\" (UID: \"77003c43-ce6b-47fc-b59e-ed19e15c9059\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-km5ql"
Sep 17 18:10:56.256008 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:56.255983    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-vdf9d"
Sep 17 18:10:56.267694 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:56.267672    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-km5ql"
Sep 17 18:10:56.532043 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:56.531981    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-vdf9d"]
Sep 17 18:10:56.534146 ip-10-0-133-139 kubenswrapper[2585]: W0917 18:10:56.534118    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4567fe39_89aa_494f_a1a6_5e3ea51e0f9f.slice/crio-5d9acc61210c181c536634e76c0313d53b84f0604407903bd12f0b1a91fa6b06 WatchSource:0}: Error finding container 5d9acc61210c181c536634e76c0313d53b84f0604407903bd12f0b1a91fa6b06: Status 404 returned error can't find the container with id 5d9acc61210c181c536634e76c0313d53b84f0604407903bd12f0b1a91fa6b06
Sep 17 18:10:56.547318 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:56.547297    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-km5ql"]
Sep 17 18:10:56.548684 ip-10-0-133-139 kubenswrapper[2585]: W0917 18:10:56.548658    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77003c43_ce6b_47fc_b59e_ed19e15c9059.slice/crio-cc7add44d6e880ce75a7bb75da832d701b22f1444102f8082a64863b749eb78c WatchSource:0}: Error finding container cc7add44d6e880ce75a7bb75da832d701b22f1444102f8082a64863b749eb78c: Status 404 returned error can't find the container with id cc7add44d6e880ce75a7bb75da832d701b22f1444102f8082a64863b749eb78c
Sep 17 18:10:57.036561 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:57.036526    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-km5ql" event={"ID":"77003c43-ce6b-47fc-b59e-ed19e15c9059","Type":"ContainerStarted","Data":"8724f0fc8acbb11ce8d685aec8d500f1a70de49cf7a06387e8780f102bcda103"}
Sep 17 18:10:57.036755 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:57.036599    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-km5ql" event={"ID":"77003c43-ce6b-47fc-b59e-ed19e15c9059","Type":"ContainerStarted","Data":"cc7add44d6e880ce75a7bb75da832d701b22f1444102f8082a64863b749eb78c"}
Sep 17 18:10:57.038213 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:57.038185    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-vdf9d" event={"ID":"4567fe39-89aa-494f-a1a6-5e3ea51e0f9f","Type":"ContainerStarted","Data":"b1b16db5c04b69a4cae3e516f039d8c37b10bfaf100c7a53c9e96434bd9e127b"}
Sep 17 18:10:57.038313 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:10:57.038235    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-vdf9d" event={"ID":"4567fe39-89aa-494f-a1a6-5e3ea51e0f9f","Type":"ContainerStarted","Data":"5d9acc61210c181c536634e76c0313d53b84f0604407903bd12f0b1a91fa6b06"}
Sep 17 18:11:01.052439 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:11:01.052405    2585 generic.go:356] "Generic (PLEG): container finished" podID="4567fe39-89aa-494f-a1a6-5e3ea51e0f9f" containerID="b1b16db5c04b69a4cae3e516f039d8c37b10bfaf100c7a53c9e96434bd9e127b" exitCode=0
Sep 17 18:11:01.052882 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:11:01.052507    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-vdf9d" event={"ID":"4567fe39-89aa-494f-a1a6-5e3ea51e0f9f","Type":"ContainerDied","Data":"b1b16db5c04b69a4cae3e516f039d8c37b10bfaf100c7a53c9e96434bd9e127b"}
Sep 17 18:11:01.054109 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:11:01.054089    2585 generic.go:356] "Generic (PLEG): container finished" podID="77003c43-ce6b-47fc-b59e-ed19e15c9059" containerID="8724f0fc8acbb11ce8d685aec8d500f1a70de49cf7a06387e8780f102bcda103" exitCode=0
Sep 17 18:11:01.054182 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:11:01.054139    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-km5ql" event={"ID":"77003c43-ce6b-47fc-b59e-ed19e15c9059","Type":"ContainerDied","Data":"8724f0fc8acbb11ce8d685aec8d500f1a70de49cf7a06387e8780f102bcda103"}
Sep 17 18:11:02.060157 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:11:02.060115    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-km5ql" event={"ID":"77003c43-ce6b-47fc-b59e-ed19e15c9059","Type":"ContainerStarted","Data":"75ca6598200e980fe4663dc556bca7d0acdc1b830f35a15a9e96f10b628e9de6"}
Sep 17 18:11:02.060609 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:11:02.060184    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-km5ql" event={"ID":"77003c43-ce6b-47fc-b59e-ed19e15c9059","Type":"ContainerStarted","Data":"654119d47f763460e2426c5a141df556dd75a405a76e4b401c52f443c006af4a"}
Sep 17 18:11:02.060609 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:11:02.060379    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-km5ql"
Sep 17 18:11:02.062231 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:11:02.062208    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-vdf9d" event={"ID":"4567fe39-89aa-494f-a1a6-5e3ea51e0f9f","Type":"ContainerStarted","Data":"7e0c14b3aa174e13103353c693d8ff12cc2ea00fe42798773debc4c5b10cfe82"}
Sep 17 18:11:02.062330 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:11:02.062255    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-vdf9d" event={"ID":"4567fe39-89aa-494f-a1a6-5e3ea51e0f9f","Type":"ContainerStarted","Data":"c22c5eb679e965f39e272f0e1b45f07d1f10e781633dc04814f3872a9c7f56d8"}
Sep 17 18:11:02.062520 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:11:02.062505    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-vdf9d"
Sep 17 18:11:02.062567 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:11:02.062553    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-vdf9d"
Sep 17 18:11:02.064013 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:11:02.063987    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-vdf9d" podUID="4567fe39-89aa-494f-a1a6-5e3ea51e0f9f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 18:11:02.087751 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:11:02.087700    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-km5ql" podStartSLOduration=4.278955356 podStartE2EDuration="8.087685719s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.808730363s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:10:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:11:02.086211699 +0000 UTC m=+2198.500002137" watchObservedRunningTime="2026-09-17 18:11:02.087685719 +0000 UTC m=+2198.501476158"
Sep 17 18:11:02.112871 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:11:02.112820    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-vdf9d" podStartSLOduration=4.429939986 podStartE2EDuration="8.112806793s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.682866807s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:10:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:11:02.111530779 +0000 UTC m=+2198.525321220" watchObservedRunningTime="2026-09-17 18:11:02.112806793 +0000 UTC m=+2198.526597232"
Sep 17 18:11:03.066012 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:11:03.065962    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-vdf9d" podUID="4567fe39-89aa-494f-a1a6-5e3ea51e0f9f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 18:11:03.066012 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:11:03.065996    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-km5ql"
Sep 17 18:11:03.067158 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:11:03.067135    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-km5ql" podUID="77003c43-ce6b-47fc-b59e-ed19e15c9059" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 18:11:04.069544 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:11:04.069502    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-km5ql" podUID="77003c43-ce6b-47fc-b59e-ed19e15c9059" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 18:11:08.070527 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:11:08.070485    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-vdf9d"
Sep 17 18:11:08.071012 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:11:08.070981    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-vdf9d" podUID="4567fe39-89aa-494f-a1a6-5e3ea51e0f9f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 18:11:09.074709 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:11:09.074681    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-km5ql"
Sep 17 18:11:09.075274 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:11:09.075249    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-km5ql" podUID="77003c43-ce6b-47fc-b59e-ed19e15c9059" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 18:11:18.071631 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:11:18.071592    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-vdf9d" podUID="4567fe39-89aa-494f-a1a6-5e3ea51e0f9f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 18:11:19.005635 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:11:19.005603    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:11:19.075319 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:11:19.075279    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-km5ql" podUID="77003c43-ce6b-47fc-b59e-ed19e15c9059" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 18:11:19.079885 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:11:19.079861    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:11:28.071758 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:11:28.071716    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-vdf9d" podUID="4567fe39-89aa-494f-a1a6-5e3ea51e0f9f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 18:11:29.075921 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:11:29.075887    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-km5ql" podUID="77003c43-ce6b-47fc-b59e-ed19e15c9059" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 18:11:38.071145 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:11:38.071105    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-vdf9d" podUID="4567fe39-89aa-494f-a1a6-5e3ea51e0f9f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 18:11:39.075596 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:11:39.075562    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-km5ql" podUID="77003c43-ce6b-47fc-b59e-ed19e15c9059" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 18:11:40.204950 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:11:40.204917    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:11:45.319329 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:11:45.319295    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:11:45.509782 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:11:45.509745    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:11:48.071187 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:11:48.071103    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-vdf9d" podUID="4567fe39-89aa-494f-a1a6-5e3ea51e0f9f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 18:11:49.075529 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:11:49.075482    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-km5ql" podUID="77003c43-ce6b-47fc-b59e-ed19e15c9059" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 18:11:58.071250 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:11:58.071206    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-vdf9d" podUID="4567fe39-89aa-494f-a1a6-5e3ea51e0f9f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 18:11:59.075889 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:11:59.075834    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-km5ql" podUID="77003c43-ce6b-47fc-b59e-ed19e15c9059" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 18:12:08.071241 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:12:08.071196    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-vdf9d" podUID="4567fe39-89aa-494f-a1a6-5e3ea51e0f9f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 18:12:09.075977 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:12:09.075932    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-km5ql" podUID="77003c43-ce6b-47fc-b59e-ed19e15c9059" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 18:12:18.071860 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:12:18.071830    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-vdf9d"
Sep 17 18:12:18.210549 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:12:18.210481    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-km5ql"]
Sep 17 18:12:18.210870 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:12:18.210834    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-km5ql" podUID="77003c43-ce6b-47fc-b59e-ed19e15c9059" containerName="kserve-container" containerID="cri-o://654119d47f763460e2426c5a141df556dd75a405a76e4b401c52f443c006af4a" gracePeriod=30
Sep 17 18:12:18.211043 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:12:18.210889    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-km5ql" podUID="77003c43-ce6b-47fc-b59e-ed19e15c9059" containerName="kube-rbac-proxy" containerID="cri-o://75ca6598200e980fe4663dc556bca7d0acdc1b830f35a15a9e96f10b628e9de6" gracePeriod=30
Sep 17 18:12:18.212557 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:12:18.212535    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-km5ql"
Sep 17 18:12:18.345808 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:12:18.345776    2585 generic.go:356] "Generic (PLEG): container finished" podID="77003c43-ce6b-47fc-b59e-ed19e15c9059" containerID="75ca6598200e980fe4663dc556bca7d0acdc1b830f35a15a9e96f10b628e9de6" exitCode=2
Sep 17 18:12:18.345961 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:12:18.345853    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-km5ql" event={"ID":"77003c43-ce6b-47fc-b59e-ed19e15c9059","Type":"ContainerDied","Data":"75ca6598200e980fe4663dc556bca7d0acdc1b830f35a15a9e96f10b628e9de6"}
Sep 17 18:12:19.070461 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:12:19.070413    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-km5ql" podUID="77003c43-ce6b-47fc-b59e-ed19e15c9059" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Sep 17 18:12:23.259563 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:12:23.259536    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-km5ql"
Sep 17 18:12:23.324063 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:12:23.324034    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77003c43-ce6b-47fc-b59e-ed19e15c9059-kube-rbac-proxy-sar-config\") pod \"77003c43-ce6b-47fc-b59e-ed19e15c9059\" (UID: \"77003c43-ce6b-47fc-b59e-ed19e15c9059\") "
Sep 17 18:12:23.324267 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:12:23.324073    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77003c43-ce6b-47fc-b59e-ed19e15c9059-kserve-provision-location\") pod \"77003c43-ce6b-47fc-b59e-ed19e15c9059\" (UID: \"77003c43-ce6b-47fc-b59e-ed19e15c9059\") "
Sep 17 18:12:23.324267 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:12:23.324101    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77003c43-ce6b-47fc-b59e-ed19e15c9059-proxy-tls\") pod \"77003c43-ce6b-47fc-b59e-ed19e15c9059\" (UID: \"77003c43-ce6b-47fc-b59e-ed19e15c9059\") "
Sep 17 18:12:23.324267 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:12:23.324115    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77003c43-ce6b-47fc-b59e-ed19e15c9059-cabundle-cert\") pod \"77003c43-ce6b-47fc-b59e-ed19e15c9059\" (UID: \"77003c43-ce6b-47fc-b59e-ed19e15c9059\") "
Sep 17 18:12:23.324267 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:12:23.324174    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g85pm\" (UniqueName: \"kubernetes.io/projected/77003c43-ce6b-47fc-b59e-ed19e15c9059-kube-api-access-g85pm\") pod \"77003c43-ce6b-47fc-b59e-ed19e15c9059\" (UID: \"77003c43-ce6b-47fc-b59e-ed19e15c9059\") "
Sep 17 18:12:23.324520 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:12:23.324404    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77003c43-ce6b-47fc-b59e-ed19e15c9059-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "77003c43-ce6b-47fc-b59e-ed19e15c9059" (UID: "77003c43-ce6b-47fc-b59e-ed19e15c9059"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:12:23.324520 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:12:23.324453    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77003c43-ce6b-47fc-b59e-ed19e15c9059-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "77003c43-ce6b-47fc-b59e-ed19e15c9059" (UID: "77003c43-ce6b-47fc-b59e-ed19e15c9059"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:12:23.324520 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:12:23.324485    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77003c43-ce6b-47fc-b59e-ed19e15c9059-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "77003c43-ce6b-47fc-b59e-ed19e15c9059" (UID: "77003c43-ce6b-47fc-b59e-ed19e15c9059"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:12:23.326339 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:12:23.326315    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77003c43-ce6b-47fc-b59e-ed19e15c9059-kube-api-access-g85pm" (OuterVolumeSpecName: "kube-api-access-g85pm") pod "77003c43-ce6b-47fc-b59e-ed19e15c9059" (UID: "77003c43-ce6b-47fc-b59e-ed19e15c9059"). InnerVolumeSpecName "kube-api-access-g85pm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:12:23.326339 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:12:23.326321    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77003c43-ce6b-47fc-b59e-ed19e15c9059-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "77003c43-ce6b-47fc-b59e-ed19e15c9059" (UID: "77003c43-ce6b-47fc-b59e-ed19e15c9059"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:12:23.367316 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:12:23.367224    2585 generic.go:356] "Generic (PLEG): container finished" podID="77003c43-ce6b-47fc-b59e-ed19e15c9059" containerID="654119d47f763460e2426c5a141df556dd75a405a76e4b401c52f443c006af4a" exitCode=0
Sep 17 18:12:23.367316 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:12:23.367272    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-km5ql"
Sep 17 18:12:23.367316 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:12:23.367303    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-km5ql" event={"ID":"77003c43-ce6b-47fc-b59e-ed19e15c9059","Type":"ContainerDied","Data":"654119d47f763460e2426c5a141df556dd75a405a76e4b401c52f443c006af4a"}
Sep 17 18:12:23.367597 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:12:23.367373    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-km5ql" event={"ID":"77003c43-ce6b-47fc-b59e-ed19e15c9059","Type":"ContainerDied","Data":"cc7add44d6e880ce75a7bb75da832d701b22f1444102f8082a64863b749eb78c"}
Sep 17 18:12:23.367597 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:12:23.367404    2585 scope.go:117] "RemoveContainer" containerID="75ca6598200e980fe4663dc556bca7d0acdc1b830f35a15a9e96f10b628e9de6"
Sep 17 18:12:23.375969 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:12:23.375953    2585 scope.go:117] "RemoveContainer" containerID="654119d47f763460e2426c5a141df556dd75a405a76e4b401c52f443c006af4a"
Sep 17 18:12:23.382974 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:12:23.382957    2585 scope.go:117] "RemoveContainer" containerID="8724f0fc8acbb11ce8d685aec8d500f1a70de49cf7a06387e8780f102bcda103"
Sep 17 18:12:23.390069 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:12:23.390052    2585 scope.go:117] "RemoveContainer" containerID="75ca6598200e980fe4663dc556bca7d0acdc1b830f35a15a9e96f10b628e9de6"
Sep 17 18:12:23.390343 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:12:23.390321    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75ca6598200e980fe4663dc556bca7d0acdc1b830f35a15a9e96f10b628e9de6\": container with ID starting with 75ca6598200e980fe4663dc556bca7d0acdc1b830f35a15a9e96f10b628e9de6 not found: ID does not exist" containerID="75ca6598200e980fe4663dc556bca7d0acdc1b830f35a15a9e96f10b628e9de6"
Sep 17 18:12:23.390430 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:12:23.390358    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75ca6598200e980fe4663dc556bca7d0acdc1b830f35a15a9e96f10b628e9de6"} err="failed to get container status \"75ca6598200e980fe4663dc556bca7d0acdc1b830f35a15a9e96f10b628e9de6\": rpc error: code = NotFound desc = could not find container \"75ca6598200e980fe4663dc556bca7d0acdc1b830f35a15a9e96f10b628e9de6\": container with ID starting with 75ca6598200e980fe4663dc556bca7d0acdc1b830f35a15a9e96f10b628e9de6 not found: ID does not exist"
Sep 17 18:12:23.390430 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:12:23.390381    2585 scope.go:117] "RemoveContainer" containerID="654119d47f763460e2426c5a141df556dd75a405a76e4b401c52f443c006af4a"
Sep 17 18:12:23.390681 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:12:23.390663    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"654119d47f763460e2426c5a141df556dd75a405a76e4b401c52f443c006af4a\": container with ID starting with 654119d47f763460e2426c5a141df556dd75a405a76e4b401c52f443c006af4a not found: ID does not exist" containerID="654119d47f763460e2426c5a141df556dd75a405a76e4b401c52f443c006af4a"
Sep 17 18:12:23.390727 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:12:23.390690    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"654119d47f763460e2426c5a141df556dd75a405a76e4b401c52f443c006af4a"} err="failed to get container status \"654119d47f763460e2426c5a141df556dd75a405a76e4b401c52f443c006af4a\": rpc error: code = NotFound desc = could not find container \"654119d47f763460e2426c5a141df556dd75a405a76e4b401c52f443c006af4a\": container with ID starting with 654119d47f763460e2426c5a141df556dd75a405a76e4b401c52f443c006af4a not found: ID does not exist"
Sep 17 18:12:23.390727 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:12:23.390706    2585 scope.go:117] "RemoveContainer" containerID="8724f0fc8acbb11ce8d685aec8d500f1a70de49cf7a06387e8780f102bcda103"
Sep 17 18:12:23.390988 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:12:23.390964    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8724f0fc8acbb11ce8d685aec8d500f1a70de49cf7a06387e8780f102bcda103\": container with ID starting with 8724f0fc8acbb11ce8d685aec8d500f1a70de49cf7a06387e8780f102bcda103 not found: ID does not exist" containerID="8724f0fc8acbb11ce8d685aec8d500f1a70de49cf7a06387e8780f102bcda103"
Sep 17 18:12:23.391041 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:12:23.390987    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8724f0fc8acbb11ce8d685aec8d500f1a70de49cf7a06387e8780f102bcda103"} err="failed to get container status \"8724f0fc8acbb11ce8d685aec8d500f1a70de49cf7a06387e8780f102bcda103\": rpc error: code = NotFound desc = could not find container \"8724f0fc8acbb11ce8d685aec8d500f1a70de49cf7a06387e8780f102bcda103\": container with ID starting with 8724f0fc8acbb11ce8d685aec8d500f1a70de49cf7a06387e8780f102bcda103 not found: ID does not exist"
Sep 17 18:12:23.394737 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:12:23.394717    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-km5ql"]
Sep 17 18:12:23.400484 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:12:23.400464    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5cdcf6545-km5ql"]
Sep 17 18:12:23.425273 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:12:23.425239    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77003c43-ce6b-47fc-b59e-ed19e15c9059-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:12:23.425273 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:12:23.425266    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77003c43-ce6b-47fc-b59e-ed19e15c9059-kserve-provision-location\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:12:23.425273 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:12:23.425281    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77003c43-ce6b-47fc-b59e-ed19e15c9059-proxy-tls\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:12:23.425478 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:12:23.425293    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77003c43-ce6b-47fc-b59e-ed19e15c9059-cabundle-cert\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:12:23.425478 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:12:23.425304    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g85pm\" (UniqueName: \"kubernetes.io/projected/77003c43-ce6b-47fc-b59e-ed19e15c9059-kube-api-access-g85pm\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:12:24.103990 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:12:24.103956    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77003c43-ce6b-47fc-b59e-ed19e15c9059" path="/var/lib/kubelet/pods/77003c43-ce6b-47fc-b59e-ed19e15c9059/volumes"
Sep 17 18:13:04.463721 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:04.463687    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-vdf9d"]
Sep 17 18:13:04.464304 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:04.463964    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-vdf9d" podUID="4567fe39-89aa-494f-a1a6-5e3ea51e0f9f" containerName="kserve-container" containerID="cri-o://c22c5eb679e965f39e272f0e1b45f07d1f10e781633dc04814f3872a9c7f56d8" gracePeriod=30
Sep 17 18:13:04.464304 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:04.464016    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-vdf9d" podUID="4567fe39-89aa-494f-a1a6-5e3ea51e0f9f" containerName="kube-rbac-proxy" containerID="cri-o://7e0c14b3aa174e13103353c693d8ff12cc2ea00fe42798773debc4c5b10cfe82" gracePeriod=30
Sep 17 18:13:04.483422 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:04.483391    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:13:05.517305 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:05.517270    2585 generic.go:356] "Generic (PLEG): container finished" podID="4567fe39-89aa-494f-a1a6-5e3ea51e0f9f" containerID="7e0c14b3aa174e13103353c693d8ff12cc2ea00fe42798773debc4c5b10cfe82" exitCode=2
Sep 17 18:13:05.517689 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:05.517338    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-vdf9d" event={"ID":"4567fe39-89aa-494f-a1a6-5e3ea51e0f9f","Type":"ContainerDied","Data":"7e0c14b3aa174e13103353c693d8ff12cc2ea00fe42798773debc4c5b10cfe82"}
Sep 17 18:13:05.907571 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:05.907472    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:13:08.066237 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:08.066194    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-vdf9d" podUID="4567fe39-89aa-494f-a1a6-5e3ea51e0f9f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Sep 17 18:13:08.071648 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:08.071612    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-vdf9d" podUID="4567fe39-89aa-494f-a1a6-5e3ea51e0f9f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 18:13:09.101371 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:09.101349    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-vdf9d"
Sep 17 18:13:09.250688 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:09.250661    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4567fe39-89aa-494f-a1a6-5e3ea51e0f9f-kserve-provision-location\") pod \"4567fe39-89aa-494f-a1a6-5e3ea51e0f9f\" (UID: \"4567fe39-89aa-494f-a1a6-5e3ea51e0f9f\") "
Sep 17 18:13:09.250874 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:09.250700    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4567fe39-89aa-494f-a1a6-5e3ea51e0f9f-kube-rbac-proxy-sar-config\") pod \"4567fe39-89aa-494f-a1a6-5e3ea51e0f9f\" (UID: \"4567fe39-89aa-494f-a1a6-5e3ea51e0f9f\") "
Sep 17 18:13:09.250874 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:09.250724    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4567fe39-89aa-494f-a1a6-5e3ea51e0f9f-proxy-tls\") pod \"4567fe39-89aa-494f-a1a6-5e3ea51e0f9f\" (UID: \"4567fe39-89aa-494f-a1a6-5e3ea51e0f9f\") "
Sep 17 18:13:09.250874 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:09.250784    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4567fe39-89aa-494f-a1a6-5e3ea51e0f9f-cabundle-cert\") pod \"4567fe39-89aa-494f-a1a6-5e3ea51e0f9f\" (UID: \"4567fe39-89aa-494f-a1a6-5e3ea51e0f9f\") "
Sep 17 18:13:09.250874 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:09.250820    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x6tm6\" (UniqueName: \"kubernetes.io/projected/4567fe39-89aa-494f-a1a6-5e3ea51e0f9f-kube-api-access-x6tm6\") pod \"4567fe39-89aa-494f-a1a6-5e3ea51e0f9f\" (UID: \"4567fe39-89aa-494f-a1a6-5e3ea51e0f9f\") "
Sep 17 18:13:09.251086 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:09.251029    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4567fe39-89aa-494f-a1a6-5e3ea51e0f9f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4567fe39-89aa-494f-a1a6-5e3ea51e0f9f" (UID: "4567fe39-89aa-494f-a1a6-5e3ea51e0f9f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:13:09.251086 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:09.251046    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4567fe39-89aa-494f-a1a6-5e3ea51e0f9f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4567fe39-89aa-494f-a1a6-5e3ea51e0f9f" (UID: "4567fe39-89aa-494f-a1a6-5e3ea51e0f9f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:13:09.251202 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:09.251171    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4567fe39-89aa-494f-a1a6-5e3ea51e0f9f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4567fe39-89aa-494f-a1a6-5e3ea51e0f9f" (UID: "4567fe39-89aa-494f-a1a6-5e3ea51e0f9f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:13:09.252879 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:09.252854    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4567fe39-89aa-494f-a1a6-5e3ea51e0f9f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4567fe39-89aa-494f-a1a6-5e3ea51e0f9f" (UID: "4567fe39-89aa-494f-a1a6-5e3ea51e0f9f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:13:09.252964 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:09.252911    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4567fe39-89aa-494f-a1a6-5e3ea51e0f9f-kube-api-access-x6tm6" (OuterVolumeSpecName: "kube-api-access-x6tm6") pod "4567fe39-89aa-494f-a1a6-5e3ea51e0f9f" (UID: "4567fe39-89aa-494f-a1a6-5e3ea51e0f9f"). InnerVolumeSpecName "kube-api-access-x6tm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:13:09.351462 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:09.351413    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4567fe39-89aa-494f-a1a6-5e3ea51e0f9f-cabundle-cert\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:13:09.351462 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:09.351454    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6tm6\" (UniqueName: \"kubernetes.io/projected/4567fe39-89aa-494f-a1a6-5e3ea51e0f9f-kube-api-access-x6tm6\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:13:09.351462 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:09.351466    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4567fe39-89aa-494f-a1a6-5e3ea51e0f9f-kserve-provision-location\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:13:09.351462 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:09.351475    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4567fe39-89aa-494f-a1a6-5e3ea51e0f9f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:13:09.351748 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:09.351486    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4567fe39-89aa-494f-a1a6-5e3ea51e0f9f-proxy-tls\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:13:09.532421 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:09.532339    2585 generic.go:356] "Generic (PLEG): container finished" podID="4567fe39-89aa-494f-a1a6-5e3ea51e0f9f" containerID="c22c5eb679e965f39e272f0e1b45f07d1f10e781633dc04814f3872a9c7f56d8" exitCode=0
Sep 17 18:13:09.532421 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:09.532406    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-vdf9d"
Sep 17 18:13:09.532639 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:09.532424    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-vdf9d" event={"ID":"4567fe39-89aa-494f-a1a6-5e3ea51e0f9f","Type":"ContainerDied","Data":"c22c5eb679e965f39e272f0e1b45f07d1f10e781633dc04814f3872a9c7f56d8"}
Sep 17 18:13:09.532639 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:09.532509    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-vdf9d" event={"ID":"4567fe39-89aa-494f-a1a6-5e3ea51e0f9f","Type":"ContainerDied","Data":"5d9acc61210c181c536634e76c0313d53b84f0604407903bd12f0b1a91fa6b06"}
Sep 17 18:13:09.532639 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:09.532539    2585 scope.go:117] "RemoveContainer" containerID="7e0c14b3aa174e13103353c693d8ff12cc2ea00fe42798773debc4c5b10cfe82"
Sep 17 18:13:09.541750 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:09.541727    2585 scope.go:117] "RemoveContainer" containerID="c22c5eb679e965f39e272f0e1b45f07d1f10e781633dc04814f3872a9c7f56d8"
Sep 17 18:13:09.550291 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:09.550273    2585 scope.go:117] "RemoveContainer" containerID="b1b16db5c04b69a4cae3e516f039d8c37b10bfaf100c7a53c9e96434bd9e127b"
Sep 17 18:13:09.554797 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:09.554775    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-vdf9d"]
Sep 17 18:13:09.557460 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:09.557427    2585 scope.go:117] "RemoveContainer" containerID="7e0c14b3aa174e13103353c693d8ff12cc2ea00fe42798773debc4c5b10cfe82"
Sep 17 18:13:09.557738 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:13:09.557713    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e0c14b3aa174e13103353c693d8ff12cc2ea00fe42798773debc4c5b10cfe82\": container with ID starting with 7e0c14b3aa174e13103353c693d8ff12cc2ea00fe42798773debc4c5b10cfe82 not found: ID does not exist" containerID="7e0c14b3aa174e13103353c693d8ff12cc2ea00fe42798773debc4c5b10cfe82"
Sep 17 18:13:09.557824 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:09.557749    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e0c14b3aa174e13103353c693d8ff12cc2ea00fe42798773debc4c5b10cfe82"} err="failed to get container status \"7e0c14b3aa174e13103353c693d8ff12cc2ea00fe42798773debc4c5b10cfe82\": rpc error: code = NotFound desc = could not find container \"7e0c14b3aa174e13103353c693d8ff12cc2ea00fe42798773debc4c5b10cfe82\": container with ID starting with 7e0c14b3aa174e13103353c693d8ff12cc2ea00fe42798773debc4c5b10cfe82 not found: ID does not exist"
Sep 17 18:13:09.557824 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:09.557774    2585 scope.go:117] "RemoveContainer" containerID="c22c5eb679e965f39e272f0e1b45f07d1f10e781633dc04814f3872a9c7f56d8"
Sep 17 18:13:09.558011 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:13:09.557994    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c22c5eb679e965f39e272f0e1b45f07d1f10e781633dc04814f3872a9c7f56d8\": container with ID starting with c22c5eb679e965f39e272f0e1b45f07d1f10e781633dc04814f3872a9c7f56d8 not found: ID does not exist" containerID="c22c5eb679e965f39e272f0e1b45f07d1f10e781633dc04814f3872a9c7f56d8"
Sep 17 18:13:09.558076 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:09.558020    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c22c5eb679e965f39e272f0e1b45f07d1f10e781633dc04814f3872a9c7f56d8"} err="failed to get container status \"c22c5eb679e965f39e272f0e1b45f07d1f10e781633dc04814f3872a9c7f56d8\": rpc error: code = NotFound desc = could not find container \"c22c5eb679e965f39e272f0e1b45f07d1f10e781633dc04814f3872a9c7f56d8\": container with ID starting with c22c5eb679e965f39e272f0e1b45f07d1f10e781633dc04814f3872a9c7f56d8 not found: ID does not exist"
Sep 17 18:13:09.558076 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:09.558041    2585 scope.go:117] "RemoveContainer" containerID="b1b16db5c04b69a4cae3e516f039d8c37b10bfaf100c7a53c9e96434bd9e127b"
Sep 17 18:13:09.558254 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:13:09.558239    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1b16db5c04b69a4cae3e516f039d8c37b10bfaf100c7a53c9e96434bd9e127b\": container with ID starting with b1b16db5c04b69a4cae3e516f039d8c37b10bfaf100c7a53c9e96434bd9e127b not found: ID does not exist" containerID="b1b16db5c04b69a4cae3e516f039d8c37b10bfaf100c7a53c9e96434bd9e127b"
Sep 17 18:13:09.558307 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:09.558260    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1b16db5c04b69a4cae3e516f039d8c37b10bfaf100c7a53c9e96434bd9e127b"} err="failed to get container status \"b1b16db5c04b69a4cae3e516f039d8c37b10bfaf100c7a53c9e96434bd9e127b\": rpc error: code = NotFound desc = could not find container \"b1b16db5c04b69a4cae3e516f039d8c37b10bfaf100c7a53c9e96434bd9e127b\": container with ID starting with b1b16db5c04b69a4cae3e516f039d8c37b10bfaf100c7a53c9e96434bd9e127b not found: ID does not exist"
Sep 17 18:13:09.558667 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:09.558648    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-596664c988-vdf9d"]
Sep 17 18:13:10.104328 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:10.104295    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4567fe39-89aa-494f-a1a6-5e3ea51e0f9f" path="/var/lib/kubelet/pods/4567fe39-89aa-494f-a1a6-5e3ea51e0f9f/volumes"
Sep 17 18:13:14.693803 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:14.693768    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:13:14.735158 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:14.735129    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dz58c"]
Sep 17 18:13:14.735393 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:14.735381    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77003c43-ce6b-47fc-b59e-ed19e15c9059" containerName="kserve-container"
Sep 17 18:13:14.735393 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:14.735394    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="77003c43-ce6b-47fc-b59e-ed19e15c9059" containerName="kserve-container"
Sep 17 18:13:14.735474 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:14.735402    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77003c43-ce6b-47fc-b59e-ed19e15c9059" containerName="kube-rbac-proxy"
Sep 17 18:13:14.735474 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:14.735408    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="77003c43-ce6b-47fc-b59e-ed19e15c9059" containerName="kube-rbac-proxy"
Sep 17 18:13:14.735474 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:14.735415    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4567fe39-89aa-494f-a1a6-5e3ea51e0f9f" containerName="kube-rbac-proxy"
Sep 17 18:13:14.735474 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:14.735421    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="4567fe39-89aa-494f-a1a6-5e3ea51e0f9f" containerName="kube-rbac-proxy"
Sep 17 18:13:14.735474 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:14.735427    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4567fe39-89aa-494f-a1a6-5e3ea51e0f9f" containerName="kserve-container"
Sep 17 18:13:14.735474 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:14.735433    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="4567fe39-89aa-494f-a1a6-5e3ea51e0f9f" containerName="kserve-container"
Sep 17 18:13:14.735474 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:14.735442    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4567fe39-89aa-494f-a1a6-5e3ea51e0f9f" containerName="storage-initializer"
Sep 17 18:13:14.735474 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:14.735448    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="4567fe39-89aa-494f-a1a6-5e3ea51e0f9f" containerName="storage-initializer"
Sep 17 18:13:14.735474 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:14.735455    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77003c43-ce6b-47fc-b59e-ed19e15c9059" containerName="storage-initializer"
Sep 17 18:13:14.735474 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:14.735460    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="77003c43-ce6b-47fc-b59e-ed19e15c9059" containerName="storage-initializer"
Sep 17 18:13:14.735784 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:14.735517    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="77003c43-ce6b-47fc-b59e-ed19e15c9059" containerName="kube-rbac-proxy"
Sep 17 18:13:14.735784 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:14.735524    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="4567fe39-89aa-494f-a1a6-5e3ea51e0f9f" containerName="kube-rbac-proxy"
Sep 17 18:13:14.735784 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:14.735531    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="4567fe39-89aa-494f-a1a6-5e3ea51e0f9f" containerName="kserve-container"
Sep 17 18:13:14.735784 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:14.735537    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="77003c43-ce6b-47fc-b59e-ed19e15c9059" containerName="kserve-container"
Sep 17 18:13:14.739668 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:14.739644    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dz58c"
Sep 17 18:13:14.742284 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:14.742264    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:13:14.742423 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:14.742332    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:13:14.742423 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:14.742373    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:13:14.742579 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:14.742516    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
Sep 17 18:13:14.743596 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:14.743580    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-predictor-serving-cert\""
Sep 17 18:13:14.743675 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:14.743584    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lps5w\""
Sep 17 18:13:14.743675 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:14.743617    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:13:14.749121 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:14.749099    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dz58c"]
Sep 17 18:13:14.887136 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:14.887098    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e9354cff-dc48-4dbd-be0d-4067fbaf6c27-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-dz58c\" (UID: \"e9354cff-dc48-4dbd-be0d-4067fbaf6c27\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dz58c"
Sep 17 18:13:14.887136 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:14.887143    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9354cff-dc48-4dbd-be0d-4067fbaf6c27-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-dz58c\" (UID: \"e9354cff-dc48-4dbd-be0d-4067fbaf6c27\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dz58c"
Sep 17 18:13:14.887355 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:14.887199    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mhgp\" (UniqueName: \"kubernetes.io/projected/e9354cff-dc48-4dbd-be0d-4067fbaf6c27-kube-api-access-5mhgp\") pod \"isvc-lightgbm-predictor-54f85ddf88-dz58c\" (UID: \"e9354cff-dc48-4dbd-be0d-4067fbaf6c27\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dz58c"
Sep 17 18:13:14.887355 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:14.887228    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9354cff-dc48-4dbd-be0d-4067fbaf6c27-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-dz58c\" (UID: \"e9354cff-dc48-4dbd-be0d-4067fbaf6c27\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dz58c"
Sep 17 18:13:14.887355 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:14.887249    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9354cff-dc48-4dbd-be0d-4067fbaf6c27-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-dz58c\" (UID: \"e9354cff-dc48-4dbd-be0d-4067fbaf6c27\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dz58c"
Sep 17 18:13:14.988179 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:14.988109    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mhgp\" (UniqueName: \"kubernetes.io/projected/e9354cff-dc48-4dbd-be0d-4067fbaf6c27-kube-api-access-5mhgp\") pod \"isvc-lightgbm-predictor-54f85ddf88-dz58c\" (UID: \"e9354cff-dc48-4dbd-be0d-4067fbaf6c27\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dz58c"
Sep 17 18:13:14.988179 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:14.988152    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9354cff-dc48-4dbd-be0d-4067fbaf6c27-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-dz58c\" (UID: \"e9354cff-dc48-4dbd-be0d-4067fbaf6c27\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dz58c"
Sep 17 18:13:14.988179 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:14.988170    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9354cff-dc48-4dbd-be0d-4067fbaf6c27-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-dz58c\" (UID: \"e9354cff-dc48-4dbd-be0d-4067fbaf6c27\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dz58c"
Sep 17 18:13:14.988375 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:14.988209    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e9354cff-dc48-4dbd-be0d-4067fbaf6c27-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-dz58c\" (UID: \"e9354cff-dc48-4dbd-be0d-4067fbaf6c27\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dz58c"
Sep 17 18:13:14.988375 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:14.988229    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9354cff-dc48-4dbd-be0d-4067fbaf6c27-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-dz58c\" (UID: \"e9354cff-dc48-4dbd-be0d-4067fbaf6c27\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dz58c"
Sep 17 18:13:14.988375 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:13:14.988272    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-predictor-serving-cert: secret "isvc-lightgbm-predictor-serving-cert" not found
Sep 17 18:13:14.988375 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:13:14.988367    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9354cff-dc48-4dbd-be0d-4067fbaf6c27-proxy-tls podName:e9354cff-dc48-4dbd-be0d-4067fbaf6c27 nodeName:}" failed. No retries permitted until 2026-09-17 18:13:15.488346759 +0000 UTC m=+2331.902137195 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e9354cff-dc48-4dbd-be0d-4067fbaf6c27-proxy-tls") pod "isvc-lightgbm-predictor-54f85ddf88-dz58c" (UID: "e9354cff-dc48-4dbd-be0d-4067fbaf6c27") : secret "isvc-lightgbm-predictor-serving-cert" not found
Sep 17 18:13:14.988714 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:14.988696    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9354cff-dc48-4dbd-be0d-4067fbaf6c27-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-dz58c\" (UID: \"e9354cff-dc48-4dbd-be0d-4067fbaf6c27\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dz58c"
Sep 17 18:13:14.988940 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:14.988924    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9354cff-dc48-4dbd-be0d-4067fbaf6c27-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-dz58c\" (UID: \"e9354cff-dc48-4dbd-be0d-4067fbaf6c27\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dz58c"
Sep 17 18:13:14.988978 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:14.988950    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e9354cff-dc48-4dbd-be0d-4067fbaf6c27-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-dz58c\" (UID: \"e9354cff-dc48-4dbd-be0d-4067fbaf6c27\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dz58c"
Sep 17 18:13:14.997712 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:14.997688    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mhgp\" (UniqueName: \"kubernetes.io/projected/e9354cff-dc48-4dbd-be0d-4067fbaf6c27-kube-api-access-5mhgp\") pod \"isvc-lightgbm-predictor-54f85ddf88-dz58c\" (UID: \"e9354cff-dc48-4dbd-be0d-4067fbaf6c27\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dz58c"
Sep 17 18:13:15.183847 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:15.183817    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:13:15.493152 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:15.493112    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9354cff-dc48-4dbd-be0d-4067fbaf6c27-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-dz58c\" (UID: \"e9354cff-dc48-4dbd-be0d-4067fbaf6c27\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dz58c"
Sep 17 18:13:15.495500 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:15.495472    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9354cff-dc48-4dbd-be0d-4067fbaf6c27-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-dz58c\" (UID: \"e9354cff-dc48-4dbd-be0d-4067fbaf6c27\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dz58c"
Sep 17 18:13:15.650118 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:15.650075    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dz58c"
Sep 17 18:13:15.923475 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:15.923258    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dz58c"]
Sep 17 18:13:15.926163 ip-10-0-133-139 kubenswrapper[2585]: W0917 18:13:15.926140    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9354cff_dc48_4dbd_be0d_4067fbaf6c27.slice/crio-ac789f685dd9131a7dbcb7de9c921450ed6eced29486752e79704c22172f39fe WatchSource:0}: Error finding container ac789f685dd9131a7dbcb7de9c921450ed6eced29486752e79704c22172f39fe: Status 404 returned error can't find the container with id ac789f685dd9131a7dbcb7de9c921450ed6eced29486752e79704c22172f39fe
Sep 17 18:13:16.564124 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:16.564084    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dz58c" event={"ID":"e9354cff-dc48-4dbd-be0d-4067fbaf6c27","Type":"ContainerStarted","Data":"c7906b4f001b76f6cfea1f94a9bb750689d9812dce6d83462e102057c4faf3b4"}
Sep 17 18:13:16.564306 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:16.564142    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dz58c" event={"ID":"e9354cff-dc48-4dbd-be0d-4067fbaf6c27","Type":"ContainerStarted","Data":"ac789f685dd9131a7dbcb7de9c921450ed6eced29486752e79704c22172f39fe"}
Sep 17 18:13:20.580546 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:20.580508    2585 generic.go:356] "Generic (PLEG): container finished" podID="e9354cff-dc48-4dbd-be0d-4067fbaf6c27" containerID="c7906b4f001b76f6cfea1f94a9bb750689d9812dce6d83462e102057c4faf3b4" exitCode=0
Sep 17 18:13:20.580546 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:20.580548    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dz58c" event={"ID":"e9354cff-dc48-4dbd-be0d-4067fbaf6c27","Type":"ContainerDied","Data":"c7906b4f001b76f6cfea1f94a9bb750689d9812dce6d83462e102057c4faf3b4"}
Sep 17 18:13:25.871121 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:25.871077    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:13:27.613342 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:27.613305    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dz58c" event={"ID":"e9354cff-dc48-4dbd-be0d-4067fbaf6c27","Type":"ContainerStarted","Data":"50b052be424a47971e3d48f801e56ef10c00bac6f107a34a63444f12c25e6431"}
Sep 17 18:13:27.613838 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:27.613374    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dz58c" event={"ID":"e9354cff-dc48-4dbd-be0d-4067fbaf6c27","Type":"ContainerStarted","Data":"7cf5beed0a1bda93430850ff544787f2ef1b386ae10792472f33d864b6b8d716"}
Sep 17 18:13:27.613838 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:27.613666    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dz58c"
Sep 17 18:13:27.613838 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:27.613719    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dz58c"
Sep 17 18:13:27.615123 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:27.615094    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dz58c" podUID="e9354cff-dc48-4dbd-be0d-4067fbaf6c27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 18:13:27.640385 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:27.640339    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dz58c" podStartSLOduration=3.350855482 podStartE2EDuration="13.640326805s" totalImagesPullingTime="6.670304379s" totalInitContainerRuntime="3.619166944s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:13:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:13:27.638320289 +0000 UTC m=+2344.052110727" watchObservedRunningTime="2026-09-17 18:13:27.640326805 +0000 UTC m=+2344.054117243"
Sep 17 18:13:28.617443 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:28.617403    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dz58c" podUID="e9354cff-dc48-4dbd-be0d-4067fbaf6c27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 18:13:30.390352 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:30.390316    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:13:31.036813 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:31.036775    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:13:31.276547 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:31.276509    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:13:33.621633 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:33.621603    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dz58c"
Sep 17 18:13:33.622147 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:33.622121    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dz58c" podUID="e9354cff-dc48-4dbd-be0d-4067fbaf6c27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 18:13:43.622262 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:43.622212    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dz58c" podUID="e9354cff-dc48-4dbd-be0d-4067fbaf6c27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 18:13:53.622241 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:13:53.622200    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dz58c" podUID="e9354cff-dc48-4dbd-be0d-4067fbaf6c27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 18:14:03.622295 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:14:03.622257    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dz58c" podUID="e9354cff-dc48-4dbd-be0d-4067fbaf6c27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 18:14:13.622285 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:14:13.622244    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dz58c" podUID="e9354cff-dc48-4dbd-be0d-4067fbaf6c27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 18:14:21.121924 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:14:21.121881    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:14:23.622348 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:14:23.622310    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dz58c" podUID="e9354cff-dc48-4dbd-be0d-4067fbaf6c27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 18:14:26.260392 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:14:26.260363    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:14:26.627680 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:14:26.627599    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:14:28.472645 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:14:28.472615    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rrf79_beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5/ovn-acl-logging/0.log"
Sep 17 18:14:28.475120 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:14:28.475100    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rrf79_beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5/ovn-acl-logging/0.log"
Sep 17 18:14:30.664971 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:14:30.664937    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:14:33.622833 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:14:33.622799    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dz58c" podUID="e9354cff-dc48-4dbd-be0d-4067fbaf6c27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 18:14:35.100453 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:14:35.100418    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dz58c" podUID="e9354cff-dc48-4dbd-be0d-4067fbaf6c27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 18:14:42.807460 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:14:42.807419    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:14:45.100853 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:14:45.100814    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dz58c" podUID="e9354cff-dc48-4dbd-be0d-4067fbaf6c27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 18:14:55.101174 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:14:55.101084    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dz58c" podUID="e9354cff-dc48-4dbd-be0d-4067fbaf6c27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 18:15:05.101401 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:05.101361    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dz58c"
Sep 17 18:15:15.322058 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:15.322019    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dz58c"]
Sep 17 18:15:15.322507 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:15.322289    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dz58c" podUID="e9354cff-dc48-4dbd-be0d-4067fbaf6c27" containerName="kserve-container" containerID="cri-o://7cf5beed0a1bda93430850ff544787f2ef1b386ae10792472f33d864b6b8d716" gracePeriod=30
Sep 17 18:15:15.322507 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:15.322339    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dz58c" podUID="e9354cff-dc48-4dbd-be0d-4067fbaf6c27" containerName="kube-rbac-proxy" containerID="cri-o://50b052be424a47971e3d48f801e56ef10c00bac6f107a34a63444f12c25e6431" gracePeriod=30
Sep 17 18:15:15.330816 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:15.330787    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:15.985815 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:15.985776    2585 generic.go:356] "Generic (PLEG): container finished" podID="e9354cff-dc48-4dbd-be0d-4067fbaf6c27" containerID="50b052be424a47971e3d48f801e56ef10c00bac6f107a34a63444f12c25e6431" exitCode=2
Sep 17 18:15:15.985995 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:15.985849    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dz58c" event={"ID":"e9354cff-dc48-4dbd-be0d-4067fbaf6c27","Type":"ContainerDied","Data":"50b052be424a47971e3d48f801e56ef10c00bac6f107a34a63444f12c25e6431"}
Sep 17 18:15:16.372105 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:16.372071    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:18.618059 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:18.618020    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dz58c" podUID="e9354cff-dc48-4dbd-be0d-4067fbaf6c27" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
Sep 17 18:15:20.506717 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:20.506687    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:20.527937 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:20.527908    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx"]
Sep 17 18:15:20.530919 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:20.530904    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx"
Sep 17 18:15:20.533467 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:20.533449    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
Sep 17 18:15:20.533580 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:20.533482    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
Sep 17 18:15:20.547378 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:20.547358    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx"]
Sep 17 18:15:20.591257 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:20.591227    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cafe1fd8-26c9-4d21-97a5-0785fa7a4123-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx\" (UID: \"cafe1fd8-26c9-4d21-97a5-0785fa7a4123\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx"
Sep 17 18:15:20.591257 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:20.591260    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cafe1fd8-26c9-4d21-97a5-0785fa7a4123-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx\" (UID: \"cafe1fd8-26c9-4d21-97a5-0785fa7a4123\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx"
Sep 17 18:15:20.591435 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:20.591289    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shpzz\" (UniqueName: \"kubernetes.io/projected/cafe1fd8-26c9-4d21-97a5-0785fa7a4123-kube-api-access-shpzz\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx\" (UID: \"cafe1fd8-26c9-4d21-97a5-0785fa7a4123\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx"
Sep 17 18:15:20.591435 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:20.591384    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cafe1fd8-26c9-4d21-97a5-0785fa7a4123-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx\" (UID: \"cafe1fd8-26c9-4d21-97a5-0785fa7a4123\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx"
Sep 17 18:15:20.591435 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:20.591406    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cafe1fd8-26c9-4d21-97a5-0785fa7a4123-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx\" (UID: \"cafe1fd8-26c9-4d21-97a5-0785fa7a4123\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx"
Sep 17 18:15:20.691696 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:20.691666    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cafe1fd8-26c9-4d21-97a5-0785fa7a4123-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx\" (UID: \"cafe1fd8-26c9-4d21-97a5-0785fa7a4123\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx"
Sep 17 18:15:20.691696 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:20.691696    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cafe1fd8-26c9-4d21-97a5-0785fa7a4123-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx\" (UID: \"cafe1fd8-26c9-4d21-97a5-0785fa7a4123\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx"
Sep 17 18:15:20.691909 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:20.691728    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cafe1fd8-26c9-4d21-97a5-0785fa7a4123-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx\" (UID: \"cafe1fd8-26c9-4d21-97a5-0785fa7a4123\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx"
Sep 17 18:15:20.691909 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:20.691745    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cafe1fd8-26c9-4d21-97a5-0785fa7a4123-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx\" (UID: \"cafe1fd8-26c9-4d21-97a5-0785fa7a4123\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx"
Sep 17 18:15:20.691909 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:20.691768    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shpzz\" (UniqueName: \"kubernetes.io/projected/cafe1fd8-26c9-4d21-97a5-0785fa7a4123-kube-api-access-shpzz\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx\" (UID: \"cafe1fd8-26c9-4d21-97a5-0785fa7a4123\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx"
Sep 17 18:15:20.692261 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:20.692237    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cafe1fd8-26c9-4d21-97a5-0785fa7a4123-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx\" (UID: \"cafe1fd8-26c9-4d21-97a5-0785fa7a4123\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx"
Sep 17 18:15:20.692449 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:20.692431    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cafe1fd8-26c9-4d21-97a5-0785fa7a4123-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx\" (UID: \"cafe1fd8-26c9-4d21-97a5-0785fa7a4123\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx"
Sep 17 18:15:20.692531 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:20.692470    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cafe1fd8-26c9-4d21-97a5-0785fa7a4123-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx\" (UID: \"cafe1fd8-26c9-4d21-97a5-0785fa7a4123\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx"
Sep 17 18:15:20.694227 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:20.694207    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cafe1fd8-26c9-4d21-97a5-0785fa7a4123-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx\" (UID: \"cafe1fd8-26c9-4d21-97a5-0785fa7a4123\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx"
Sep 17 18:15:20.699920 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:20.699899    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shpzz\" (UniqueName: \"kubernetes.io/projected/cafe1fd8-26c9-4d21-97a5-0785fa7a4123-kube-api-access-shpzz\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx\" (UID: \"cafe1fd8-26c9-4d21-97a5-0785fa7a4123\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx"
Sep 17 18:15:20.708166 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:20.708136    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:20.840640 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:20.840610    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx"
Sep 17 18:15:20.861723 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:20.861698    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dz58c"
Sep 17 18:15:20.892385 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:20.892346    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e9354cff-dc48-4dbd-be0d-4067fbaf6c27-cabundle-cert\") pod \"e9354cff-dc48-4dbd-be0d-4067fbaf6c27\" (UID: \"e9354cff-dc48-4dbd-be0d-4067fbaf6c27\") "
Sep 17 18:15:20.892530 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:20.892408    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5mhgp\" (UniqueName: \"kubernetes.io/projected/e9354cff-dc48-4dbd-be0d-4067fbaf6c27-kube-api-access-5mhgp\") pod \"e9354cff-dc48-4dbd-be0d-4067fbaf6c27\" (UID: \"e9354cff-dc48-4dbd-be0d-4067fbaf6c27\") "
Sep 17 18:15:20.892530 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:20.892448    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9354cff-dc48-4dbd-be0d-4067fbaf6c27-kserve-provision-location\") pod \"e9354cff-dc48-4dbd-be0d-4067fbaf6c27\" (UID: \"e9354cff-dc48-4dbd-be0d-4067fbaf6c27\") "
Sep 17 18:15:20.892530 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:20.892481    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9354cff-dc48-4dbd-be0d-4067fbaf6c27-proxy-tls\") pod \"e9354cff-dc48-4dbd-be0d-4067fbaf6c27\" (UID: \"e9354cff-dc48-4dbd-be0d-4067fbaf6c27\") "
Sep 17 18:15:20.892711 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:20.892558    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9354cff-dc48-4dbd-be0d-4067fbaf6c27-kube-rbac-proxy-sar-config\") pod \"e9354cff-dc48-4dbd-be0d-4067fbaf6c27\" (UID: \"e9354cff-dc48-4dbd-be0d-4067fbaf6c27\") "
Sep 17 18:15:20.892811 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:20.892784    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9354cff-dc48-4dbd-be0d-4067fbaf6c27-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e9354cff-dc48-4dbd-be0d-4067fbaf6c27" (UID: "e9354cff-dc48-4dbd-be0d-4067fbaf6c27"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:15:20.893095 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:20.893058    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9354cff-dc48-4dbd-be0d-4067fbaf6c27-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e9354cff-dc48-4dbd-be0d-4067fbaf6c27" (UID: "e9354cff-dc48-4dbd-be0d-4067fbaf6c27"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:15:20.893214 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:20.893195    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9354cff-dc48-4dbd-be0d-4067fbaf6c27-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e9354cff-dc48-4dbd-be0d-4067fbaf6c27" (UID: "e9354cff-dc48-4dbd-be0d-4067fbaf6c27"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:15:20.895147 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:20.895119    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9354cff-dc48-4dbd-be0d-4067fbaf6c27-kube-api-access-5mhgp" (OuterVolumeSpecName: "kube-api-access-5mhgp") pod "e9354cff-dc48-4dbd-be0d-4067fbaf6c27" (UID: "e9354cff-dc48-4dbd-be0d-4067fbaf6c27"). InnerVolumeSpecName "kube-api-access-5mhgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:15:20.896809 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:20.896784    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9354cff-dc48-4dbd-be0d-4067fbaf6c27-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e9354cff-dc48-4dbd-be0d-4067fbaf6c27" (UID: "e9354cff-dc48-4dbd-be0d-4067fbaf6c27"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:15:20.993104 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:20.993071    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9354cff-dc48-4dbd-be0d-4067fbaf6c27-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:15:20.993104 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:20.993098    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e9354cff-dc48-4dbd-be0d-4067fbaf6c27-cabundle-cert\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:15:20.993104 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:20.993109    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mhgp\" (UniqueName: \"kubernetes.io/projected/e9354cff-dc48-4dbd-be0d-4067fbaf6c27-kube-api-access-5mhgp\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:15:20.993320 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:20.993118    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9354cff-dc48-4dbd-be0d-4067fbaf6c27-kserve-provision-location\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:15:20.993320 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:20.993127    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9354cff-dc48-4dbd-be0d-4067fbaf6c27-proxy-tls\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:15:21.003620 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:21.003593    2585 generic.go:356] "Generic (PLEG): container finished" podID="e9354cff-dc48-4dbd-be0d-4067fbaf6c27" containerID="7cf5beed0a1bda93430850ff544787f2ef1b386ae10792472f33d864b6b8d716" exitCode=0
Sep 17 18:15:21.003742 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:21.003654    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dz58c" event={"ID":"e9354cff-dc48-4dbd-be0d-4067fbaf6c27","Type":"ContainerDied","Data":"7cf5beed0a1bda93430850ff544787f2ef1b386ae10792472f33d864b6b8d716"}
Sep 17 18:15:21.003742 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:21.003703    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dz58c" event={"ID":"e9354cff-dc48-4dbd-be0d-4067fbaf6c27","Type":"ContainerDied","Data":"ac789f685dd9131a7dbcb7de9c921450ed6eced29486752e79704c22172f39fe"}
Sep 17 18:15:21.003742 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:21.003729    2585 scope.go:117] "RemoveContainer" containerID="50b052be424a47971e3d48f801e56ef10c00bac6f107a34a63444f12c25e6431"
Sep 17 18:15:21.003852 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:21.003657    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dz58c"
Sep 17 18:15:21.011530 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:21.011517    2585 scope.go:117] "RemoveContainer" containerID="7cf5beed0a1bda93430850ff544787f2ef1b386ae10792472f33d864b6b8d716"
Sep 17 18:15:21.018404 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:21.018388    2585 scope.go:117] "RemoveContainer" containerID="c7906b4f001b76f6cfea1f94a9bb750689d9812dce6d83462e102057c4faf3b4"
Sep 17 18:15:21.026190 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:21.026169    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dz58c"]
Sep 17 18:15:21.029001 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:21.028982    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-dz58c"]
Sep 17 18:15:21.030841 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:21.030819    2585 scope.go:117] "RemoveContainer" containerID="50b052be424a47971e3d48f801e56ef10c00bac6f107a34a63444f12c25e6431"
Sep 17 18:15:21.031096 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:15:21.031077    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50b052be424a47971e3d48f801e56ef10c00bac6f107a34a63444f12c25e6431\": container with ID starting with 50b052be424a47971e3d48f801e56ef10c00bac6f107a34a63444f12c25e6431 not found: ID does not exist" containerID="50b052be424a47971e3d48f801e56ef10c00bac6f107a34a63444f12c25e6431"
Sep 17 18:15:21.031152 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:21.031105    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50b052be424a47971e3d48f801e56ef10c00bac6f107a34a63444f12c25e6431"} err="failed to get container status \"50b052be424a47971e3d48f801e56ef10c00bac6f107a34a63444f12c25e6431\": rpc error: code = NotFound desc = could not find container \"50b052be424a47971e3d48f801e56ef10c00bac6f107a34a63444f12c25e6431\": container with ID starting with 50b052be424a47971e3d48f801e56ef10c00bac6f107a34a63444f12c25e6431 not found: ID does not exist"
Sep 17 18:15:21.031152 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:21.031122    2585 scope.go:117] "RemoveContainer" containerID="7cf5beed0a1bda93430850ff544787f2ef1b386ae10792472f33d864b6b8d716"
Sep 17 18:15:21.031401 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:15:21.031362    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cf5beed0a1bda93430850ff544787f2ef1b386ae10792472f33d864b6b8d716\": container with ID starting with 7cf5beed0a1bda93430850ff544787f2ef1b386ae10792472f33d864b6b8d716 not found: ID does not exist" containerID="7cf5beed0a1bda93430850ff544787f2ef1b386ae10792472f33d864b6b8d716"
Sep 17 18:15:21.031401 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:21.031381    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cf5beed0a1bda93430850ff544787f2ef1b386ae10792472f33d864b6b8d716"} err="failed to get container status \"7cf5beed0a1bda93430850ff544787f2ef1b386ae10792472f33d864b6b8d716\": rpc error: code = NotFound desc = could not find container \"7cf5beed0a1bda93430850ff544787f2ef1b386ae10792472f33d864b6b8d716\": container with ID starting with 7cf5beed0a1bda93430850ff544787f2ef1b386ae10792472f33d864b6b8d716 not found: ID does not exist"
Sep 17 18:15:21.031401 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:21.031394    2585 scope.go:117] "RemoveContainer" containerID="c7906b4f001b76f6cfea1f94a9bb750689d9812dce6d83462e102057c4faf3b4"
Sep 17 18:15:21.031673 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:15:21.031658    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7906b4f001b76f6cfea1f94a9bb750689d9812dce6d83462e102057c4faf3b4\": container with ID starting with c7906b4f001b76f6cfea1f94a9bb750689d9812dce6d83462e102057c4faf3b4 not found: ID does not exist" containerID="c7906b4f001b76f6cfea1f94a9bb750689d9812dce6d83462e102057c4faf3b4"
Sep 17 18:15:21.031715 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:21.031676    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7906b4f001b76f6cfea1f94a9bb750689d9812dce6d83462e102057c4faf3b4"} err="failed to get container status \"c7906b4f001b76f6cfea1f94a9bb750689d9812dce6d83462e102057c4faf3b4\": rpc error: code = NotFound desc = could not find container \"c7906b4f001b76f6cfea1f94a9bb750689d9812dce6d83462e102057c4faf3b4\": container with ID starting with c7906b4f001b76f6cfea1f94a9bb750689d9812dce6d83462e102057c4faf3b4 not found: ID does not exist"
Sep 17 18:15:21.112159 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:21.112133    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx"]
Sep 17 18:15:21.115045 ip-10-0-133-139 kubenswrapper[2585]: W0917 18:15:21.115010    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcafe1fd8_26c9_4d21_97a5_0785fa7a4123.slice/crio-5c72ed9cf71aebe58f4ad2a14811eeb1baa796ee2626c4c3536ba7472bd660b2 WatchSource:0}: Error finding container 5c72ed9cf71aebe58f4ad2a14811eeb1baa796ee2626c4c3536ba7472bd660b2: Status 404 returned error can't find the container with id 5c72ed9cf71aebe58f4ad2a14811eeb1baa796ee2626c4c3536ba7472bd660b2
Sep 17 18:15:21.490159 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:21.490080    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:21.513333 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:21.513302    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-7mjlq"]
Sep 17 18:15:21.513705 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:21.513575    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9354cff-dc48-4dbd-be0d-4067fbaf6c27" containerName="kube-rbac-proxy"
Sep 17 18:15:21.513705 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:21.513586    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9354cff-dc48-4dbd-be0d-4067fbaf6c27" containerName="kube-rbac-proxy"
Sep 17 18:15:21.513705 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:21.513597    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9354cff-dc48-4dbd-be0d-4067fbaf6c27" containerName="storage-initializer"
Sep 17 18:15:21.513705 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:21.513603    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9354cff-dc48-4dbd-be0d-4067fbaf6c27" containerName="storage-initializer"
Sep 17 18:15:21.513705 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:21.513612    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9354cff-dc48-4dbd-be0d-4067fbaf6c27" containerName="kserve-container"
Sep 17 18:15:21.513705 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:21.513617    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9354cff-dc48-4dbd-be0d-4067fbaf6c27" containerName="kserve-container"
Sep 17 18:15:21.513705 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:21.513659    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9354cff-dc48-4dbd-be0d-4067fbaf6c27" containerName="kube-rbac-proxy"
Sep 17 18:15:21.513705 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:21.513666    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9354cff-dc48-4dbd-be0d-4067fbaf6c27" containerName="kserve-container"
Sep 17 18:15:21.516484 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:21.516469    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-7mjlq"
Sep 17 18:15:21.518895 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:21.518875    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
Sep 17 18:15:21.519011 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:21.518928    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
Sep 17 18:15:21.528240 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:21.528219    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-7mjlq"]
Sep 17 18:15:21.596757 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:21.596721    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f08897f-73ad-448f-8db7-4fdd6d94a417-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-7mjlq\" (UID: \"7f08897f-73ad-448f-8db7-4fdd6d94a417\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-7mjlq"
Sep 17 18:15:21.596917 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:21.596770    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f08897f-73ad-448f-8db7-4fdd6d94a417-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-7mjlq\" (UID: \"7f08897f-73ad-448f-8db7-4fdd6d94a417\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-7mjlq"
Sep 17 18:15:21.596917 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:21.596825    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f08897f-73ad-448f-8db7-4fdd6d94a417-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-7mjlq\" (UID: \"7f08897f-73ad-448f-8db7-4fdd6d94a417\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-7mjlq"
Sep 17 18:15:21.596917 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:21.596855    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f08897f-73ad-448f-8db7-4fdd6d94a417-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-7mjlq\" (UID: \"7f08897f-73ad-448f-8db7-4fdd6d94a417\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-7mjlq"
Sep 17 18:15:21.596917 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:21.596904    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv648\" (UniqueName: \"kubernetes.io/projected/7f08897f-73ad-448f-8db7-4fdd6d94a417-kube-api-access-wv648\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-7mjlq\" (UID: \"7f08897f-73ad-448f-8db7-4fdd6d94a417\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-7mjlq"
Sep 17 18:15:21.698075 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:21.698045    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv648\" (UniqueName: \"kubernetes.io/projected/7f08897f-73ad-448f-8db7-4fdd6d94a417-kube-api-access-wv648\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-7mjlq\" (UID: \"7f08897f-73ad-448f-8db7-4fdd6d94a417\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-7mjlq"
Sep 17 18:15:21.698215 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:21.698092    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f08897f-73ad-448f-8db7-4fdd6d94a417-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-7mjlq\" (UID: \"7f08897f-73ad-448f-8db7-4fdd6d94a417\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-7mjlq"
Sep 17 18:15:21.698215 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:21.698115    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f08897f-73ad-448f-8db7-4fdd6d94a417-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-7mjlq\" (UID: \"7f08897f-73ad-448f-8db7-4fdd6d94a417\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-7mjlq"
Sep 17 18:15:21.698215 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:21.698135    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f08897f-73ad-448f-8db7-4fdd6d94a417-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-7mjlq\" (UID: \"7f08897f-73ad-448f-8db7-4fdd6d94a417\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-7mjlq"
Sep 17 18:15:21.698215 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:21.698156    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f08897f-73ad-448f-8db7-4fdd6d94a417-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-7mjlq\" (UID: \"7f08897f-73ad-448f-8db7-4fdd6d94a417\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-7mjlq"
Sep 17 18:15:21.698445 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:15:21.698310    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-serving-cert: secret "isvc-xgboost-runtime-predictor-serving-cert" not found
Sep 17 18:15:21.698445 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:15:21.698370    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f08897f-73ad-448f-8db7-4fdd6d94a417-proxy-tls podName:7f08897f-73ad-448f-8db7-4fdd6d94a417 nodeName:}" failed. No retries permitted until 2026-09-17 18:15:22.19834825 +0000 UTC m=+2458.612138666 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7f08897f-73ad-448f-8db7-4fdd6d94a417-proxy-tls") pod "isvc-xgboost-runtime-predictor-54dd488c85-7mjlq" (UID: "7f08897f-73ad-448f-8db7-4fdd6d94a417") : secret "isvc-xgboost-runtime-predictor-serving-cert" not found
Sep 17 18:15:21.698561 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:21.698529    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f08897f-73ad-448f-8db7-4fdd6d94a417-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-7mjlq\" (UID: \"7f08897f-73ad-448f-8db7-4fdd6d94a417\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-7mjlq"
Sep 17 18:15:21.698921 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:21.698904    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f08897f-73ad-448f-8db7-4fdd6d94a417-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-7mjlq\" (UID: \"7f08897f-73ad-448f-8db7-4fdd6d94a417\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-7mjlq"
Sep 17 18:15:21.698959 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:21.698904    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f08897f-73ad-448f-8db7-4fdd6d94a417-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-7mjlq\" (UID: \"7f08897f-73ad-448f-8db7-4fdd6d94a417\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-7mjlq"
Sep 17 18:15:21.708994 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:21.708972    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv648\" (UniqueName: \"kubernetes.io/projected/7f08897f-73ad-448f-8db7-4fdd6d94a417-kube-api-access-wv648\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-7mjlq\" (UID: \"7f08897f-73ad-448f-8db7-4fdd6d94a417\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-7mjlq"
Sep 17 18:15:21.822196 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:21.822160    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:22.008843 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:22.008804    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx" event={"ID":"cafe1fd8-26c9-4d21-97a5-0785fa7a4123","Type":"ContainerStarted","Data":"6ce4b2be269210560dd891b51b31a33cf0042244b2255773e6a5a3f640b8029b"}
Sep 17 18:15:22.009051 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:22.008872    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx" event={"ID":"cafe1fd8-26c9-4d21-97a5-0785fa7a4123","Type":"ContainerStarted","Data":"5c72ed9cf71aebe58f4ad2a14811eeb1baa796ee2626c4c3536ba7472bd660b2"}
Sep 17 18:15:22.104170 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:22.104087    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9354cff-dc48-4dbd-be0d-4067fbaf6c27" path="/var/lib/kubelet/pods/e9354cff-dc48-4dbd-be0d-4067fbaf6c27/volumes"
Sep 17 18:15:22.202118 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:22.202086    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f08897f-73ad-448f-8db7-4fdd6d94a417-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-7mjlq\" (UID: \"7f08897f-73ad-448f-8db7-4fdd6d94a417\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-7mjlq"
Sep 17 18:15:22.204428 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:22.204402    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f08897f-73ad-448f-8db7-4fdd6d94a417-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-7mjlq\" (UID: \"7f08897f-73ad-448f-8db7-4fdd6d94a417\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-7mjlq"
Sep 17 18:15:22.426438 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:22.426356    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-7mjlq"
Sep 17 18:15:22.700663 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:22.700638    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-7mjlq"]
Sep 17 18:15:22.702700 ip-10-0-133-139 kubenswrapper[2585]: W0917 18:15:22.702675    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f08897f_73ad_448f_8db7_4fdd6d94a417.slice/crio-4790a1207a07565a4d9a0cb5e98ebffb0bb812e117ba5b60dcc94edecac3c574 WatchSource:0}: Error finding container 4790a1207a07565a4d9a0cb5e98ebffb0bb812e117ba5b60dcc94edecac3c574: Status 404 returned error can't find the container with id 4790a1207a07565a4d9a0cb5e98ebffb0bb812e117ba5b60dcc94edecac3c574
Sep 17 18:15:23.015541 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:23.015507    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-7mjlq" event={"ID":"7f08897f-73ad-448f-8db7-4fdd6d94a417","Type":"ContainerStarted","Data":"c266a688d9300dc6dfedc845b263df5145a367f87fec5db49cbcaa6699e2946f"}
Sep 17 18:15:23.015755 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:23.015575    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-7mjlq" event={"ID":"7f08897f-73ad-448f-8db7-4fdd6d94a417","Type":"ContainerStarted","Data":"4790a1207a07565a4d9a0cb5e98ebffb0bb812e117ba5b60dcc94edecac3c574"}
Sep 17 18:15:26.028167 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:26.028130    2585 generic.go:356] "Generic (PLEG): container finished" podID="cafe1fd8-26c9-4d21-97a5-0785fa7a4123" containerID="6ce4b2be269210560dd891b51b31a33cf0042244b2255773e6a5a3f640b8029b" exitCode=0
Sep 17 18:15:26.028844 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:26.028200    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx" event={"ID":"cafe1fd8-26c9-4d21-97a5-0785fa7a4123","Type":"ContainerDied","Data":"6ce4b2be269210560dd891b51b31a33cf0042244b2255773e6a5a3f640b8029b"}
Sep 17 18:15:27.034114 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:27.034076    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx" event={"ID":"cafe1fd8-26c9-4d21-97a5-0785fa7a4123","Type":"ContainerStarted","Data":"2a6d5f217a021871684cec76e0e91159d3f419fdff654d564bf7f8018846c14f"}
Sep 17 18:15:27.034629 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:27.034153    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx" event={"ID":"cafe1fd8-26c9-4d21-97a5-0785fa7a4123","Type":"ContainerStarted","Data":"be4c6953b6138da084688cd3f4073593e87692aecf2f06a2e0f505245433110b"}
Sep 17 18:15:27.034629 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:27.034433    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx"
Sep 17 18:15:27.034629 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:27.034485    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx"
Sep 17 18:15:27.035562 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:27.035538    2585 generic.go:356] "Generic (PLEG): container finished" podID="7f08897f-73ad-448f-8db7-4fdd6d94a417" containerID="c266a688d9300dc6dfedc845b263df5145a367f87fec5db49cbcaa6699e2946f" exitCode=0
Sep 17 18:15:27.035671 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:27.035564    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-7mjlq" event={"ID":"7f08897f-73ad-448f-8db7-4fdd6d94a417","Type":"ContainerDied","Data":"c266a688d9300dc6dfedc845b263df5145a367f87fec5db49cbcaa6699e2946f"}
Sep 17 18:15:27.035985 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:27.035963    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx" podUID="cafe1fd8-26c9-4d21-97a5-0785fa7a4123" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 18:15:27.036651 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:27.036637    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:15:27.057665 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:27.057613    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx" podStartSLOduration=3.212586157 podStartE2EDuration="7.057592998s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.845006841s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:15:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:15:27.056879944 +0000 UTC m=+2463.470670375" watchObservedRunningTime="2026-09-17 18:15:27.057592998 +0000 UTC m=+2463.471383439"
Sep 17 18:15:28.040265 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:28.040227    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx" podUID="cafe1fd8-26c9-4d21-97a5-0785fa7a4123" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 18:15:33.044583 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:33.044556    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx"
Sep 17 18:15:33.045052 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:33.045027    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx" podUID="cafe1fd8-26c9-4d21-97a5-0785fa7a4123" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 18:15:43.045075 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:43.045024    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx" podUID="cafe1fd8-26c9-4d21-97a5-0785fa7a4123" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 18:15:47.121287 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:47.121194    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-7mjlq" event={"ID":"7f08897f-73ad-448f-8db7-4fdd6d94a417","Type":"ContainerStarted","Data":"58d7054be3dbd47f615f7e350970aae4a29ecb8fdb8a83f8a0c2969a0f69e2ca"}
Sep 17 18:15:47.121287 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:47.121262    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-7mjlq" event={"ID":"7f08897f-73ad-448f-8db7-4fdd6d94a417","Type":"ContainerStarted","Data":"64ab8cac8e273f1a9a71171e2d2d0a5be2e85a33df31cc57edd2ae528edfa1fb"}
Sep 17 18:15:47.121829 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:47.121463    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-7mjlq"
Sep 17 18:15:47.142519 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:47.142467    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-7mjlq" podStartSLOduration=2.7299192100000003 podStartE2EDuration="26.142454024s" totalImagesPullingTime="19.711853187s" totalInitContainerRuntime="3.700681627s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:15:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:15:47.140259999 +0000 UTC m=+2483.554050449" watchObservedRunningTime="2026-09-17 18:15:47.142454024 +0000 UTC m=+2483.556244463"
Sep 17 18:15:48.124829 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:48.124796    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-7mjlq"
Sep 17 18:15:48.126204 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:48.126176    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-7mjlq" podUID="7f08897f-73ad-448f-8db7-4fdd6d94a417" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 18:15:49.128616 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:49.128578    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-7mjlq" podUID="7f08897f-73ad-448f-8db7-4fdd6d94a417" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 18:15:53.045749 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:53.045710    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx" podUID="cafe1fd8-26c9-4d21-97a5-0785fa7a4123" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 18:15:54.132626 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:54.132596    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-7mjlq"
Sep 17 18:15:54.137700 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:15:54.137469    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-7mjlq" podUID="7f08897f-73ad-448f-8db7-4fdd6d94a417" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 18:16:03.045144 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:16:03.045104    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx" podUID="cafe1fd8-26c9-4d21-97a5-0785fa7a4123" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 18:16:04.133708 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:16:04.133674    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-7mjlq" podUID="7f08897f-73ad-448f-8db7-4fdd6d94a417" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 18:16:13.045166 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:16:13.045127    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx" podUID="cafe1fd8-26c9-4d21-97a5-0785fa7a4123" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 18:16:14.133700 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:16:14.133662    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-7mjlq" podUID="7f08897f-73ad-448f-8db7-4fdd6d94a417" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 18:16:23.045328 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:16:23.045246    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx" podUID="cafe1fd8-26c9-4d21-97a5-0785fa7a4123" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 18:16:24.133841 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:16:24.133799    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-7mjlq" podUID="7f08897f-73ad-448f-8db7-4fdd6d94a417" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 18:16:33.045017 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:16:33.044968    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx" podUID="cafe1fd8-26c9-4d21-97a5-0785fa7a4123" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 18:16:34.133747 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:16:34.133703    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-7mjlq" podUID="7f08897f-73ad-448f-8db7-4fdd6d94a417" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 18:16:43.045083 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:16:43.045038    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx" podUID="cafe1fd8-26c9-4d21-97a5-0785fa7a4123" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 18:16:44.133041 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:16:44.132999    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-7mjlq" podUID="7f08897f-73ad-448f-8db7-4fdd6d94a417" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 18:16:48.100114 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:16:48.100070    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx" podUID="cafe1fd8-26c9-4d21-97a5-0785fa7a4123" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 18:16:50.442736 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:16:50.442691    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:54.134728 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:16:54.134702    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-7mjlq"
Sep 17 18:16:58.100058 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:16:58.100018    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx" podUID="cafe1fd8-26c9-4d21-97a5-0785fa7a4123" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 18:17:02.078172 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:02.078137    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-7mjlq"]
Sep 17 18:17:02.078627 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:02.078420    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-7mjlq" podUID="7f08897f-73ad-448f-8db7-4fdd6d94a417" containerName="kserve-container" containerID="cri-o://64ab8cac8e273f1a9a71171e2d2d0a5be2e85a33df31cc57edd2ae528edfa1fb" gracePeriod=30
Sep 17 18:17:02.078627 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:02.078455    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-7mjlq" podUID="7f08897f-73ad-448f-8db7-4fdd6d94a417" containerName="kube-rbac-proxy" containerID="cri-o://58d7054be3dbd47f615f7e350970aae4a29ecb8fdb8a83f8a0c2969a0f69e2ca" gracePeriod=30
Sep 17 18:17:02.086399 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:02.086371    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:02.163464 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:02.163424    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:02.388653 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:02.388562    2585 generic.go:356] "Generic (PLEG): container finished" podID="7f08897f-73ad-448f-8db7-4fdd6d94a417" containerID="58d7054be3dbd47f615f7e350970aae4a29ecb8fdb8a83f8a0c2969a0f69e2ca" exitCode=2
Sep 17 18:17:02.388653 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:02.388630    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-7mjlq" event={"ID":"7f08897f-73ad-448f-8db7-4fdd6d94a417","Type":"ContainerDied","Data":"58d7054be3dbd47f615f7e350970aae4a29ecb8fdb8a83f8a0c2969a0f69e2ca"}
Sep 17 18:17:04.129048 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:04.129005    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-7mjlq" podUID="7f08897f-73ad-448f-8db7-4fdd6d94a417" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Sep 17 18:17:04.133783 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:04.133759    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-7mjlq" podUID="7f08897f-73ad-448f-8db7-4fdd6d94a417" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 18:17:06.215227 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:06.215204    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-7mjlq"
Sep 17 18:17:06.286570 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:06.286479    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f08897f-73ad-448f-8db7-4fdd6d94a417-kserve-provision-location\") pod \"7f08897f-73ad-448f-8db7-4fdd6d94a417\" (UID: \"7f08897f-73ad-448f-8db7-4fdd6d94a417\") "
Sep 17 18:17:06.286570 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:06.286525    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wv648\" (UniqueName: \"kubernetes.io/projected/7f08897f-73ad-448f-8db7-4fdd6d94a417-kube-api-access-wv648\") pod \"7f08897f-73ad-448f-8db7-4fdd6d94a417\" (UID: \"7f08897f-73ad-448f-8db7-4fdd6d94a417\") "
Sep 17 18:17:06.286570 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:06.286543    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f08897f-73ad-448f-8db7-4fdd6d94a417-kube-rbac-proxy-sar-config\") pod \"7f08897f-73ad-448f-8db7-4fdd6d94a417\" (UID: \"7f08897f-73ad-448f-8db7-4fdd6d94a417\") "
Sep 17 18:17:06.286570 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:06.286566    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f08897f-73ad-448f-8db7-4fdd6d94a417-proxy-tls\") pod \"7f08897f-73ad-448f-8db7-4fdd6d94a417\" (UID: \"7f08897f-73ad-448f-8db7-4fdd6d94a417\") "
Sep 17 18:17:06.286895 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:06.286596    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f08897f-73ad-448f-8db7-4fdd6d94a417-cabundle-cert\") pod \"7f08897f-73ad-448f-8db7-4fdd6d94a417\" (UID: \"7f08897f-73ad-448f-8db7-4fdd6d94a417\") "
Sep 17 18:17:06.286952 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:06.286862    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f08897f-73ad-448f-8db7-4fdd6d94a417-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7f08897f-73ad-448f-8db7-4fdd6d94a417" (UID: "7f08897f-73ad-448f-8db7-4fdd6d94a417"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:17:06.286995 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:06.286975    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f08897f-73ad-448f-8db7-4fdd6d94a417-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7f08897f-73ad-448f-8db7-4fdd6d94a417" (UID: "7f08897f-73ad-448f-8db7-4fdd6d94a417"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:17:06.287037 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:06.287018    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f08897f-73ad-448f-8db7-4fdd6d94a417-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7f08897f-73ad-448f-8db7-4fdd6d94a417" (UID: "7f08897f-73ad-448f-8db7-4fdd6d94a417"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:17:06.288705 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:06.288684    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f08897f-73ad-448f-8db7-4fdd6d94a417-kube-api-access-wv648" (OuterVolumeSpecName: "kube-api-access-wv648") pod "7f08897f-73ad-448f-8db7-4fdd6d94a417" (UID: "7f08897f-73ad-448f-8db7-4fdd6d94a417"). InnerVolumeSpecName "kube-api-access-wv648". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:17:06.288778 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:06.288738    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f08897f-73ad-448f-8db7-4fdd6d94a417-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7f08897f-73ad-448f-8db7-4fdd6d94a417" (UID: "7f08897f-73ad-448f-8db7-4fdd6d94a417"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:17:06.387016 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:06.386989    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f08897f-73ad-448f-8db7-4fdd6d94a417-kserve-provision-location\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:17:06.387016 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:06.387012    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv648\" (UniqueName: \"kubernetes.io/projected/7f08897f-73ad-448f-8db7-4fdd6d94a417-kube-api-access-wv648\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:17:06.387016 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:06.387021    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7f08897f-73ad-448f-8db7-4fdd6d94a417-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:17:06.387231 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:06.387031    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f08897f-73ad-448f-8db7-4fdd6d94a417-proxy-tls\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:17:06.387231 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:06.387040    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f08897f-73ad-448f-8db7-4fdd6d94a417-cabundle-cert\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:17:06.403981 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:06.403952    2585 generic.go:356] "Generic (PLEG): container finished" podID="7f08897f-73ad-448f-8db7-4fdd6d94a417" containerID="64ab8cac8e273f1a9a71171e2d2d0a5be2e85a33df31cc57edd2ae528edfa1fb" exitCode=0
Sep 17 18:17:06.404113 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:06.403985    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-7mjlq" event={"ID":"7f08897f-73ad-448f-8db7-4fdd6d94a417","Type":"ContainerDied","Data":"64ab8cac8e273f1a9a71171e2d2d0a5be2e85a33df31cc57edd2ae528edfa1fb"}
Sep 17 18:17:06.404113 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:06.404020    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-7mjlq"
Sep 17 18:17:06.404113 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:06.404033    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-7mjlq" event={"ID":"7f08897f-73ad-448f-8db7-4fdd6d94a417","Type":"ContainerDied","Data":"4790a1207a07565a4d9a0cb5e98ebffb0bb812e117ba5b60dcc94edecac3c574"}
Sep 17 18:17:06.404113 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:06.404060    2585 scope.go:117] "RemoveContainer" containerID="58d7054be3dbd47f615f7e350970aae4a29ecb8fdb8a83f8a0c2969a0f69e2ca"
Sep 17 18:17:06.412733 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:06.412602    2585 scope.go:117] "RemoveContainer" containerID="64ab8cac8e273f1a9a71171e2d2d0a5be2e85a33df31cc57edd2ae528edfa1fb"
Sep 17 18:17:06.419648 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:06.419626    2585 scope.go:117] "RemoveContainer" containerID="c266a688d9300dc6dfedc845b263df5145a367f87fec5db49cbcaa6699e2946f"
Sep 17 18:17:06.426828 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:06.426810    2585 scope.go:117] "RemoveContainer" containerID="58d7054be3dbd47f615f7e350970aae4a29ecb8fdb8a83f8a0c2969a0f69e2ca"
Sep 17 18:17:06.427109 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:17:06.427090    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58d7054be3dbd47f615f7e350970aae4a29ecb8fdb8a83f8a0c2969a0f69e2ca\": container with ID starting with 58d7054be3dbd47f615f7e350970aae4a29ecb8fdb8a83f8a0c2969a0f69e2ca not found: ID does not exist" containerID="58d7054be3dbd47f615f7e350970aae4a29ecb8fdb8a83f8a0c2969a0f69e2ca"
Sep 17 18:17:06.427173 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:06.427116    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58d7054be3dbd47f615f7e350970aae4a29ecb8fdb8a83f8a0c2969a0f69e2ca"} err="failed to get container status \"58d7054be3dbd47f615f7e350970aae4a29ecb8fdb8a83f8a0c2969a0f69e2ca\": rpc error: code = NotFound desc = could not find container \"58d7054be3dbd47f615f7e350970aae4a29ecb8fdb8a83f8a0c2969a0f69e2ca\": container with ID starting with 58d7054be3dbd47f615f7e350970aae4a29ecb8fdb8a83f8a0c2969a0f69e2ca not found: ID does not exist"
Sep 17 18:17:06.427173 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:06.427133    2585 scope.go:117] "RemoveContainer" containerID="64ab8cac8e273f1a9a71171e2d2d0a5be2e85a33df31cc57edd2ae528edfa1fb"
Sep 17 18:17:06.427270 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:06.427192    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-7mjlq"]
Sep 17 18:17:06.427400 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:17:06.427383    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64ab8cac8e273f1a9a71171e2d2d0a5be2e85a33df31cc57edd2ae528edfa1fb\": container with ID starting with 64ab8cac8e273f1a9a71171e2d2d0a5be2e85a33df31cc57edd2ae528edfa1fb not found: ID does not exist" containerID="64ab8cac8e273f1a9a71171e2d2d0a5be2e85a33df31cc57edd2ae528edfa1fb"
Sep 17 18:17:06.427443 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:06.427408    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64ab8cac8e273f1a9a71171e2d2d0a5be2e85a33df31cc57edd2ae528edfa1fb"} err="failed to get container status \"64ab8cac8e273f1a9a71171e2d2d0a5be2e85a33df31cc57edd2ae528edfa1fb\": rpc error: code = NotFound desc = could not find container \"64ab8cac8e273f1a9a71171e2d2d0a5be2e85a33df31cc57edd2ae528edfa1fb\": container with ID starting with 64ab8cac8e273f1a9a71171e2d2d0a5be2e85a33df31cc57edd2ae528edfa1fb not found: ID does not exist"
Sep 17 18:17:06.427443 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:06.427423    2585 scope.go:117] "RemoveContainer" containerID="c266a688d9300dc6dfedc845b263df5145a367f87fec5db49cbcaa6699e2946f"
Sep 17 18:17:06.427678 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:17:06.427658    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c266a688d9300dc6dfedc845b263df5145a367f87fec5db49cbcaa6699e2946f\": container with ID starting with c266a688d9300dc6dfedc845b263df5145a367f87fec5db49cbcaa6699e2946f not found: ID does not exist" containerID="c266a688d9300dc6dfedc845b263df5145a367f87fec5db49cbcaa6699e2946f"
Sep 17 18:17:06.427722 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:06.427683    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c266a688d9300dc6dfedc845b263df5145a367f87fec5db49cbcaa6699e2946f"} err="failed to get container status \"c266a688d9300dc6dfedc845b263df5145a367f87fec5db49cbcaa6699e2946f\": rpc error: code = NotFound desc = could not find container \"c266a688d9300dc6dfedc845b263df5145a367f87fec5db49cbcaa6699e2946f\": container with ID starting with c266a688d9300dc6dfedc845b263df5145a367f87fec5db49cbcaa6699e2946f not found: ID does not exist"
Sep 17 18:17:06.430981 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:06.430961    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-7mjlq"]
Sep 17 18:17:07.244900 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:07.244862    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:07.441595 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:07.441553    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:08.104059 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:08.104023    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f08897f-73ad-448f-8db7-4fdd6d94a417" path="/var/lib/kubelet/pods/7f08897f-73ad-448f-8db7-4fdd6d94a417/volumes"
Sep 17 18:17:08.104403 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:08.104390    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx"
Sep 17 18:17:15.649074 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:15.649038    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx"]
Sep 17 18:17:15.649559 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:15.649334    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx" podUID="cafe1fd8-26c9-4d21-97a5-0785fa7a4123" containerName="kserve-container" containerID="cri-o://be4c6953b6138da084688cd3f4073593e87692aecf2f06a2e0f505245433110b" gracePeriod=30
Sep 17 18:17:15.649559 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:15.649384    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx" podUID="cafe1fd8-26c9-4d21-97a5-0785fa7a4123" containerName="kube-rbac-proxy" containerID="cri-o://2a6d5f217a021871684cec76e0e91159d3f419fdff654d564bf7f8018846c14f" gracePeriod=30
Sep 17 18:17:15.665228 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:15.665204    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:16.443045 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:16.443014    2585 generic.go:356] "Generic (PLEG): container finished" podID="cafe1fd8-26c9-4d21-97a5-0785fa7a4123" containerID="2a6d5f217a021871684cec76e0e91159d3f419fdff654d564bf7f8018846c14f" exitCode=2
Sep 17 18:17:16.443045 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:16.443049    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx" event={"ID":"cafe1fd8-26c9-4d21-97a5-0785fa7a4123","Type":"ContainerDied","Data":"2a6d5f217a021871684cec76e0e91159d3f419fdff654d564bf7f8018846c14f"}
Sep 17 18:17:18.040506 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:18.040455    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx" podUID="cafe1fd8-26c9-4d21-97a5-0785fa7a4123" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.39:8643/healthz\": dial tcp 10.133.0.39:8643: connect: connection refused"
Sep 17 18:17:18.100480 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:18.100441    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx" podUID="cafe1fd8-26c9-4d21-97a5-0785fa7a4123" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 18:17:20.781327 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:20.781280    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:20.920362 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:20.920325    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:21.083823 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:21.083802    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx"
Sep 17 18:17:21.189112 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:21.189083    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cafe1fd8-26c9-4d21-97a5-0785fa7a4123-kserve-provision-location\") pod \"cafe1fd8-26c9-4d21-97a5-0785fa7a4123\" (UID: \"cafe1fd8-26c9-4d21-97a5-0785fa7a4123\") "
Sep 17 18:17:21.189308 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:21.189120    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cafe1fd8-26c9-4d21-97a5-0785fa7a4123-kube-rbac-proxy-sar-config\") pod \"cafe1fd8-26c9-4d21-97a5-0785fa7a4123\" (UID: \"cafe1fd8-26c9-4d21-97a5-0785fa7a4123\") "
Sep 17 18:17:21.189308 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:21.189140    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cafe1fd8-26c9-4d21-97a5-0785fa7a4123-proxy-tls\") pod \"cafe1fd8-26c9-4d21-97a5-0785fa7a4123\" (UID: \"cafe1fd8-26c9-4d21-97a5-0785fa7a4123\") "
Sep 17 18:17:21.189308 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:21.189157    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cafe1fd8-26c9-4d21-97a5-0785fa7a4123-cabundle-cert\") pod \"cafe1fd8-26c9-4d21-97a5-0785fa7a4123\" (UID: \"cafe1fd8-26c9-4d21-97a5-0785fa7a4123\") "
Sep 17 18:17:21.189308 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:21.189186    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shpzz\" (UniqueName: \"kubernetes.io/projected/cafe1fd8-26c9-4d21-97a5-0785fa7a4123-kube-api-access-shpzz\") pod \"cafe1fd8-26c9-4d21-97a5-0785fa7a4123\" (UID: \"cafe1fd8-26c9-4d21-97a5-0785fa7a4123\") "
Sep 17 18:17:21.189572 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:21.189514    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cafe1fd8-26c9-4d21-97a5-0785fa7a4123-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cafe1fd8-26c9-4d21-97a5-0785fa7a4123" (UID: "cafe1fd8-26c9-4d21-97a5-0785fa7a4123"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:17:21.189572 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:21.189514    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cafe1fd8-26c9-4d21-97a5-0785fa7a4123-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cafe1fd8-26c9-4d21-97a5-0785fa7a4123" (UID: "cafe1fd8-26c9-4d21-97a5-0785fa7a4123"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:17:21.189572 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:21.189553    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cafe1fd8-26c9-4d21-97a5-0785fa7a4123-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cafe1fd8-26c9-4d21-97a5-0785fa7a4123" (UID: "cafe1fd8-26c9-4d21-97a5-0785fa7a4123"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:17:21.191290 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:21.191265    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cafe1fd8-26c9-4d21-97a5-0785fa7a4123-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cafe1fd8-26c9-4d21-97a5-0785fa7a4123" (UID: "cafe1fd8-26c9-4d21-97a5-0785fa7a4123"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:17:21.191351 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:21.191322    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cafe1fd8-26c9-4d21-97a5-0785fa7a4123-kube-api-access-shpzz" (OuterVolumeSpecName: "kube-api-access-shpzz") pod "cafe1fd8-26c9-4d21-97a5-0785fa7a4123" (UID: "cafe1fd8-26c9-4d21-97a5-0785fa7a4123"). InnerVolumeSpecName "kube-api-access-shpzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:17:21.289829 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:21.289798    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cafe1fd8-26c9-4d21-97a5-0785fa7a4123-kserve-provision-location\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:17:21.289829 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:21.289825    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cafe1fd8-26c9-4d21-97a5-0785fa7a4123-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:17:21.289829 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:21.289835    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cafe1fd8-26c9-4d21-97a5-0785fa7a4123-proxy-tls\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:17:21.290052 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:21.289843    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cafe1fd8-26c9-4d21-97a5-0785fa7a4123-cabundle-cert\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:17:21.290052 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:21.289852    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shpzz\" (UniqueName: \"kubernetes.io/projected/cafe1fd8-26c9-4d21-97a5-0785fa7a4123-kube-api-access-shpzz\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:17:21.461549 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:21.461470    2585 generic.go:356] "Generic (PLEG): container finished" podID="cafe1fd8-26c9-4d21-97a5-0785fa7a4123" containerID="be4c6953b6138da084688cd3f4073593e87692aecf2f06a2e0f505245433110b" exitCode=0
Sep 17 18:17:21.461549 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:21.461536    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx" event={"ID":"cafe1fd8-26c9-4d21-97a5-0785fa7a4123","Type":"ContainerDied","Data":"be4c6953b6138da084688cd3f4073593e87692aecf2f06a2e0f505245433110b"}
Sep 17 18:17:21.461723 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:21.461553    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx"
Sep 17 18:17:21.461723 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:21.461587    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx" event={"ID":"cafe1fd8-26c9-4d21-97a5-0785fa7a4123","Type":"ContainerDied","Data":"5c72ed9cf71aebe58f4ad2a14811eeb1baa796ee2626c4c3536ba7472bd660b2"}
Sep 17 18:17:21.461723 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:21.461616    2585 scope.go:117] "RemoveContainer" containerID="2a6d5f217a021871684cec76e0e91159d3f419fdff654d564bf7f8018846c14f"
Sep 17 18:17:21.470405 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:21.470389    2585 scope.go:117] "RemoveContainer" containerID="be4c6953b6138da084688cd3f4073593e87692aecf2f06a2e0f505245433110b"
Sep 17 18:17:21.477313 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:21.477298    2585 scope.go:117] "RemoveContainer" containerID="6ce4b2be269210560dd891b51b31a33cf0042244b2255773e6a5a3f640b8029b"
Sep 17 18:17:21.484552 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:21.484510    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx"]
Sep 17 18:17:21.484602 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:21.484575    2585 scope.go:117] "RemoveContainer" containerID="2a6d5f217a021871684cec76e0e91159d3f419fdff654d564bf7f8018846c14f"
Sep 17 18:17:21.484863 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:17:21.484846    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a6d5f217a021871684cec76e0e91159d3f419fdff654d564bf7f8018846c14f\": container with ID starting with 2a6d5f217a021871684cec76e0e91159d3f419fdff654d564bf7f8018846c14f not found: ID does not exist" containerID="2a6d5f217a021871684cec76e0e91159d3f419fdff654d564bf7f8018846c14f"
Sep 17 18:17:21.484905 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:21.484871    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a6d5f217a021871684cec76e0e91159d3f419fdff654d564bf7f8018846c14f"} err="failed to get container status \"2a6d5f217a021871684cec76e0e91159d3f419fdff654d564bf7f8018846c14f\": rpc error: code = NotFound desc = could not find container \"2a6d5f217a021871684cec76e0e91159d3f419fdff654d564bf7f8018846c14f\": container with ID starting with 2a6d5f217a021871684cec76e0e91159d3f419fdff654d564bf7f8018846c14f not found: ID does not exist"
Sep 17 18:17:21.484905 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:21.484888    2585 scope.go:117] "RemoveContainer" containerID="be4c6953b6138da084688cd3f4073593e87692aecf2f06a2e0f505245433110b"
Sep 17 18:17:21.485138 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:17:21.485121    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be4c6953b6138da084688cd3f4073593e87692aecf2f06a2e0f505245433110b\": container with ID starting with be4c6953b6138da084688cd3f4073593e87692aecf2f06a2e0f505245433110b not found: ID does not exist" containerID="be4c6953b6138da084688cd3f4073593e87692aecf2f06a2e0f505245433110b"
Sep 17 18:17:21.485174 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:21.485145    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be4c6953b6138da084688cd3f4073593e87692aecf2f06a2e0f505245433110b"} err="failed to get container status \"be4c6953b6138da084688cd3f4073593e87692aecf2f06a2e0f505245433110b\": rpc error: code = NotFound desc = could not find container \"be4c6953b6138da084688cd3f4073593e87692aecf2f06a2e0f505245433110b\": container with ID starting with be4c6953b6138da084688cd3f4073593e87692aecf2f06a2e0f505245433110b not found: ID does not exist"
Sep 17 18:17:21.485174 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:21.485160    2585 scope.go:117] "RemoveContainer" containerID="6ce4b2be269210560dd891b51b31a33cf0042244b2255773e6a5a3f640b8029b"
Sep 17 18:17:21.485393 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:17:21.485377    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ce4b2be269210560dd891b51b31a33cf0042244b2255773e6a5a3f640b8029b\": container with ID starting with 6ce4b2be269210560dd891b51b31a33cf0042244b2255773e6a5a3f640b8029b not found: ID does not exist" containerID="6ce4b2be269210560dd891b51b31a33cf0042244b2255773e6a5a3f640b8029b"
Sep 17 18:17:21.485426 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:21.485399    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ce4b2be269210560dd891b51b31a33cf0042244b2255773e6a5a3f640b8029b"} err="failed to get container status \"6ce4b2be269210560dd891b51b31a33cf0042244b2255773e6a5a3f640b8029b\": rpc error: code = NotFound desc = could not find container \"6ce4b2be269210560dd891b51b31a33cf0042244b2255773e6a5a3f640b8029b\": container with ID starting with 6ce4b2be269210560dd891b51b31a33cf0042244b2255773e6a5a3f640b8029b not found: ID does not exist"
Sep 17 18:17:21.489156 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:21.489137    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-cp9dx"]
Sep 17 18:17:22.103994 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:17:22.103962    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cafe1fd8-26c9-4d21-97a5-0785fa7a4123" path="/var/lib/kubelet/pods/cafe1fd8-26c9-4d21-97a5-0785fa7a4123/volumes"
Sep 17 18:18:07.325404 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:07.325370    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:10.889321 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:10.889289    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:12.429806 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:12.429777    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:12.460794 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:12.460764    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-mlbjv"]
Sep 17 18:18:12.461028 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:12.461017    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f08897f-73ad-448f-8db7-4fdd6d94a417" containerName="kube-rbac-proxy"
Sep 17 18:18:12.461072 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:12.461029    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f08897f-73ad-448f-8db7-4fdd6d94a417" containerName="kube-rbac-proxy"
Sep 17 18:18:12.461072 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:12.461041    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f08897f-73ad-448f-8db7-4fdd6d94a417" containerName="kserve-container"
Sep 17 18:18:12.461072 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:12.461046    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f08897f-73ad-448f-8db7-4fdd6d94a417" containerName="kserve-container"
Sep 17 18:18:12.461072 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:12.461053    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f08897f-73ad-448f-8db7-4fdd6d94a417" containerName="storage-initializer"
Sep 17 18:18:12.461072 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:12.461059    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f08897f-73ad-448f-8db7-4fdd6d94a417" containerName="storage-initializer"
Sep 17 18:18:12.461072 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:12.461066    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cafe1fd8-26c9-4d21-97a5-0785fa7a4123" containerName="kube-rbac-proxy"
Sep 17 18:18:12.461072 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:12.461071    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="cafe1fd8-26c9-4d21-97a5-0785fa7a4123" containerName="kube-rbac-proxy"
Sep 17 18:18:12.461285 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:12.461079    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cafe1fd8-26c9-4d21-97a5-0785fa7a4123" containerName="kserve-container"
Sep 17 18:18:12.461285 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:12.461085    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="cafe1fd8-26c9-4d21-97a5-0785fa7a4123" containerName="kserve-container"
Sep 17 18:18:12.461285 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:12.461096    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cafe1fd8-26c9-4d21-97a5-0785fa7a4123" containerName="storage-initializer"
Sep 17 18:18:12.461285 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:12.461101    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="cafe1fd8-26c9-4d21-97a5-0785fa7a4123" containerName="storage-initializer"
Sep 17 18:18:12.461285 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:12.461141    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f08897f-73ad-448f-8db7-4fdd6d94a417" containerName="kube-rbac-proxy"
Sep 17 18:18:12.461285 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:12.461148    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="cafe1fd8-26c9-4d21-97a5-0785fa7a4123" containerName="kserve-container"
Sep 17 18:18:12.461285 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:12.461153    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f08897f-73ad-448f-8db7-4fdd6d94a417" containerName="kserve-container"
Sep 17 18:18:12.461285 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:12.461160    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="cafe1fd8-26c9-4d21-97a5-0785fa7a4123" containerName="kube-rbac-proxy"
Sep 17 18:18:12.464071 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:12.464052    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-mlbjv"
Sep 17 18:18:12.466718 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:12.466694    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-predictor-serving-cert\""
Sep 17 18:18:12.466844 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:12.466776    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:18:12.466910 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:12.466892    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:18:12.466959 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:12.466915    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lps5w\""
Sep 17 18:18:12.467034 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:12.467012    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:18:12.467581 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:12.467565    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:18:12.467667 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:12.467602    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
Sep 17 18:18:12.473408 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:12.473390    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-mlbjv"]
Sep 17 18:18:12.546237 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:12.546205    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7b3d958-fc41-40a2-bdcc-ad67808e2e59-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-mlbjv\" (UID: \"f7b3d958-fc41-40a2-bdcc-ad67808e2e59\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-mlbjv"
Sep 17 18:18:12.546410 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:12.546259    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgjm2\" (UniqueName: \"kubernetes.io/projected/f7b3d958-fc41-40a2-bdcc-ad67808e2e59-kube-api-access-pgjm2\") pod \"isvc-xgboost-v2-predictor-fcc668bc-mlbjv\" (UID: \"f7b3d958-fc41-40a2-bdcc-ad67808e2e59\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-mlbjv"
Sep 17 18:18:12.546410 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:12.546282    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7b3d958-fc41-40a2-bdcc-ad67808e2e59-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-mlbjv\" (UID: \"f7b3d958-fc41-40a2-bdcc-ad67808e2e59\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-mlbjv"
Sep 17 18:18:12.546410 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:12.546323    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7b3d958-fc41-40a2-bdcc-ad67808e2e59-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-mlbjv\" (UID: \"f7b3d958-fc41-40a2-bdcc-ad67808e2e59\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-mlbjv"
Sep 17 18:18:12.546410 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:12.546397    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7b3d958-fc41-40a2-bdcc-ad67808e2e59-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-mlbjv\" (UID: \"f7b3d958-fc41-40a2-bdcc-ad67808e2e59\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-mlbjv"
Sep 17 18:18:12.647409 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:12.647375    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7b3d958-fc41-40a2-bdcc-ad67808e2e59-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-mlbjv\" (UID: \"f7b3d958-fc41-40a2-bdcc-ad67808e2e59\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-mlbjv"
Sep 17 18:18:12.647609 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:12.647413    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7b3d958-fc41-40a2-bdcc-ad67808e2e59-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-mlbjv\" (UID: \"f7b3d958-fc41-40a2-bdcc-ad67808e2e59\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-mlbjv"
Sep 17 18:18:12.647609 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:12.647457    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7b3d958-fc41-40a2-bdcc-ad67808e2e59-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-mlbjv\" (UID: \"f7b3d958-fc41-40a2-bdcc-ad67808e2e59\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-mlbjv"
Sep 17 18:18:12.647609 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:12.647510    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7b3d958-fc41-40a2-bdcc-ad67808e2e59-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-mlbjv\" (UID: \"f7b3d958-fc41-40a2-bdcc-ad67808e2e59\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-mlbjv"
Sep 17 18:18:12.647609 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:12.647571    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgjm2\" (UniqueName: \"kubernetes.io/projected/f7b3d958-fc41-40a2-bdcc-ad67808e2e59-kube-api-access-pgjm2\") pod \"isvc-xgboost-v2-predictor-fcc668bc-mlbjv\" (UID: \"f7b3d958-fc41-40a2-bdcc-ad67808e2e59\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-mlbjv"
Sep 17 18:18:12.647609 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:18:12.647603    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-predictor-serving-cert: secret "isvc-xgboost-v2-predictor-serving-cert" not found
Sep 17 18:18:12.647859 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:18:12.647733    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7b3d958-fc41-40a2-bdcc-ad67808e2e59-proxy-tls podName:f7b3d958-fc41-40a2-bdcc-ad67808e2e59 nodeName:}" failed. No retries permitted until 2026-09-17 18:18:13.147709616 +0000 UTC m=+2629.561500046 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f7b3d958-fc41-40a2-bdcc-ad67808e2e59-proxy-tls") pod "isvc-xgboost-v2-predictor-fcc668bc-mlbjv" (UID: "f7b3d958-fc41-40a2-bdcc-ad67808e2e59") : secret "isvc-xgboost-v2-predictor-serving-cert" not found
Sep 17 18:18:12.647921 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:12.647900    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7b3d958-fc41-40a2-bdcc-ad67808e2e59-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-mlbjv\" (UID: \"f7b3d958-fc41-40a2-bdcc-ad67808e2e59\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-mlbjv"
Sep 17 18:18:12.648154 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:12.648139    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7b3d958-fc41-40a2-bdcc-ad67808e2e59-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-mlbjv\" (UID: \"f7b3d958-fc41-40a2-bdcc-ad67808e2e59\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-mlbjv"
Sep 17 18:18:12.648191 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:12.648140    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7b3d958-fc41-40a2-bdcc-ad67808e2e59-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-mlbjv\" (UID: \"f7b3d958-fc41-40a2-bdcc-ad67808e2e59\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-mlbjv"
Sep 17 18:18:12.656935 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:12.656915    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgjm2\" (UniqueName: \"kubernetes.io/projected/f7b3d958-fc41-40a2-bdcc-ad67808e2e59-kube-api-access-pgjm2\") pod \"isvc-xgboost-v2-predictor-fcc668bc-mlbjv\" (UID: \"f7b3d958-fc41-40a2-bdcc-ad67808e2e59\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-mlbjv"
Sep 17 18:18:12.981298 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:12.981256    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:13.151417 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:13.151384    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7b3d958-fc41-40a2-bdcc-ad67808e2e59-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-mlbjv\" (UID: \"f7b3d958-fc41-40a2-bdcc-ad67808e2e59\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-mlbjv"
Sep 17 18:18:13.153778 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:13.153750    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7b3d958-fc41-40a2-bdcc-ad67808e2e59-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-mlbjv\" (UID: \"f7b3d958-fc41-40a2-bdcc-ad67808e2e59\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-mlbjv"
Sep 17 18:18:13.375225 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:13.375187    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-mlbjv"
Sep 17 18:18:13.647639 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:13.647617    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-mlbjv"]
Sep 17 18:18:13.650347 ip-10-0-133-139 kubenswrapper[2585]: W0917 18:18:13.650321    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7b3d958_fc41_40a2_bdcc_ad67808e2e59.slice/crio-7d8bddaf6559c4d59686c8a17c001693e6ff468530c8dc38c0584d64db774da1 WatchSource:0}: Error finding container 7d8bddaf6559c4d59686c8a17c001693e6ff468530c8dc38c0584d64db774da1: Status 404 returned error can't find the container with id 7d8bddaf6559c4d59686c8a17c001693e6ff468530c8dc38c0584d64db774da1
Sep 17 18:18:14.643346 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:14.643301    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-mlbjv" event={"ID":"f7b3d958-fc41-40a2-bdcc-ad67808e2e59","Type":"ContainerStarted","Data":"659512309ea85f54b7d718b4167bd81f5eaeadf7c6458abe4fdb95109d99e513"}
Sep 17 18:18:14.643537 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:14.643394    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-mlbjv" event={"ID":"f7b3d958-fc41-40a2-bdcc-ad67808e2e59","Type":"ContainerStarted","Data":"7d8bddaf6559c4d59686c8a17c001693e6ff468530c8dc38c0584d64db774da1"}
Sep 17 18:18:16.024620 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:16.024587    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:16.055479 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:16.055449    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ptdxs"]
Sep 17 18:18:16.059004 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:16.058985    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ptdxs"
Sep 17 18:18:16.061466 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:16.061449    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
Sep 17 18:18:16.061741 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:16.061727    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 17 18:18:16.067745 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:16.067722    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ptdxs"]
Sep 17 18:18:16.174892 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:16.174859    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdkdp\" (UniqueName: \"kubernetes.io/projected/d6ebb086-489d-4c26-963e-cc6688274e7e-kube-api-access-cdkdp\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ptdxs\" (UID: \"d6ebb086-489d-4c26-963e-cc6688274e7e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ptdxs"
Sep 17 18:18:16.175073 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:16.174902    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6ebb086-489d-4c26-963e-cc6688274e7e-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ptdxs\" (UID: \"d6ebb086-489d-4c26-963e-cc6688274e7e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ptdxs"
Sep 17 18:18:16.175073 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:16.174925    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6ebb086-489d-4c26-963e-cc6688274e7e-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ptdxs\" (UID: \"d6ebb086-489d-4c26-963e-cc6688274e7e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ptdxs"
Sep 17 18:18:16.175073 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:16.175000    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6ebb086-489d-4c26-963e-cc6688274e7e-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ptdxs\" (UID: \"d6ebb086-489d-4c26-963e-cc6688274e7e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ptdxs"
Sep 17 18:18:16.175073 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:16.175020    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6ebb086-489d-4c26-963e-cc6688274e7e-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ptdxs\" (UID: \"d6ebb086-489d-4c26-963e-cc6688274e7e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ptdxs"
Sep 17 18:18:16.276318 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:16.276242    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6ebb086-489d-4c26-963e-cc6688274e7e-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ptdxs\" (UID: \"d6ebb086-489d-4c26-963e-cc6688274e7e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ptdxs"
Sep 17 18:18:16.276318 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:16.276276    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6ebb086-489d-4c26-963e-cc6688274e7e-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ptdxs\" (UID: \"d6ebb086-489d-4c26-963e-cc6688274e7e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ptdxs"
Sep 17 18:18:16.276318 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:16.276295    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6ebb086-489d-4c26-963e-cc6688274e7e-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ptdxs\" (UID: \"d6ebb086-489d-4c26-963e-cc6688274e7e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ptdxs"
Sep 17 18:18:16.276680 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:16.276326    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdkdp\" (UniqueName: \"kubernetes.io/projected/d6ebb086-489d-4c26-963e-cc6688274e7e-kube-api-access-cdkdp\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ptdxs\" (UID: \"d6ebb086-489d-4c26-963e-cc6688274e7e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ptdxs"
Sep 17 18:18:16.276680 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:16.276349    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6ebb086-489d-4c26-963e-cc6688274e7e-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ptdxs\" (UID: \"d6ebb086-489d-4c26-963e-cc6688274e7e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ptdxs"
Sep 17 18:18:16.276680 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:18:16.276410    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-serving-cert: secret "isvc-lightgbm-v2-kserve-predictor-serving-cert" not found
Sep 17 18:18:16.276680 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:18:16.276478    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6ebb086-489d-4c26-963e-cc6688274e7e-proxy-tls podName:d6ebb086-489d-4c26-963e-cc6688274e7e nodeName:}" failed. No retries permitted until 2026-09-17 18:18:16.776456237 +0000 UTC m=+2633.190246673 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d6ebb086-489d-4c26-963e-cc6688274e7e-proxy-tls") pod "isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ptdxs" (UID: "d6ebb086-489d-4c26-963e-cc6688274e7e") : secret "isvc-lightgbm-v2-kserve-predictor-serving-cert" not found
Sep 17 18:18:16.276908 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:16.276720    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6ebb086-489d-4c26-963e-cc6688274e7e-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ptdxs\" (UID: \"d6ebb086-489d-4c26-963e-cc6688274e7e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ptdxs"
Sep 17 18:18:16.277080 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:16.277061    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6ebb086-489d-4c26-963e-cc6688274e7e-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ptdxs\" (UID: \"d6ebb086-489d-4c26-963e-cc6688274e7e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ptdxs"
Sep 17 18:18:16.277154 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:16.277129    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6ebb086-489d-4c26-963e-cc6688274e7e-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ptdxs\" (UID: \"d6ebb086-489d-4c26-963e-cc6688274e7e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ptdxs"
Sep 17 18:18:16.285053 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:16.285030    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdkdp\" (UniqueName: \"kubernetes.io/projected/d6ebb086-489d-4c26-963e-cc6688274e7e-kube-api-access-cdkdp\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ptdxs\" (UID: \"d6ebb086-489d-4c26-963e-cc6688274e7e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ptdxs"
Sep 17 18:18:16.342722 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:16.342688    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:16.780808 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:16.780761    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6ebb086-489d-4c26-963e-cc6688274e7e-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ptdxs\" (UID: \"d6ebb086-489d-4c26-963e-cc6688274e7e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ptdxs"
Sep 17 18:18:16.783167 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:16.783149    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6ebb086-489d-4c26-963e-cc6688274e7e-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ptdxs\" (UID: \"d6ebb086-489d-4c26-963e-cc6688274e7e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ptdxs"
Sep 17 18:18:16.969387 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:16.969341    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ptdxs"
Sep 17 18:18:17.238473 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:17.238445    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ptdxs"]
Sep 17 18:18:17.240672 ip-10-0-133-139 kubenswrapper[2585]: W0917 18:18:17.240647    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6ebb086_489d_4c26_963e_cc6688274e7e.slice/crio-79a698291a3e608888ee978555f7daa931b663466b8077ca0170d80ea0020525 WatchSource:0}: Error finding container 79a698291a3e608888ee978555f7daa931b663466b8077ca0170d80ea0020525: Status 404 returned error can't find the container with id 79a698291a3e608888ee978555f7daa931b663466b8077ca0170d80ea0020525
Sep 17 18:18:17.659387 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:17.659350    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ptdxs" event={"ID":"d6ebb086-489d-4c26-963e-cc6688274e7e","Type":"ContainerStarted","Data":"e146d36fee9ba7b718cb99e6857827c46ff0c6ea39cd8cdd0d1e0b551486c760"}
Sep 17 18:18:17.659563 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:17.659434    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ptdxs" event={"ID":"d6ebb086-489d-4c26-963e-cc6688274e7e","Type":"ContainerStarted","Data":"79a698291a3e608888ee978555f7daa931b663466b8077ca0170d80ea0020525"}
Sep 17 18:18:17.660988 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:17.660964    2585 generic.go:356] "Generic (PLEG): container finished" podID="f7b3d958-fc41-40a2-bdcc-ad67808e2e59" containerID="659512309ea85f54b7d718b4167bd81f5eaeadf7c6458abe4fdb95109d99e513" exitCode=0
Sep 17 18:18:17.661076 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:17.661018    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-mlbjv" event={"ID":"f7b3d958-fc41-40a2-bdcc-ad67808e2e59","Type":"ContainerDied","Data":"659512309ea85f54b7d718b4167bd81f5eaeadf7c6458abe4fdb95109d99e513"}
Sep 17 18:18:18.666697 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:18.666661    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-mlbjv" event={"ID":"f7b3d958-fc41-40a2-bdcc-ad67808e2e59","Type":"ContainerStarted","Data":"a88b2114dc148eaefab4cec6df6a20b665c7cbc8be435a0a3b761e48a765dc70"}
Sep 17 18:18:18.667093 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:18.666724    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-mlbjv" event={"ID":"f7b3d958-fc41-40a2-bdcc-ad67808e2e59","Type":"ContainerStarted","Data":"32670166ee7764232b551e378dafa3f5be677bbca05a366ac97886395225a6c6"}
Sep 17 18:18:18.667093 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:18.666990    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-mlbjv"
Sep 17 18:18:18.686943 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:18.686899    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-mlbjv" podStartSLOduration=2.78122235 podStartE2EDuration="6.686885625s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.905663275s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:18:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:18:18.685646893 +0000 UTC m=+2635.099437334" watchObservedRunningTime="2026-09-17 18:18:18.686885625 +0000 UTC m=+2635.100676055"
Sep 17 18:18:19.670428 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:19.670396    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-mlbjv"
Sep 17 18:18:19.671864 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:19.671835    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-mlbjv" podUID="f7b3d958-fc41-40a2-bdcc-ad67808e2e59" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 18:18:20.674199 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:20.674160    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-mlbjv" podUID="f7b3d958-fc41-40a2-bdcc-ad67808e2e59" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 18:18:21.678878 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:21.678844    2585 generic.go:356] "Generic (PLEG): container finished" podID="d6ebb086-489d-4c26-963e-cc6688274e7e" containerID="e146d36fee9ba7b718cb99e6857827c46ff0c6ea39cd8cdd0d1e0b551486c760" exitCode=0
Sep 17 18:18:21.679223 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:21.678903    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ptdxs" event={"ID":"d6ebb086-489d-4c26-963e-cc6688274e7e","Type":"ContainerDied","Data":"e146d36fee9ba7b718cb99e6857827c46ff0c6ea39cd8cdd0d1e0b551486c760"}
Sep 17 18:18:22.685470 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:22.685431    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ptdxs" event={"ID":"d6ebb086-489d-4c26-963e-cc6688274e7e","Type":"ContainerStarted","Data":"f7b5b4bdfc3f97dd00ef04aa2216e729ad5ea0aba56f563ba7571f30978ea6fb"}
Sep 17 18:18:22.685852 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:22.685528    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ptdxs" event={"ID":"d6ebb086-489d-4c26-963e-cc6688274e7e","Type":"ContainerStarted","Data":"f1a961fd18b45322a98b0068c19609d56ac4d00be236ad5e7b34bdf72f5deefe"}
Sep 17 18:18:22.685852 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:22.685741    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ptdxs"
Sep 17 18:18:22.706449 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:22.706390    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ptdxs" podStartSLOduration=2.992832095 podStartE2EDuration="6.706372303s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.713540208s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:18:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:18:22.705580548 +0000 UTC m=+2639.119370986" watchObservedRunningTime="2026-09-17 18:18:22.706372303 +0000 UTC m=+2639.120162743"
Sep 17 18:18:23.690633 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:23.690603    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ptdxs"
Sep 17 18:18:23.691703 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:23.691677    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ptdxs" podUID="d6ebb086-489d-4c26-963e-cc6688274e7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 18:18:24.693687 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:24.693648    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ptdxs" podUID="d6ebb086-489d-4c26-963e-cc6688274e7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 18:18:25.678304 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:25.678274    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-mlbjv"
Sep 17 18:18:25.678865 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:25.678834    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-mlbjv" podUID="f7b3d958-fc41-40a2-bdcc-ad67808e2e59" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 18:18:29.698288 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:29.698257    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ptdxs"
Sep 17 18:18:29.698863 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:29.698840    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ptdxs" podUID="d6ebb086-489d-4c26-963e-cc6688274e7e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 18:18:35.679629 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:35.679588    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-mlbjv" podUID="f7b3d958-fc41-40a2-bdcc-ad67808e2e59" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 18:18:39.700171 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:39.700141    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ptdxs"
Sep 17 18:18:45.678857 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:45.678817    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-mlbjv" podUID="f7b3d958-fc41-40a2-bdcc-ad67808e2e59" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 18:18:46.144381 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:46.144345    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ptdxs"]
Sep 17 18:18:46.144665 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:46.144631    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ptdxs" podUID="d6ebb086-489d-4c26-963e-cc6688274e7e" containerName="kserve-container" containerID="cri-o://f1a961fd18b45322a98b0068c19609d56ac4d00be236ad5e7b34bdf72f5deefe" gracePeriod=30
Sep 17 18:18:46.144823 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:46.144648    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ptdxs" podUID="d6ebb086-489d-4c26-963e-cc6688274e7e" containerName="kube-rbac-proxy" containerID="cri-o://f7b5b4bdfc3f97dd00ef04aa2216e729ad5ea0aba56f563ba7571f30978ea6fb" gracePeriod=30
Sep 17 18:18:46.170975 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:46.170924    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:46.774869 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:46.774837    2585 generic.go:356] "Generic (PLEG): container finished" podID="d6ebb086-489d-4c26-963e-cc6688274e7e" containerID="f7b5b4bdfc3f97dd00ef04aa2216e729ad5ea0aba56f563ba7571f30978ea6fb" exitCode=2
Sep 17 18:18:46.774869 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:46.774866    2585 generic.go:356] "Generic (PLEG): container finished" podID="d6ebb086-489d-4c26-963e-cc6688274e7e" containerID="f1a961fd18b45322a98b0068c19609d56ac4d00be236ad5e7b34bdf72f5deefe" exitCode=0
Sep 17 18:18:46.775248 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:46.774906    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ptdxs" event={"ID":"d6ebb086-489d-4c26-963e-cc6688274e7e","Type":"ContainerDied","Data":"f7b5b4bdfc3f97dd00ef04aa2216e729ad5ea0aba56f563ba7571f30978ea6fb"}
Sep 17 18:18:46.775248 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:46.774980    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ptdxs" event={"ID":"d6ebb086-489d-4c26-963e-cc6688274e7e","Type":"ContainerDied","Data":"f1a961fd18b45322a98b0068c19609d56ac4d00be236ad5e7b34bdf72f5deefe"}
Sep 17 18:18:46.775248 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:46.775012    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ptdxs" event={"ID":"d6ebb086-489d-4c26-963e-cc6688274e7e","Type":"ContainerDied","Data":"79a698291a3e608888ee978555f7daa931b663466b8077ca0170d80ea0020525"}
Sep 17 18:18:46.775248 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:46.775043    2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79a698291a3e608888ee978555f7daa931b663466b8077ca0170d80ea0020525"
Sep 17 18:18:46.783287 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:46.783270    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ptdxs"
Sep 17 18:18:46.790250 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:46.790234    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cdkdp\" (UniqueName: \"kubernetes.io/projected/d6ebb086-489d-4c26-963e-cc6688274e7e-kube-api-access-cdkdp\") pod \"d6ebb086-489d-4c26-963e-cc6688274e7e\" (UID: \"d6ebb086-489d-4c26-963e-cc6688274e7e\") "
Sep 17 18:18:46.790351 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:46.790291    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6ebb086-489d-4c26-963e-cc6688274e7e-kube-rbac-proxy-sar-config\") pod \"d6ebb086-489d-4c26-963e-cc6688274e7e\" (UID: \"d6ebb086-489d-4c26-963e-cc6688274e7e\") "
Sep 17 18:18:46.790351 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:46.790312    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6ebb086-489d-4c26-963e-cc6688274e7e-cabundle-cert\") pod \"d6ebb086-489d-4c26-963e-cc6688274e7e\" (UID: \"d6ebb086-489d-4c26-963e-cc6688274e7e\") "
Sep 17 18:18:46.790351 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:46.790329    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6ebb086-489d-4c26-963e-cc6688274e7e-kserve-provision-location\") pod \"d6ebb086-489d-4c26-963e-cc6688274e7e\" (UID: \"d6ebb086-489d-4c26-963e-cc6688274e7e\") "
Sep 17 18:18:46.790351 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:46.790344    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6ebb086-489d-4c26-963e-cc6688274e7e-proxy-tls\") pod \"d6ebb086-489d-4c26-963e-cc6688274e7e\" (UID: \"d6ebb086-489d-4c26-963e-cc6688274e7e\") "
Sep 17 18:18:46.790700 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:46.790679    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6ebb086-489d-4c26-963e-cc6688274e7e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d6ebb086-489d-4c26-963e-cc6688274e7e" (UID: "d6ebb086-489d-4c26-963e-cc6688274e7e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:18:46.790757 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:46.790694    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6ebb086-489d-4c26-963e-cc6688274e7e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d6ebb086-489d-4c26-963e-cc6688274e7e" (UID: "d6ebb086-489d-4c26-963e-cc6688274e7e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:18:46.790757 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:46.790692    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6ebb086-489d-4c26-963e-cc6688274e7e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d6ebb086-489d-4c26-963e-cc6688274e7e" (UID: "d6ebb086-489d-4c26-963e-cc6688274e7e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:18:46.792392 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:46.792368    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6ebb086-489d-4c26-963e-cc6688274e7e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d6ebb086-489d-4c26-963e-cc6688274e7e" (UID: "d6ebb086-489d-4c26-963e-cc6688274e7e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:18:46.792392 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:46.792386    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6ebb086-489d-4c26-963e-cc6688274e7e-kube-api-access-cdkdp" (OuterVolumeSpecName: "kube-api-access-cdkdp") pod "d6ebb086-489d-4c26-963e-cc6688274e7e" (UID: "d6ebb086-489d-4c26-963e-cc6688274e7e"). InnerVolumeSpecName "kube-api-access-cdkdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:18:46.891220 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:46.891145    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdkdp\" (UniqueName: \"kubernetes.io/projected/d6ebb086-489d-4c26-963e-cc6688274e7e-kube-api-access-cdkdp\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:18:46.891220 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:46.891170    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6ebb086-489d-4c26-963e-cc6688274e7e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:18:46.891220 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:46.891180    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6ebb086-489d-4c26-963e-cc6688274e7e-cabundle-cert\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:18:46.891220 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:46.891190    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6ebb086-489d-4c26-963e-cc6688274e7e-kserve-provision-location\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:18:46.891220 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:46.891204    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6ebb086-489d-4c26-963e-cc6688274e7e-proxy-tls\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:18:47.780022 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:47.779995    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ptdxs"
Sep 17 18:18:47.802526 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:47.802482    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ptdxs"]
Sep 17 18:18:47.806025 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:47.806004    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-ptdxs"]
Sep 17 18:18:48.103913 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:48.103824    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6ebb086-489d-4c26-963e-cc6688274e7e" path="/var/lib/kubelet/pods/d6ebb086-489d-4c26-963e-cc6688274e7e/volumes"
Sep 17 18:18:51.315422 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:51.315377    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:51.682553 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:51.682439    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:55.679702 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:18:55.679663    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-mlbjv" podUID="f7b3d958-fc41-40a2-bdcc-ad67808e2e59" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 18:19:05.679189 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:05.679146    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-mlbjv" podUID="f7b3d958-fc41-40a2-bdcc-ad67808e2e59" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 18:19:15.678781 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:15.678742    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-mlbjv" podUID="f7b3d958-fc41-40a2-bdcc-ad67808e2e59" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 18:19:25.680223 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:25.680195    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-mlbjv"
Sep 17 18:19:28.494472 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:28.494445    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rrf79_beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5/ovn-acl-logging/0.log"
Sep 17 18:19:28.497771 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:28.497749    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rrf79_beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5/ovn-acl-logging/0.log"
Sep 17 18:19:33.123285 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:33.123241    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-mlbjv"]
Sep 17 18:19:33.123760 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:33.123554    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-mlbjv" podUID="f7b3d958-fc41-40a2-bdcc-ad67808e2e59" containerName="kserve-container" containerID="cri-o://32670166ee7764232b551e378dafa3f5be677bbca05a366ac97886395225a6c6" gracePeriod=30
Sep 17 18:19:33.123760 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:33.123574    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-mlbjv" podUID="f7b3d958-fc41-40a2-bdcc-ad67808e2e59" containerName="kube-rbac-proxy" containerID="cri-o://a88b2114dc148eaefab4cec6df6a20b665c7cbc8be435a0a3b761e48a765dc70" gracePeriod=30
Sep 17 18:19:33.143674 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:33.143633    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:33.946632 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:33.946601    2585 generic.go:356] "Generic (PLEG): container finished" podID="f7b3d958-fc41-40a2-bdcc-ad67808e2e59" containerID="a88b2114dc148eaefab4cec6df6a20b665c7cbc8be435a0a3b761e48a765dc70" exitCode=2
Sep 17 18:19:33.946800 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:33.946672    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-mlbjv" event={"ID":"f7b3d958-fc41-40a2-bdcc-ad67808e2e59","Type":"ContainerDied","Data":"a88b2114dc148eaefab4cec6df6a20b665c7cbc8be435a0a3b761e48a765dc70"}
Sep 17 18:19:35.675386 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:35.675338    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-mlbjv" podUID="f7b3d958-fc41-40a2-bdcc-ad67808e2e59" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused"
Sep 17 18:19:35.679717 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:35.679685    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-mlbjv" podUID="f7b3d958-fc41-40a2-bdcc-ad67808e2e59" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 18:19:37.163276 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:37.163253    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-mlbjv"
Sep 17 18:19:37.232358 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:37.232328    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7b3d958-fc41-40a2-bdcc-ad67808e2e59-cabundle-cert\") pod \"f7b3d958-fc41-40a2-bdcc-ad67808e2e59\" (UID: \"f7b3d958-fc41-40a2-bdcc-ad67808e2e59\") "
Sep 17 18:19:37.232358 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:37.232362    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7b3d958-fc41-40a2-bdcc-ad67808e2e59-proxy-tls\") pod \"f7b3d958-fc41-40a2-bdcc-ad67808e2e59\" (UID: \"f7b3d958-fc41-40a2-bdcc-ad67808e2e59\") "
Sep 17 18:19:37.232579 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:37.232402    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7b3d958-fc41-40a2-bdcc-ad67808e2e59-kserve-provision-location\") pod \"f7b3d958-fc41-40a2-bdcc-ad67808e2e59\" (UID: \"f7b3d958-fc41-40a2-bdcc-ad67808e2e59\") "
Sep 17 18:19:37.232579 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:37.232469    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7b3d958-fc41-40a2-bdcc-ad67808e2e59-kube-rbac-proxy-sar-config\") pod \"f7b3d958-fc41-40a2-bdcc-ad67808e2e59\" (UID: \"f7b3d958-fc41-40a2-bdcc-ad67808e2e59\") "
Sep 17 18:19:37.232579 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:37.232514    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgjm2\" (UniqueName: \"kubernetes.io/projected/f7b3d958-fc41-40a2-bdcc-ad67808e2e59-kube-api-access-pgjm2\") pod \"f7b3d958-fc41-40a2-bdcc-ad67808e2e59\" (UID: \"f7b3d958-fc41-40a2-bdcc-ad67808e2e59\") "
Sep 17 18:19:37.232746 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:37.232710    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7b3d958-fc41-40a2-bdcc-ad67808e2e59-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f7b3d958-fc41-40a2-bdcc-ad67808e2e59" (UID: "f7b3d958-fc41-40a2-bdcc-ad67808e2e59"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:19:37.232786 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:37.232749    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7b3d958-fc41-40a2-bdcc-ad67808e2e59-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f7b3d958-fc41-40a2-bdcc-ad67808e2e59" (UID: "f7b3d958-fc41-40a2-bdcc-ad67808e2e59"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:19:37.232829 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:37.232800    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7b3d958-fc41-40a2-bdcc-ad67808e2e59-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f7b3d958-fc41-40a2-bdcc-ad67808e2e59" (UID: "f7b3d958-fc41-40a2-bdcc-ad67808e2e59"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:19:37.234484 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:37.234463    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7b3d958-fc41-40a2-bdcc-ad67808e2e59-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f7b3d958-fc41-40a2-bdcc-ad67808e2e59" (UID: "f7b3d958-fc41-40a2-bdcc-ad67808e2e59"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:19:37.234592 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:37.234565    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7b3d958-fc41-40a2-bdcc-ad67808e2e59-kube-api-access-pgjm2" (OuterVolumeSpecName: "kube-api-access-pgjm2") pod "f7b3d958-fc41-40a2-bdcc-ad67808e2e59" (UID: "f7b3d958-fc41-40a2-bdcc-ad67808e2e59"). InnerVolumeSpecName "kube-api-access-pgjm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:19:37.333546 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:37.333464    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7b3d958-fc41-40a2-bdcc-ad67808e2e59-proxy-tls\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:19:37.333546 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:37.333510    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7b3d958-fc41-40a2-bdcc-ad67808e2e59-kserve-provision-location\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:19:37.333546 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:37.333521    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7b3d958-fc41-40a2-bdcc-ad67808e2e59-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:19:37.333546 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:37.333530    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgjm2\" (UniqueName: \"kubernetes.io/projected/f7b3d958-fc41-40a2-bdcc-ad67808e2e59-kube-api-access-pgjm2\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:19:37.333546 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:37.333538    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7b3d958-fc41-40a2-bdcc-ad67808e2e59-cabundle-cert\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:19:37.965122 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:37.965087    2585 generic.go:356] "Generic (PLEG): container finished" podID="f7b3d958-fc41-40a2-bdcc-ad67808e2e59" containerID="32670166ee7764232b551e378dafa3f5be677bbca05a366ac97886395225a6c6" exitCode=0
Sep 17 18:19:37.965296 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:37.965142    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-mlbjv"
Sep 17 18:19:37.965296 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:37.965175    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-mlbjv" event={"ID":"f7b3d958-fc41-40a2-bdcc-ad67808e2e59","Type":"ContainerDied","Data":"32670166ee7764232b551e378dafa3f5be677bbca05a366ac97886395225a6c6"}
Sep 17 18:19:37.965296 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:37.965234    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-mlbjv" event={"ID":"f7b3d958-fc41-40a2-bdcc-ad67808e2e59","Type":"ContainerDied","Data":"7d8bddaf6559c4d59686c8a17c001693e6ff468530c8dc38c0584d64db774da1"}
Sep 17 18:19:37.965296 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:37.965264    2585 scope.go:117] "RemoveContainer" containerID="a88b2114dc148eaefab4cec6df6a20b665c7cbc8be435a0a3b761e48a765dc70"
Sep 17 18:19:37.973428 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:37.973410    2585 scope.go:117] "RemoveContainer" containerID="32670166ee7764232b551e378dafa3f5be677bbca05a366ac97886395225a6c6"
Sep 17 18:19:37.980403 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:37.980386    2585 scope.go:117] "RemoveContainer" containerID="659512309ea85f54b7d718b4167bd81f5eaeadf7c6458abe4fdb95109d99e513"
Sep 17 18:19:37.987414 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:37.987370    2585 scope.go:117] "RemoveContainer" containerID="a88b2114dc148eaefab4cec6df6a20b665c7cbc8be435a0a3b761e48a765dc70"
Sep 17 18:19:37.987806 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:19:37.987779    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a88b2114dc148eaefab4cec6df6a20b665c7cbc8be435a0a3b761e48a765dc70\": container with ID starting with a88b2114dc148eaefab4cec6df6a20b665c7cbc8be435a0a3b761e48a765dc70 not found: ID does not exist" containerID="a88b2114dc148eaefab4cec6df6a20b665c7cbc8be435a0a3b761e48a765dc70"
Sep 17 18:19:37.987898 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:37.987815    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a88b2114dc148eaefab4cec6df6a20b665c7cbc8be435a0a3b761e48a765dc70"} err="failed to get container status \"a88b2114dc148eaefab4cec6df6a20b665c7cbc8be435a0a3b761e48a765dc70\": rpc error: code = NotFound desc = could not find container \"a88b2114dc148eaefab4cec6df6a20b665c7cbc8be435a0a3b761e48a765dc70\": container with ID starting with a88b2114dc148eaefab4cec6df6a20b665c7cbc8be435a0a3b761e48a765dc70 not found: ID does not exist"
Sep 17 18:19:37.987898 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:37.987835    2585 scope.go:117] "RemoveContainer" containerID="32670166ee7764232b551e378dafa3f5be677bbca05a366ac97886395225a6c6"
Sep 17 18:19:37.988106 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:19:37.988087    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32670166ee7764232b551e378dafa3f5be677bbca05a366ac97886395225a6c6\": container with ID starting with 32670166ee7764232b551e378dafa3f5be677bbca05a366ac97886395225a6c6 not found: ID does not exist" containerID="32670166ee7764232b551e378dafa3f5be677bbca05a366ac97886395225a6c6"
Sep 17 18:19:37.988149 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:37.988112    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32670166ee7764232b551e378dafa3f5be677bbca05a366ac97886395225a6c6"} err="failed to get container status \"32670166ee7764232b551e378dafa3f5be677bbca05a366ac97886395225a6c6\": rpc error: code = NotFound desc = could not find container \"32670166ee7764232b551e378dafa3f5be677bbca05a366ac97886395225a6c6\": container with ID starting with 32670166ee7764232b551e378dafa3f5be677bbca05a366ac97886395225a6c6 not found: ID does not exist"
Sep 17 18:19:37.988149 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:37.988128    2585 scope.go:117] "RemoveContainer" containerID="659512309ea85f54b7d718b4167bd81f5eaeadf7c6458abe4fdb95109d99e513"
Sep 17 18:19:37.988335 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:37.988317    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-mlbjv"]
Sep 17 18:19:37.988393 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:19:37.988372    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"659512309ea85f54b7d718b4167bd81f5eaeadf7c6458abe4fdb95109d99e513\": container with ID starting with 659512309ea85f54b7d718b4167bd81f5eaeadf7c6458abe4fdb95109d99e513 not found: ID does not exist" containerID="659512309ea85f54b7d718b4167bd81f5eaeadf7c6458abe4fdb95109d99e513"
Sep 17 18:19:37.988432 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:37.988388    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"659512309ea85f54b7d718b4167bd81f5eaeadf7c6458abe4fdb95109d99e513"} err="failed to get container status \"659512309ea85f54b7d718b4167bd81f5eaeadf7c6458abe4fdb95109d99e513\": rpc error: code = NotFound desc = could not find container \"659512309ea85f54b7d718b4167bd81f5eaeadf7c6458abe4fdb95109d99e513\": container with ID starting with 659512309ea85f54b7d718b4167bd81f5eaeadf7c6458abe4fdb95109d99e513 not found: ID does not exist"
Sep 17 18:19:37.991862 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:37.991842    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-mlbjv"]
Sep 17 18:19:38.103628 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:38.103592    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7b3d958-fc41-40a2-bdcc-ad67808e2e59" path="/var/lib/kubelet/pods/f7b3d958-fc41-40a2-bdcc-ad67808e2e59/volumes"
Sep 17 18:19:38.339205 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:38.339173    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:38.387863 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:38.387836    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-tphrr"]
Sep 17 18:19:38.388095 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:38.388083    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7b3d958-fc41-40a2-bdcc-ad67808e2e59" containerName="storage-initializer"
Sep 17 18:19:38.388138 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:38.388096    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7b3d958-fc41-40a2-bdcc-ad67808e2e59" containerName="storage-initializer"
Sep 17 18:19:38.388138 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:38.388112    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7b3d958-fc41-40a2-bdcc-ad67808e2e59" containerName="kube-rbac-proxy"
Sep 17 18:19:38.388138 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:38.388118    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7b3d958-fc41-40a2-bdcc-ad67808e2e59" containerName="kube-rbac-proxy"
Sep 17 18:19:38.388138 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:38.388127    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6ebb086-489d-4c26-963e-cc6688274e7e" containerName="kube-rbac-proxy"
Sep 17 18:19:38.388138 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:38.388133    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6ebb086-489d-4c26-963e-cc6688274e7e" containerName="kube-rbac-proxy"
Sep 17 18:19:38.388293 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:38.388140    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6ebb086-489d-4c26-963e-cc6688274e7e" containerName="storage-initializer"
Sep 17 18:19:38.388293 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:38.388145    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6ebb086-489d-4c26-963e-cc6688274e7e" containerName="storage-initializer"
Sep 17 18:19:38.388293 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:38.388152    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7b3d958-fc41-40a2-bdcc-ad67808e2e59" containerName="kserve-container"
Sep 17 18:19:38.388293 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:38.388157    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7b3d958-fc41-40a2-bdcc-ad67808e2e59" containerName="kserve-container"
Sep 17 18:19:38.388293 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:38.388164    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6ebb086-489d-4c26-963e-cc6688274e7e" containerName="kserve-container"
Sep 17 18:19:38.388293 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:38.388169    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6ebb086-489d-4c26-963e-cc6688274e7e" containerName="kserve-container"
Sep 17 18:19:38.388293 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:38.388205    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6ebb086-489d-4c26-963e-cc6688274e7e" containerName="kube-rbac-proxy"
Sep 17 18:19:38.388293 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:38.388213    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7b3d958-fc41-40a2-bdcc-ad67808e2e59" containerName="kube-rbac-proxy"
Sep 17 18:19:38.388293 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:38.388219    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7b3d958-fc41-40a2-bdcc-ad67808e2e59" containerName="kserve-container"
Sep 17 18:19:38.388293 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:38.388225    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6ebb086-489d-4c26-963e-cc6688274e7e" containerName="kserve-container"
Sep 17 18:19:38.392379 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:38.392363    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-tphrr"
Sep 17 18:19:38.395070 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:38.395051    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:19:38.396117 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:38.396095    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 17 18:19:38.396214 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:38.396171    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-predictor-serving-cert\""
Sep 17 18:19:38.396214 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:38.396178    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:19:38.396425 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:38.396408    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
Sep 17 18:19:38.396521 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:38.396412    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lps5w\""
Sep 17 18:19:38.401005 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:38.400983    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-tphrr"]
Sep 17 18:19:38.441661 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:38.441630    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c023c631-197e-4554-b37d-8e3c35f452d7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-84d46f45-tphrr\" (UID: \"c023c631-197e-4554-b37d-8e3c35f452d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-tphrr"
Sep 17 18:19:38.441782 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:38.441679    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7smc\" (UniqueName: \"kubernetes.io/projected/c023c631-197e-4554-b37d-8e3c35f452d7-kube-api-access-b7smc\") pod \"isvc-sklearn-s3-predictor-84d46f45-tphrr\" (UID: \"c023c631-197e-4554-b37d-8e3c35f452d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-tphrr"
Sep 17 18:19:38.441782 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:38.441696    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c023c631-197e-4554-b37d-8e3c35f452d7-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-84d46f45-tphrr\" (UID: \"c023c631-197e-4554-b37d-8e3c35f452d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-tphrr"
Sep 17 18:19:38.441782 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:38.441768    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c023c631-197e-4554-b37d-8e3c35f452d7-proxy-tls\") pod \"isvc-sklearn-s3-predictor-84d46f45-tphrr\" (UID: \"c023c631-197e-4554-b37d-8e3c35f452d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-tphrr"
Sep 17 18:19:38.443941 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:38.443919    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:38.542751 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:38.542723    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c023c631-197e-4554-b37d-8e3c35f452d7-proxy-tls\") pod \"isvc-sklearn-s3-predictor-84d46f45-tphrr\" (UID: \"c023c631-197e-4554-b37d-8e3c35f452d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-tphrr"
Sep 17 18:19:38.542890 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:38.542766    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c023c631-197e-4554-b37d-8e3c35f452d7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-84d46f45-tphrr\" (UID: \"c023c631-197e-4554-b37d-8e3c35f452d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-tphrr"
Sep 17 18:19:38.542930 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:38.542886    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7smc\" (UniqueName: \"kubernetes.io/projected/c023c631-197e-4554-b37d-8e3c35f452d7-kube-api-access-b7smc\") pod \"isvc-sklearn-s3-predictor-84d46f45-tphrr\" (UID: \"c023c631-197e-4554-b37d-8e3c35f452d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-tphrr"
Sep 17 18:19:38.542973 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:38.542925    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c023c631-197e-4554-b37d-8e3c35f452d7-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-84d46f45-tphrr\" (UID: \"c023c631-197e-4554-b37d-8e3c35f452d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-tphrr"
Sep 17 18:19:38.543299 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:38.543281    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c023c631-197e-4554-b37d-8e3c35f452d7-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-84d46f45-tphrr\" (UID: \"c023c631-197e-4554-b37d-8e3c35f452d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-tphrr"
Sep 17 18:19:38.543369 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:38.543345    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c023c631-197e-4554-b37d-8e3c35f452d7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-84d46f45-tphrr\" (UID: \"c023c631-197e-4554-b37d-8e3c35f452d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-tphrr"
Sep 17 18:19:38.545174 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:38.545157    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c023c631-197e-4554-b37d-8e3c35f452d7-proxy-tls\") pod \"isvc-sklearn-s3-predictor-84d46f45-tphrr\" (UID: \"c023c631-197e-4554-b37d-8e3c35f452d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-tphrr"
Sep 17 18:19:38.551550 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:38.551531    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7smc\" (UniqueName: \"kubernetes.io/projected/c023c631-197e-4554-b37d-8e3c35f452d7-kube-api-access-b7smc\") pod \"isvc-sklearn-s3-predictor-84d46f45-tphrr\" (UID: \"c023c631-197e-4554-b37d-8e3c35f452d7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-tphrr"
Sep 17 18:19:38.703812 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:38.703734    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-tphrr"
Sep 17 18:19:38.973151 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:38.973128    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-tphrr"]
Sep 17 18:19:38.975026 ip-10-0-133-139 kubenswrapper[2585]: W0917 18:19:38.974997    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc023c631_197e_4554_b37d_8e3c35f452d7.slice/crio-bb0a8a9ebe7e7127ff8ed508d316ce836c284cb40163f8ad1ee9c5e2e7b171a0 WatchSource:0}: Error finding container bb0a8a9ebe7e7127ff8ed508d316ce836c284cb40163f8ad1ee9c5e2e7b171a0: Status 404 returned error can't find the container with id bb0a8a9ebe7e7127ff8ed508d316ce836c284cb40163f8ad1ee9c5e2e7b171a0
Sep 17 18:19:39.975334 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:39.975302    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-tphrr" event={"ID":"c023c631-197e-4554-b37d-8e3c35f452d7","Type":"ContainerStarted","Data":"f6e179711580c974c37d8993864b8277aa624531fed689568788e400d20c134a"}
Sep 17 18:19:39.975734 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:39.975367    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-tphrr" event={"ID":"c023c631-197e-4554-b37d-8e3c35f452d7","Type":"ContainerStarted","Data":"bb0a8a9ebe7e7127ff8ed508d316ce836c284cb40163f8ad1ee9c5e2e7b171a0"}
Sep 17 18:19:40.387687 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:40.387652    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:40.980235 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:40.980191    2585 generic.go:356] "Generic (PLEG): container finished" podID="c023c631-197e-4554-b37d-8e3c35f452d7" containerID="f6e179711580c974c37d8993864b8277aa624531fed689568788e400d20c134a" exitCode=0
Sep 17 18:19:40.980622 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:40.980251    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-tphrr" event={"ID":"c023c631-197e-4554-b37d-8e3c35f452d7","Type":"ContainerDied","Data":"f6e179711580c974c37d8993864b8277aa624531fed689568788e400d20c134a"}
Sep 17 18:19:41.424681 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:41.424651    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:41.985956 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:41.985921    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-tphrr" event={"ID":"c023c631-197e-4554-b37d-8e3c35f452d7","Type":"ContainerStarted","Data":"7bbb3cb79df1a7e5df68238d8f9c56baf9617efa48f09811f9eb482dde641840"}
Sep 17 18:19:41.986365 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:41.985975    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-tphrr" event={"ID":"c023c631-197e-4554-b37d-8e3c35f452d7","Type":"ContainerStarted","Data":"fb7c5a2ab04977263dc34bcd1c4ff640653d632d3b7363a9ca39144a35979277"}
Sep 17 18:19:41.986365 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:41.986138    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-tphrr"
Sep 17 18:19:41.986365 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:41.986246    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-tphrr"
Sep 17 18:19:41.987589 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:41.987564    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-tphrr" podUID="c023c631-197e-4554-b37d-8e3c35f452d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 18:19:42.005098 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:42.005044    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-tphrr" podStartSLOduration=2.981357609 podStartE2EDuration="4.005027296s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.023669687s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:19:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:19:42.00442945 +0000 UTC m=+2718.418219890" watchObservedRunningTime="2026-09-17 18:19:42.005027296 +0000 UTC m=+2718.418817749"
Sep 17 18:19:42.989555 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:42.989516    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-tphrr" podUID="c023c631-197e-4554-b37d-8e3c35f452d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 18:19:46.576377 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:46.576343    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:46.821307 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:46.821269    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:47.995399 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:47.995373    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-tphrr"
Sep 17 18:19:47.995925 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:47.995899    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-tphrr" podUID="c023c631-197e-4554-b37d-8e3c35f452d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 18:19:57.996057 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:19:57.996021    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-tphrr" podUID="c023c631-197e-4554-b37d-8e3c35f452d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 18:20:07.995877 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:20:07.995832    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-tphrr" podUID="c023c631-197e-4554-b37d-8e3c35f452d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 18:20:17.996218 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:20:17.996176    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-tphrr" podUID="c023c631-197e-4554-b37d-8e3c35f452d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 18:20:27.996739 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:20:27.996698    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-tphrr" podUID="c023c631-197e-4554-b37d-8e3c35f452d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 18:20:37.996446 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:20:37.996402    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-tphrr" podUID="c023c631-197e-4554-b37d-8e3c35f452d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 18:20:47.996734 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:20:47.996652    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-tphrr" podUID="c023c631-197e-4554-b37d-8e3c35f452d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 18:20:55.108018 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:20:55.107982    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:57.997004 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:20:57.996966    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-tphrr"
Sep 17 18:20:58.416264 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:20:58.416181    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-tphrr"]
Sep 17 18:20:58.416470 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:20:58.416444    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-tphrr" podUID="c023c631-197e-4554-b37d-8e3c35f452d7" containerName="kserve-container" containerID="cri-o://fb7c5a2ab04977263dc34bcd1c4ff640653d632d3b7363a9ca39144a35979277" gracePeriod=30
Sep 17 18:20:58.416623 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:20:58.416526    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-tphrr" podUID="c023c631-197e-4554-b37d-8e3c35f452d7" containerName="kube-rbac-proxy" containerID="cri-o://7bbb3cb79df1a7e5df68238d8f9c56baf9617efa48f09811f9eb482dde641840" gracePeriod=30
Sep 17 18:20:58.442357 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:20:58.442331    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:58.570975 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:20:58.570945    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:58.601865 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:20:58.601835    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z"]
Sep 17 18:20:58.605050 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:20:58.605031    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z"
Sep 17 18:20:58.607549 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:20:58.607525    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:20:58.607667 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:20:58.607649    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
Sep 17 18:20:58.607737 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:20:58.607654    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
Sep 17 18:20:58.616662 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:20:58.616640    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z"]
Sep 17 18:20:58.702045 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:20:58.701957    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggpwk\" (UniqueName: \"kubernetes.io/projected/9f50ee9c-fa9c-4777-81bb-685444d46422-kube-api-access-ggpwk\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z\" (UID: \"9f50ee9c-fa9c-4777-81bb-685444d46422\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z"
Sep 17 18:20:58.702045 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:20:58.702017    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f50ee9c-fa9c-4777-81bb-685444d46422-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z\" (UID: \"9f50ee9c-fa9c-4777-81bb-685444d46422\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z"
Sep 17 18:20:58.702277 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:20:58.702068    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f50ee9c-fa9c-4777-81bb-685444d46422-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z\" (UID: \"9f50ee9c-fa9c-4777-81bb-685444d46422\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z"
Sep 17 18:20:58.702277 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:20:58.702150    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f50ee9c-fa9c-4777-81bb-685444d46422-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z\" (UID: \"9f50ee9c-fa9c-4777-81bb-685444d46422\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z"
Sep 17 18:20:58.702277 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:20:58.702229    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f50ee9c-fa9c-4777-81bb-685444d46422-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z\" (UID: \"9f50ee9c-fa9c-4777-81bb-685444d46422\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z"
Sep 17 18:20:58.802740 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:20:58.802684    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f50ee9c-fa9c-4777-81bb-685444d46422-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z\" (UID: \"9f50ee9c-fa9c-4777-81bb-685444d46422\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z"
Sep 17 18:20:58.802740 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:20:58.802751    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f50ee9c-fa9c-4777-81bb-685444d46422-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z\" (UID: \"9f50ee9c-fa9c-4777-81bb-685444d46422\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z"
Sep 17 18:20:58.802989 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:20:58.802773    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f50ee9c-fa9c-4777-81bb-685444d46422-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z\" (UID: \"9f50ee9c-fa9c-4777-81bb-685444d46422\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z"
Sep 17 18:20:58.802989 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:20:58.802815    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f50ee9c-fa9c-4777-81bb-685444d46422-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z\" (UID: \"9f50ee9c-fa9c-4777-81bb-685444d46422\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z"
Sep 17 18:20:58.802989 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:20:58.802837    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggpwk\" (UniqueName: \"kubernetes.io/projected/9f50ee9c-fa9c-4777-81bb-685444d46422-kube-api-access-ggpwk\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z\" (UID: \"9f50ee9c-fa9c-4777-81bb-685444d46422\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z"
Sep 17 18:20:58.802989 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:20:58.802876    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-global-pass-predictor-serving-cert" not found
Sep 17 18:20:58.802989 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:20:58.802961    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f50ee9c-fa9c-4777-81bb-685444d46422-proxy-tls podName:9f50ee9c-fa9c-4777-81bb-685444d46422 nodeName:}" failed. No retries permitted until 2026-09-17 18:20:59.302940531 +0000 UTC m=+2795.716730974 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9f50ee9c-fa9c-4777-81bb-685444d46422-proxy-tls") pod "isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z" (UID: "9f50ee9c-fa9c-4777-81bb-685444d46422") : secret "isvc-sklearn-s3-tls-global-pass-predictor-serving-cert" not found
Sep 17 18:20:58.803289 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:20:58.803271    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f50ee9c-fa9c-4777-81bb-685444d46422-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z\" (UID: \"9f50ee9c-fa9c-4777-81bb-685444d46422\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z"
Sep 17 18:20:58.803582 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:20:58.803559    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f50ee9c-fa9c-4777-81bb-685444d46422-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z\" (UID: \"9f50ee9c-fa9c-4777-81bb-685444d46422\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z"
Sep 17 18:20:58.803630 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:20:58.803559    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f50ee9c-fa9c-4777-81bb-685444d46422-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z\" (UID: \"9f50ee9c-fa9c-4777-81bb-685444d46422\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z"
Sep 17 18:20:58.813912 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:20:58.813878    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggpwk\" (UniqueName: \"kubernetes.io/projected/9f50ee9c-fa9c-4777-81bb-685444d46422-kube-api-access-ggpwk\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z\" (UID: \"9f50ee9c-fa9c-4777-81bb-685444d46422\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z"
Sep 17 18:20:59.031595 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:20:59.031558    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:59.270034 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:20:59.269999    2585 generic.go:356] "Generic (PLEG): container finished" podID="c023c631-197e-4554-b37d-8e3c35f452d7" containerID="7bbb3cb79df1a7e5df68238d8f9c56baf9617efa48f09811f9eb482dde641840" exitCode=2
Sep 17 18:20:59.270183 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:20:59.270071    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-tphrr" event={"ID":"c023c631-197e-4554-b37d-8e3c35f452d7","Type":"ContainerDied","Data":"7bbb3cb79df1a7e5df68238d8f9c56baf9617efa48f09811f9eb482dde641840"}
Sep 17 18:20:59.307525 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:20:59.307437    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f50ee9c-fa9c-4777-81bb-685444d46422-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z\" (UID: \"9f50ee9c-fa9c-4777-81bb-685444d46422\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z"
Sep 17 18:20:59.309932 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:20:59.309909    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f50ee9c-fa9c-4777-81bb-685444d46422-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z\" (UID: \"9f50ee9c-fa9c-4777-81bb-685444d46422\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z"
Sep 17 18:20:59.514689 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:20:59.514645    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z"
Sep 17 18:20:59.790403 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:20:59.790373    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z"]
Sep 17 18:20:59.793151 ip-10-0-133-139 kubenswrapper[2585]: W0917 18:20:59.793122    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f50ee9c_fa9c_4777_81bb_685444d46422.slice/crio-4af86305c559f20e3ab1747f8efdbdc024db9a85bb02d2e127de68dea2c3185a WatchSource:0}: Error finding container 4af86305c559f20e3ab1747f8efdbdc024db9a85bb02d2e127de68dea2c3185a: Status 404 returned error can't find the container with id 4af86305c559f20e3ab1747f8efdbdc024db9a85bb02d2e127de68dea2c3185a
Sep 17 18:21:00.280379 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:00.280343    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z" event={"ID":"9f50ee9c-fa9c-4777-81bb-685444d46422","Type":"ContainerStarted","Data":"df594bc073987ede9e5e133bf877e74cc677e31e3b20edbfde0ef2c9ad723f52"}
Sep 17 18:21:00.280829 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:00.280420    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z" event={"ID":"9f50ee9c-fa9c-4777-81bb-685444d46422","Type":"ContainerStarted","Data":"4af86305c559f20e3ab1747f8efdbdc024db9a85bb02d2e127de68dea2c3185a"}
Sep 17 18:21:01.285476 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:01.285439    2585 generic.go:356] "Generic (PLEG): container finished" podID="9f50ee9c-fa9c-4777-81bb-685444d46422" containerID="df594bc073987ede9e5e133bf877e74cc677e31e3b20edbfde0ef2c9ad723f52" exitCode=0
Sep 17 18:21:01.285871 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:01.285528    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z" event={"ID":"9f50ee9c-fa9c-4777-81bb-685444d46422","Type":"ContainerDied","Data":"df594bc073987ede9e5e133bf877e74cc677e31e3b20edbfde0ef2c9ad723f52"}
Sep 17 18:21:02.291082 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:02.291048    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z" event={"ID":"9f50ee9c-fa9c-4777-81bb-685444d46422","Type":"ContainerStarted","Data":"9c87438347b8de58f0c12bf7f65e729ffcff36c9fe4f4356d3e9a0ac5c48e7ab"}
Sep 17 18:21:02.291581 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:02.291103    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z" event={"ID":"9f50ee9c-fa9c-4777-81bb-685444d46422","Type":"ContainerStarted","Data":"06b6149245d3ce54aee36fc3c2d3b8d2e37eaa86de1a8a9e118484167f7e8e38"}
Sep 17 18:21:02.291581 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:02.291210    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z"
Sep 17 18:21:02.310747 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:02.310686    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z" podStartSLOduration=3.208446098 podStartE2EDuration="4.310667437s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.102221339s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:20:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:21:02.310101122 +0000 UTC m=+2798.723891559" watchObservedRunningTime="2026-09-17 18:21:02.310667437 +0000 UTC m=+2798.724457878"
Sep 17 18:21:02.990665 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:02.990624    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-tphrr" podUID="c023c631-197e-4554-b37d-8e3c35f452d7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.43:8643/healthz\": dial tcp 10.133.0.43:8643: connect: connection refused"
Sep 17 18:21:03.250759 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:03.250737    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-tphrr"
Sep 17 18:21:03.296227 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:03.296193    2585 generic.go:356] "Generic (PLEG): container finished" podID="c023c631-197e-4554-b37d-8e3c35f452d7" containerID="fb7c5a2ab04977263dc34bcd1c4ff640653d632d3b7363a9ca39144a35979277" exitCode=0
Sep 17 18:21:03.296601 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:03.296254    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-tphrr"
Sep 17 18:21:03.296601 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:03.296268    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-tphrr" event={"ID":"c023c631-197e-4554-b37d-8e3c35f452d7","Type":"ContainerDied","Data":"fb7c5a2ab04977263dc34bcd1c4ff640653d632d3b7363a9ca39144a35979277"}
Sep 17 18:21:03.296601 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:03.296338    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-tphrr" event={"ID":"c023c631-197e-4554-b37d-8e3c35f452d7","Type":"ContainerDied","Data":"bb0a8a9ebe7e7127ff8ed508d316ce836c284cb40163f8ad1ee9c5e2e7b171a0"}
Sep 17 18:21:03.296601 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:03.296370    2585 scope.go:117] "RemoveContainer" containerID="7bbb3cb79df1a7e5df68238d8f9c56baf9617efa48f09811f9eb482dde641840"
Sep 17 18:21:03.296810 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:03.296708    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z"
Sep 17 18:21:03.297829 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:03.297802    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z" podUID="9f50ee9c-fa9c-4777-81bb-685444d46422" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 17 18:21:03.304162 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:03.304147    2585 scope.go:117] "RemoveContainer" containerID="fb7c5a2ab04977263dc34bcd1c4ff640653d632d3b7363a9ca39144a35979277"
Sep 17 18:21:03.311171 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:03.311150    2585 scope.go:117] "RemoveContainer" containerID="f6e179711580c974c37d8993864b8277aa624531fed689568788e400d20c134a"
Sep 17 18:21:03.317912 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:03.317894    2585 scope.go:117] "RemoveContainer" containerID="7bbb3cb79df1a7e5df68238d8f9c56baf9617efa48f09811f9eb482dde641840"
Sep 17 18:21:03.318163 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:21:03.318146    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bbb3cb79df1a7e5df68238d8f9c56baf9617efa48f09811f9eb482dde641840\": container with ID starting with 7bbb3cb79df1a7e5df68238d8f9c56baf9617efa48f09811f9eb482dde641840 not found: ID does not exist" containerID="7bbb3cb79df1a7e5df68238d8f9c56baf9617efa48f09811f9eb482dde641840"
Sep 17 18:21:03.318224 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:03.318170    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bbb3cb79df1a7e5df68238d8f9c56baf9617efa48f09811f9eb482dde641840"} err="failed to get container status \"7bbb3cb79df1a7e5df68238d8f9c56baf9617efa48f09811f9eb482dde641840\": rpc error: code = NotFound desc = could not find container \"7bbb3cb79df1a7e5df68238d8f9c56baf9617efa48f09811f9eb482dde641840\": container with ID starting with 7bbb3cb79df1a7e5df68238d8f9c56baf9617efa48f09811f9eb482dde641840 not found: ID does not exist"
Sep 17 18:21:03.318224 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:03.318186    2585 scope.go:117] "RemoveContainer" containerID="fb7c5a2ab04977263dc34bcd1c4ff640653d632d3b7363a9ca39144a35979277"
Sep 17 18:21:03.318402 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:21:03.318387    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb7c5a2ab04977263dc34bcd1c4ff640653d632d3b7363a9ca39144a35979277\": container with ID starting with fb7c5a2ab04977263dc34bcd1c4ff640653d632d3b7363a9ca39144a35979277 not found: ID does not exist" containerID="fb7c5a2ab04977263dc34bcd1c4ff640653d632d3b7363a9ca39144a35979277"
Sep 17 18:21:03.318442 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:03.318407    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb7c5a2ab04977263dc34bcd1c4ff640653d632d3b7363a9ca39144a35979277"} err="failed to get container status \"fb7c5a2ab04977263dc34bcd1c4ff640653d632d3b7363a9ca39144a35979277\": rpc error: code = NotFound desc = could not find container \"fb7c5a2ab04977263dc34bcd1c4ff640653d632d3b7363a9ca39144a35979277\": container with ID starting with fb7c5a2ab04977263dc34bcd1c4ff640653d632d3b7363a9ca39144a35979277 not found: ID does not exist"
Sep 17 18:21:03.318442 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:03.318419    2585 scope.go:117] "RemoveContainer" containerID="f6e179711580c974c37d8993864b8277aa624531fed689568788e400d20c134a"
Sep 17 18:21:03.318663 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:21:03.318649    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6e179711580c974c37d8993864b8277aa624531fed689568788e400d20c134a\": container with ID starting with f6e179711580c974c37d8993864b8277aa624531fed689568788e400d20c134a not found: ID does not exist" containerID="f6e179711580c974c37d8993864b8277aa624531fed689568788e400d20c134a"
Sep 17 18:21:03.318720 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:03.318666    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6e179711580c974c37d8993864b8277aa624531fed689568788e400d20c134a"} err="failed to get container status \"f6e179711580c974c37d8993864b8277aa624531fed689568788e400d20c134a\": rpc error: code = NotFound desc = could not find container \"f6e179711580c974c37d8993864b8277aa624531fed689568788e400d20c134a\": container with ID starting with f6e179711580c974c37d8993864b8277aa624531fed689568788e400d20c134a not found: ID does not exist"
Sep 17 18:21:03.335978 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:03.335961    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7smc\" (UniqueName: \"kubernetes.io/projected/c023c631-197e-4554-b37d-8e3c35f452d7-kube-api-access-b7smc\") pod \"c023c631-197e-4554-b37d-8e3c35f452d7\" (UID: \"c023c631-197e-4554-b37d-8e3c35f452d7\") "
Sep 17 18:21:03.336053 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:03.336000    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c023c631-197e-4554-b37d-8e3c35f452d7-proxy-tls\") pod \"c023c631-197e-4554-b37d-8e3c35f452d7\" (UID: \"c023c631-197e-4554-b37d-8e3c35f452d7\") "
Sep 17 18:21:03.336053 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:03.336024    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c023c631-197e-4554-b37d-8e3c35f452d7-kserve-provision-location\") pod \"c023c631-197e-4554-b37d-8e3c35f452d7\" (UID: \"c023c631-197e-4554-b37d-8e3c35f452d7\") "
Sep 17 18:21:03.336053 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:03.336041    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c023c631-197e-4554-b37d-8e3c35f452d7-kube-rbac-proxy-sar-config\") pod \"c023c631-197e-4554-b37d-8e3c35f452d7\" (UID: \"c023c631-197e-4554-b37d-8e3c35f452d7\") "
Sep 17 18:21:03.336345 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:03.336322    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c023c631-197e-4554-b37d-8e3c35f452d7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c023c631-197e-4554-b37d-8e3c35f452d7" (UID: "c023c631-197e-4554-b37d-8e3c35f452d7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:21:03.336518 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:03.336428    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c023c631-197e-4554-b37d-8e3c35f452d7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c023c631-197e-4554-b37d-8e3c35f452d7" (UID: "c023c631-197e-4554-b37d-8e3c35f452d7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:21:03.337908 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:03.337887    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c023c631-197e-4554-b37d-8e3c35f452d7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c023c631-197e-4554-b37d-8e3c35f452d7" (UID: "c023c631-197e-4554-b37d-8e3c35f452d7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:21:03.338207 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:03.338189    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c023c631-197e-4554-b37d-8e3c35f452d7-kube-api-access-b7smc" (OuterVolumeSpecName: "kube-api-access-b7smc") pod "c023c631-197e-4554-b37d-8e3c35f452d7" (UID: "c023c631-197e-4554-b37d-8e3c35f452d7"). InnerVolumeSpecName "kube-api-access-b7smc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:21:03.437445 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:03.437410    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c023c631-197e-4554-b37d-8e3c35f452d7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:21:03.437445 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:03.437437    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7smc\" (UniqueName: \"kubernetes.io/projected/c023c631-197e-4554-b37d-8e3c35f452d7-kube-api-access-b7smc\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:21:03.437445 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:03.437447    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c023c631-197e-4554-b37d-8e3c35f452d7-proxy-tls\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:21:03.437445 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:03.437455    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c023c631-197e-4554-b37d-8e3c35f452d7-kserve-provision-location\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:21:03.622312 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:03.622280    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-tphrr"]
Sep 17 18:21:03.625998 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:03.625975    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-84d46f45-tphrr"]
Sep 17 18:21:04.105818 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:04.105780    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c023c631-197e-4554-b37d-8e3c35f452d7" path="/var/lib/kubelet/pods/c023c631-197e-4554-b37d-8e3c35f452d7/volumes"
Sep 17 18:21:04.301809 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:04.301770    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z" podUID="9f50ee9c-fa9c-4777-81bb-685444d46422" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 17 18:21:06.677916 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:06.677878    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:09.307076 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:09.307048    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z"
Sep 17 18:21:09.307567 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:09.307539    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z" podUID="9f50ee9c-fa9c-4777-81bb-685444d46422" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 17 18:21:11.820548 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:11.820478    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:11.849948 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:11.849917    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bqq4q"]
Sep 17 18:21:11.850174 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:11.850163    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c023c631-197e-4554-b37d-8e3c35f452d7" containerName="storage-initializer"
Sep 17 18:21:11.850219 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:11.850175    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c023c631-197e-4554-b37d-8e3c35f452d7" containerName="storage-initializer"
Sep 17 18:21:11.850219 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:11.850184    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c023c631-197e-4554-b37d-8e3c35f452d7" containerName="kserve-container"
Sep 17 18:21:11.850219 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:11.850190    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c023c631-197e-4554-b37d-8e3c35f452d7" containerName="kserve-container"
Sep 17 18:21:11.850219 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:11.850207    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c023c631-197e-4554-b37d-8e3c35f452d7" containerName="kube-rbac-proxy"
Sep 17 18:21:11.850219 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:11.850213    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c023c631-197e-4554-b37d-8e3c35f452d7" containerName="kube-rbac-proxy"
Sep 17 18:21:11.850373 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:11.850257    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c023c631-197e-4554-b37d-8e3c35f452d7" containerName="kserve-container"
Sep 17 18:21:11.850373 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:11.850266    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c023c631-197e-4554-b37d-8e3c35f452d7" containerName="kube-rbac-proxy"
Sep 17 18:21:11.853213 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:11.853196    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bqq4q"
Sep 17 18:21:11.855703 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:11.855680    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-predictor-serving-cert\""
Sep 17 18:21:11.855826 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:11.855705    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-kube-rbac-proxy-sar-config\""
Sep 17 18:21:11.855873 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:11.855860    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:21:11.862594 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:11.862572    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bqq4q"]
Sep 17 18:21:11.898719 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:11.898686    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1fcef854-6afc-4563-83e9-19758c935a18-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-bqq4q\" (UID: \"1fcef854-6afc-4563-83e9-19758c935a18\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bqq4q"
Sep 17 18:21:11.898905 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:11.898751    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1fcef854-6afc-4563-83e9-19758c935a18-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-bqq4q\" (UID: \"1fcef854-6afc-4563-83e9-19758c935a18\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bqq4q"
Sep 17 18:21:11.898905 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:11.898812    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1fcef854-6afc-4563-83e9-19758c935a18-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-bqq4q\" (UID: \"1fcef854-6afc-4563-83e9-19758c935a18\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bqq4q"
Sep 17 18:21:11.898905 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:11.898839    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1fcef854-6afc-4563-83e9-19758c935a18-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-bqq4q\" (UID: \"1fcef854-6afc-4563-83e9-19758c935a18\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bqq4q"
Sep 17 18:21:11.899343 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:11.898906    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n92ps\" (UniqueName: \"kubernetes.io/projected/1fcef854-6afc-4563-83e9-19758c935a18-kube-api-access-n92ps\") pod \"isvc-paddle-predictor-6d5c9fd4f6-bqq4q\" (UID: \"1fcef854-6afc-4563-83e9-19758c935a18\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bqq4q"
Sep 17 18:21:11.999759 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:11.999728    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n92ps\" (UniqueName: \"kubernetes.io/projected/1fcef854-6afc-4563-83e9-19758c935a18-kube-api-access-n92ps\") pod \"isvc-paddle-predictor-6d5c9fd4f6-bqq4q\" (UID: \"1fcef854-6afc-4563-83e9-19758c935a18\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bqq4q"
Sep 17 18:21:11.999759 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:11.999759    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1fcef854-6afc-4563-83e9-19758c935a18-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-bqq4q\" (UID: \"1fcef854-6afc-4563-83e9-19758c935a18\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bqq4q"
Sep 17 18:21:12.000044 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:11.999789    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1fcef854-6afc-4563-83e9-19758c935a18-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-bqq4q\" (UID: \"1fcef854-6afc-4563-83e9-19758c935a18\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bqq4q"
Sep 17 18:21:12.000044 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:11.999810    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1fcef854-6afc-4563-83e9-19758c935a18-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-bqq4q\" (UID: \"1fcef854-6afc-4563-83e9-19758c935a18\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bqq4q"
Sep 17 18:21:12.000044 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:11.999838    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1fcef854-6afc-4563-83e9-19758c935a18-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-bqq4q\" (UID: \"1fcef854-6afc-4563-83e9-19758c935a18\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bqq4q"
Sep 17 18:21:12.000044 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:21:11.999967    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-predictor-serving-cert: secret "isvc-paddle-predictor-serving-cert" not found
Sep 17 18:21:12.000044 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:21:12.000037    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fcef854-6afc-4563-83e9-19758c935a18-proxy-tls podName:1fcef854-6afc-4563-83e9-19758c935a18 nodeName:}" failed. No retries permitted until 2026-09-17 18:21:12.500014859 +0000 UTC m=+2808.913805298 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1fcef854-6afc-4563-83e9-19758c935a18-proxy-tls") pod "isvc-paddle-predictor-6d5c9fd4f6-bqq4q" (UID: "1fcef854-6afc-4563-83e9-19758c935a18") : secret "isvc-paddle-predictor-serving-cert" not found
Sep 17 18:21:12.000342 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:12.000194    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1fcef854-6afc-4563-83e9-19758c935a18-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-bqq4q\" (UID: \"1fcef854-6afc-4563-83e9-19758c935a18\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bqq4q"
Sep 17 18:21:12.000658 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:12.000634    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1fcef854-6afc-4563-83e9-19758c935a18-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-bqq4q\" (UID: \"1fcef854-6afc-4563-83e9-19758c935a18\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bqq4q"
Sep 17 18:21:12.000762 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:12.000634    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1fcef854-6afc-4563-83e9-19758c935a18-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-bqq4q\" (UID: \"1fcef854-6afc-4563-83e9-19758c935a18\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bqq4q"
Sep 17 18:21:12.011445 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:12.011421    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n92ps\" (UniqueName: \"kubernetes.io/projected/1fcef854-6afc-4563-83e9-19758c935a18-kube-api-access-n92ps\") pod \"isvc-paddle-predictor-6d5c9fd4f6-bqq4q\" (UID: \"1fcef854-6afc-4563-83e9-19758c935a18\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bqq4q"
Sep 17 18:21:12.400989 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:12.400942    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:12.504132 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:12.504092    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1fcef854-6afc-4563-83e9-19758c935a18-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-bqq4q\" (UID: \"1fcef854-6afc-4563-83e9-19758c935a18\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bqq4q"
Sep 17 18:21:12.506483 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:12.506452    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1fcef854-6afc-4563-83e9-19758c935a18-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-bqq4q\" (UID: \"1fcef854-6afc-4563-83e9-19758c935a18\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bqq4q"
Sep 17 18:21:12.764068 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:12.764032    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bqq4q"
Sep 17 18:21:13.036159 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:13.036131    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bqq4q"]
Sep 17 18:21:13.038371 ip-10-0-133-139 kubenswrapper[2585]: W0917 18:21:13.038341    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fcef854_6afc_4563_83e9_19758c935a18.slice/crio-937858c871936308de292cb24de2e7c7ebbadcf6e707267a2f722e7d54187c4f WatchSource:0}: Error finding container 937858c871936308de292cb24de2e7c7ebbadcf6e707267a2f722e7d54187c4f: Status 404 returned error can't find the container with id 937858c871936308de292cb24de2e7c7ebbadcf6e707267a2f722e7d54187c4f
Sep 17 18:21:13.337406 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:13.337320    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bqq4q" event={"ID":"1fcef854-6afc-4563-83e9-19758c935a18","Type":"ContainerStarted","Data":"94d7946592350096a5ac5e02499f5a23a4d9974dbc15144101661c01200a994f"}
Sep 17 18:21:13.337406 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:13.337396    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bqq4q" event={"ID":"1fcef854-6afc-4563-83e9-19758c935a18","Type":"ContainerStarted","Data":"937858c871936308de292cb24de2e7c7ebbadcf6e707267a2f722e7d54187c4f"}
Sep 17 18:21:18.358814 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:18.358781    2585 generic.go:356] "Generic (PLEG): container finished" podID="1fcef854-6afc-4563-83e9-19758c935a18" containerID="94d7946592350096a5ac5e02499f5a23a4d9974dbc15144101661c01200a994f" exitCode=0
Sep 17 18:21:18.359237 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:18.358849    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bqq4q" event={"ID":"1fcef854-6afc-4563-83e9-19758c935a18","Type":"ContainerDied","Data":"94d7946592350096a5ac5e02499f5a23a4d9974dbc15144101661c01200a994f"}
Sep 17 18:21:18.360079 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:18.360064    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:21:19.308468 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:19.308421    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z" podUID="9f50ee9c-fa9c-4777-81bb-685444d46422" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 17 18:21:29.308310 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:29.308257    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z" podUID="9f50ee9c-fa9c-4777-81bb-685444d46422" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 17 18:21:31.422791 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:31.422756    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bqq4q" event={"ID":"1fcef854-6afc-4563-83e9-19758c935a18","Type":"ContainerStarted","Data":"3698dea7af00c4749a323a8f8af1e906b7de31f2912e00cc6efb518b0ca78b78"}
Sep 17 18:21:31.423162 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:31.422817    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bqq4q" event={"ID":"1fcef854-6afc-4563-83e9-19758c935a18","Type":"ContainerStarted","Data":"f03632178b15bb9f510e9f404187bd927384bf392d2d93f1c656a2cdf3414973"}
Sep 17 18:21:31.423162 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:31.423103    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bqq4q"
Sep 17 18:21:31.448813 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:31.448762    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bqq4q" podStartSLOduration=3.865904257 podStartE2EDuration="20.448746968s" totalImagesPullingTime="12.22508932s" totalInitContainerRuntime="4.357753391s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:21:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:21:31.448049832 +0000 UTC m=+2827.861840271" watchObservedRunningTime="2026-09-17 18:21:31.448746968 +0000 UTC m=+2827.862537404"
Sep 17 18:21:32.427139 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:32.427102    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bqq4q"
Sep 17 18:21:32.428252 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:32.428228    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bqq4q" podUID="1fcef854-6afc-4563-83e9-19758c935a18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 17 18:21:33.430957 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:33.430920    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bqq4q" podUID="1fcef854-6afc-4563-83e9-19758c935a18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 17 18:21:38.435169 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:38.435140    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bqq4q"
Sep 17 18:21:38.435802 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:38.435772    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bqq4q" podUID="1fcef854-6afc-4563-83e9-19758c935a18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 17 18:21:39.307543 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:39.307480    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z" podUID="9f50ee9c-fa9c-4777-81bb-685444d46422" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 17 18:21:48.436141 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:48.436095    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bqq4q" podUID="1fcef854-6afc-4563-83e9-19758c935a18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 17 18:21:49.308276 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:49.308227    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z" podUID="9f50ee9c-fa9c-4777-81bb-685444d46422" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 17 18:21:56.655086 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:56.655049    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:58.435701 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:58.435662    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bqq4q" podUID="1fcef854-6afc-4563-83e9-19758c935a18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 17 18:21:59.307710 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:21:59.307660    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z" podUID="9f50ee9c-fa9c-4777-81bb-685444d46422" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 17 18:22:08.436117 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:08.436033    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bqq4q" podUID="1fcef854-6afc-4563-83e9-19758c935a18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 17 18:22:09.308296 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:09.308253    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z" podUID="9f50ee9c-fa9c-4777-81bb-685444d46422" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 17 18:22:18.436734 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:18.436659    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bqq4q"
Sep 17 18:22:19.309265 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:19.309240    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z"
Sep 17 18:22:23.255531 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:23.255478    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bqq4q"]
Sep 17 18:22:23.255940 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:23.255703    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bqq4q" podUID="1fcef854-6afc-4563-83e9-19758c935a18" containerName="kserve-container" containerID="cri-o://f03632178b15bb9f510e9f404187bd927384bf392d2d93f1c656a2cdf3414973" gracePeriod=30
Sep 17 18:22:23.255940 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:23.255741    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bqq4q" podUID="1fcef854-6afc-4563-83e9-19758c935a18" containerName="kube-rbac-proxy" containerID="cri-o://3698dea7af00c4749a323a8f8af1e906b7de31f2912e00cc6efb518b0ca78b78" gracePeriod=30
Sep 17 18:22:23.271695 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:23.271671    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:23.431288 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:23.431227    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bqq4q" podUID="1fcef854-6afc-4563-83e9-19758c935a18" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.45:8643/healthz\": dial tcp 10.133.0.45:8643: connect: connection refused"
Sep 17 18:22:23.618798 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:23.618713    2585 generic.go:356] "Generic (PLEG): container finished" podID="1fcef854-6afc-4563-83e9-19758c935a18" containerID="3698dea7af00c4749a323a8f8af1e906b7de31f2912e00cc6efb518b0ca78b78" exitCode=2
Sep 17 18:22:23.618798 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:23.618778    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bqq4q" event={"ID":"1fcef854-6afc-4563-83e9-19758c935a18","Type":"ContainerDied","Data":"3698dea7af00c4749a323a8f8af1e906b7de31f2912e00cc6efb518b0ca78b78"}
Sep 17 18:22:26.095086 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:26.095057    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bqq4q"
Sep 17 18:22:26.138938 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:26.138909    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1fcef854-6afc-4563-83e9-19758c935a18-proxy-tls\") pod \"1fcef854-6afc-4563-83e9-19758c935a18\" (UID: \"1fcef854-6afc-4563-83e9-19758c935a18\") "
Sep 17 18:22:26.139094 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:26.138952    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1fcef854-6afc-4563-83e9-19758c935a18-kserve-provision-location\") pod \"1fcef854-6afc-4563-83e9-19758c935a18\" (UID: \"1fcef854-6afc-4563-83e9-19758c935a18\") "
Sep 17 18:22:26.139094 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:26.138980    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n92ps\" (UniqueName: \"kubernetes.io/projected/1fcef854-6afc-4563-83e9-19758c935a18-kube-api-access-n92ps\") pod \"1fcef854-6afc-4563-83e9-19758c935a18\" (UID: \"1fcef854-6afc-4563-83e9-19758c935a18\") "
Sep 17 18:22:26.139094 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:26.139022    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1fcef854-6afc-4563-83e9-19758c935a18-kube-rbac-proxy-sar-config\") pod \"1fcef854-6afc-4563-83e9-19758c935a18\" (UID: \"1fcef854-6afc-4563-83e9-19758c935a18\") "
Sep 17 18:22:26.139094 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:26.139050    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1fcef854-6afc-4563-83e9-19758c935a18-cabundle-cert\") pod \"1fcef854-6afc-4563-83e9-19758c935a18\" (UID: \"1fcef854-6afc-4563-83e9-19758c935a18\") "
Sep 17 18:22:26.139422 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:26.139395    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fcef854-6afc-4563-83e9-19758c935a18-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1fcef854-6afc-4563-83e9-19758c935a18" (UID: "1fcef854-6afc-4563-83e9-19758c935a18"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:22:26.139524 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:26.139444    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fcef854-6afc-4563-83e9-19758c935a18-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1fcef854-6afc-4563-83e9-19758c935a18" (UID: "1fcef854-6afc-4563-83e9-19758c935a18"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:22:26.141103 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:26.141082    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fcef854-6afc-4563-83e9-19758c935a18-kube-api-access-n92ps" (OuterVolumeSpecName: "kube-api-access-n92ps") pod "1fcef854-6afc-4563-83e9-19758c935a18" (UID: "1fcef854-6afc-4563-83e9-19758c935a18"). InnerVolumeSpecName "kube-api-access-n92ps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:22:26.141184 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:26.141120    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fcef854-6afc-4563-83e9-19758c935a18-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1fcef854-6afc-4563-83e9-19758c935a18" (UID: "1fcef854-6afc-4563-83e9-19758c935a18"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:22:26.148224 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:26.148201    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fcef854-6afc-4563-83e9-19758c935a18-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1fcef854-6afc-4563-83e9-19758c935a18" (UID: "1fcef854-6afc-4563-83e9-19758c935a18"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:22:26.240228 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:26.240150    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1fcef854-6afc-4563-83e9-19758c935a18-kserve-provision-location\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:22:26.240228 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:26.240176    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n92ps\" (UniqueName: \"kubernetes.io/projected/1fcef854-6afc-4563-83e9-19758c935a18-kube-api-access-n92ps\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:22:26.240228 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:26.240185    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1fcef854-6afc-4563-83e9-19758c935a18-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:22:26.240228 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:26.240194    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1fcef854-6afc-4563-83e9-19758c935a18-cabundle-cert\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:22:26.240228 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:26.240203    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1fcef854-6afc-4563-83e9-19758c935a18-proxy-tls\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:22:26.631482 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:26.631445    2585 generic.go:356] "Generic (PLEG): container finished" podID="1fcef854-6afc-4563-83e9-19758c935a18" containerID="f03632178b15bb9f510e9f404187bd927384bf392d2d93f1c656a2cdf3414973" exitCode=0
Sep 17 18:22:26.631644 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:26.631518    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bqq4q" event={"ID":"1fcef854-6afc-4563-83e9-19758c935a18","Type":"ContainerDied","Data":"f03632178b15bb9f510e9f404187bd927384bf392d2d93f1c656a2cdf3414973"}
Sep 17 18:22:26.631644 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:26.631574    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bqq4q" event={"ID":"1fcef854-6afc-4563-83e9-19758c935a18","Type":"ContainerDied","Data":"937858c871936308de292cb24de2e7c7ebbadcf6e707267a2f722e7d54187c4f"}
Sep 17 18:22:26.631644 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:26.631601    2585 scope.go:117] "RemoveContainer" containerID="3698dea7af00c4749a323a8f8af1e906b7de31f2912e00cc6efb518b0ca78b78"
Sep 17 18:22:26.631644 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:26.631521    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bqq4q"
Sep 17 18:22:26.640081 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:26.640060    2585 scope.go:117] "RemoveContainer" containerID="f03632178b15bb9f510e9f404187bd927384bf392d2d93f1c656a2cdf3414973"
Sep 17 18:22:26.647254 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:26.647233    2585 scope.go:117] "RemoveContainer" containerID="94d7946592350096a5ac5e02499f5a23a4d9974dbc15144101661c01200a994f"
Sep 17 18:22:26.654207 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:26.654187    2585 scope.go:117] "RemoveContainer" containerID="3698dea7af00c4749a323a8f8af1e906b7de31f2912e00cc6efb518b0ca78b78"
Sep 17 18:22:26.654475 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:22:26.654456    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3698dea7af00c4749a323a8f8af1e906b7de31f2912e00cc6efb518b0ca78b78\": container with ID starting with 3698dea7af00c4749a323a8f8af1e906b7de31f2912e00cc6efb518b0ca78b78 not found: ID does not exist" containerID="3698dea7af00c4749a323a8f8af1e906b7de31f2912e00cc6efb518b0ca78b78"
Sep 17 18:22:26.654533 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:26.654484    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3698dea7af00c4749a323a8f8af1e906b7de31f2912e00cc6efb518b0ca78b78"} err="failed to get container status \"3698dea7af00c4749a323a8f8af1e906b7de31f2912e00cc6efb518b0ca78b78\": rpc error: code = NotFound desc = could not find container \"3698dea7af00c4749a323a8f8af1e906b7de31f2912e00cc6efb518b0ca78b78\": container with ID starting with 3698dea7af00c4749a323a8f8af1e906b7de31f2912e00cc6efb518b0ca78b78 not found: ID does not exist"
Sep 17 18:22:26.654533 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:26.654523    2585 scope.go:117] "RemoveContainer" containerID="f03632178b15bb9f510e9f404187bd927384bf392d2d93f1c656a2cdf3414973"
Sep 17 18:22:26.654810 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:22:26.654792    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f03632178b15bb9f510e9f404187bd927384bf392d2d93f1c656a2cdf3414973\": container with ID starting with f03632178b15bb9f510e9f404187bd927384bf392d2d93f1c656a2cdf3414973 not found: ID does not exist" containerID="f03632178b15bb9f510e9f404187bd927384bf392d2d93f1c656a2cdf3414973"
Sep 17 18:22:26.654854 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:26.654815    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f03632178b15bb9f510e9f404187bd927384bf392d2d93f1c656a2cdf3414973"} err="failed to get container status \"f03632178b15bb9f510e9f404187bd927384bf392d2d93f1c656a2cdf3414973\": rpc error: code = NotFound desc = could not find container \"f03632178b15bb9f510e9f404187bd927384bf392d2d93f1c656a2cdf3414973\": container with ID starting with f03632178b15bb9f510e9f404187bd927384bf392d2d93f1c656a2cdf3414973 not found: ID does not exist"
Sep 17 18:22:26.654854 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:26.654831    2585 scope.go:117] "RemoveContainer" containerID="94d7946592350096a5ac5e02499f5a23a4d9974dbc15144101661c01200a994f"
Sep 17 18:22:26.655065 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:22:26.655047    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94d7946592350096a5ac5e02499f5a23a4d9974dbc15144101661c01200a994f\": container with ID starting with 94d7946592350096a5ac5e02499f5a23a4d9974dbc15144101661c01200a994f not found: ID does not exist" containerID="94d7946592350096a5ac5e02499f5a23a4d9974dbc15144101661c01200a994f"
Sep 17 18:22:26.655104 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:26.655070    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94d7946592350096a5ac5e02499f5a23a4d9974dbc15144101661c01200a994f"} err="failed to get container status \"94d7946592350096a5ac5e02499f5a23a4d9974dbc15144101661c01200a994f\": rpc error: code = NotFound desc = could not find container \"94d7946592350096a5ac5e02499f5a23a4d9974dbc15144101661c01200a994f\": container with ID starting with 94d7946592350096a5ac5e02499f5a23a4d9974dbc15144101661c01200a994f not found: ID does not exist"
Sep 17 18:22:26.659448 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:26.659426    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bqq4q"]
Sep 17 18:22:26.662376 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:26.662355    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-bqq4q"]
Sep 17 18:22:28.103986 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:28.103952    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1fcef854-6afc-4563-83e9-19758c935a18" path="/var/lib/kubelet/pods/1fcef854-6afc-4563-83e9-19758c935a18/volumes"
Sep 17 18:22:28.513905 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:28.513870    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:28.766264 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:28.766183    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z"]
Sep 17 18:22:28.766414 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:28.766395    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z" podUID="9f50ee9c-fa9c-4777-81bb-685444d46422" containerName="kserve-container" containerID="cri-o://06b6149245d3ce54aee36fc3c2d3b8d2e37eaa86de1a8a9e118484167f7e8e38" gracePeriod=30
Sep 17 18:22:28.766475 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:28.766437    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z" podUID="9f50ee9c-fa9c-4777-81bb-685444d46422" containerName="kube-rbac-proxy" containerID="cri-o://9c87438347b8de58f0c12bf7f65e729ffcff36c9fe4f4356d3e9a0ac5c48e7ab" gracePeriod=30
Sep 17 18:22:28.788887 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:28.788857    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:28.913996 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:28.913960    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:29.302877 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:29.302837    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z" podUID="9f50ee9c-fa9c-4777-81bb-685444d46422" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": dial tcp 10.133.0.44:8643: connect: connection refused"
Sep 17 18:22:29.308161 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:29.308138    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z" podUID="9f50ee9c-fa9c-4777-81bb-685444d46422" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 17 18:22:29.650282 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:29.650190    2585 generic.go:356] "Generic (PLEG): container finished" podID="9f50ee9c-fa9c-4777-81bb-685444d46422" containerID="9c87438347b8de58f0c12bf7f65e729ffcff36c9fe4f4356d3e9a0ac5c48e7ab" exitCode=2
Sep 17 18:22:29.650282 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:29.650259    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z" event={"ID":"9f50ee9c-fa9c-4777-81bb-685444d46422","Type":"ContainerDied","Data":"9c87438347b8de58f0c12bf7f65e729ffcff36c9fe4f4356d3e9a0ac5c48e7ab"}
Sep 17 18:22:29.799353 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:29.799318    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:29.864413 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:29.864380    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-bhcwm"]
Sep 17 18:22:29.864654 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:29.864642    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fcef854-6afc-4563-83e9-19758c935a18" containerName="storage-initializer"
Sep 17 18:22:29.864704 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:29.864656    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fcef854-6afc-4563-83e9-19758c935a18" containerName="storage-initializer"
Sep 17 18:22:29.864704 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:29.864663    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fcef854-6afc-4563-83e9-19758c935a18" containerName="kube-rbac-proxy"
Sep 17 18:22:29.864704 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:29.864669    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fcef854-6afc-4563-83e9-19758c935a18" containerName="kube-rbac-proxy"
Sep 17 18:22:29.864704 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:29.864683    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fcef854-6afc-4563-83e9-19758c935a18" containerName="kserve-container"
Sep 17 18:22:29.864704 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:29.864688    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fcef854-6afc-4563-83e9-19758c935a18" containerName="kserve-container"
Sep 17 18:22:29.864865 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:29.864740    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fcef854-6afc-4563-83e9-19758c935a18" containerName="kserve-container"
Sep 17 18:22:29.864865 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:29.864746    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fcef854-6afc-4563-83e9-19758c935a18" containerName="kube-rbac-proxy"
Sep 17 18:22:29.867708 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:29.867689    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-bhcwm"
Sep 17 18:22:29.870044 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:29.870021    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
Sep 17 18:22:29.870811 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:29.870794    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
Sep 17 18:22:29.883937 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:29.883918    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-bhcwm"]
Sep 17 18:22:29.966803 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:29.966735    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74d8307f-490d-46cc-b49f-986e78b1e89b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-bhcwm\" (UID: \"74d8307f-490d-46cc-b49f-986e78b1e89b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-bhcwm"
Sep 17 18:22:29.966803 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:29.966779    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74d8307f-490d-46cc-b49f-986e78b1e89b-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-bhcwm\" (UID: \"74d8307f-490d-46cc-b49f-986e78b1e89b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-bhcwm"
Sep 17 18:22:29.966957 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:29.966813    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxrdw\" (UniqueName: \"kubernetes.io/projected/74d8307f-490d-46cc-b49f-986e78b1e89b-kube-api-access-zxrdw\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-bhcwm\" (UID: \"74d8307f-490d-46cc-b49f-986e78b1e89b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-bhcwm"
Sep 17 18:22:29.966957 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:29.966870    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74d8307f-490d-46cc-b49f-986e78b1e89b-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-bhcwm\" (UID: \"74d8307f-490d-46cc-b49f-986e78b1e89b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-bhcwm"
Sep 17 18:22:30.067936 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:30.067907    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74d8307f-490d-46cc-b49f-986e78b1e89b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-bhcwm\" (UID: \"74d8307f-490d-46cc-b49f-986e78b1e89b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-bhcwm"
Sep 17 18:22:30.068083 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:30.067946    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74d8307f-490d-46cc-b49f-986e78b1e89b-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-bhcwm\" (UID: \"74d8307f-490d-46cc-b49f-986e78b1e89b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-bhcwm"
Sep 17 18:22:30.068083 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:30.067971    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxrdw\" (UniqueName: \"kubernetes.io/projected/74d8307f-490d-46cc-b49f-986e78b1e89b-kube-api-access-zxrdw\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-bhcwm\" (UID: \"74d8307f-490d-46cc-b49f-986e78b1e89b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-bhcwm"
Sep 17 18:22:30.068083 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:30.067990    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74d8307f-490d-46cc-b49f-986e78b1e89b-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-bhcwm\" (UID: \"74d8307f-490d-46cc-b49f-986e78b1e89b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-bhcwm"
Sep 17 18:22:30.068337 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:30.068315    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74d8307f-490d-46cc-b49f-986e78b1e89b-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-bhcwm\" (UID: \"74d8307f-490d-46cc-b49f-986e78b1e89b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-bhcwm"
Sep 17 18:22:30.068636 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:30.068616    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74d8307f-490d-46cc-b49f-986e78b1e89b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-bhcwm\" (UID: \"74d8307f-490d-46cc-b49f-986e78b1e89b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-bhcwm"
Sep 17 18:22:30.070647 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:30.070624    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74d8307f-490d-46cc-b49f-986e78b1e89b-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-bhcwm\" (UID: \"74d8307f-490d-46cc-b49f-986e78b1e89b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-bhcwm"
Sep 17 18:22:30.083047 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:30.083028    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxrdw\" (UniqueName: \"kubernetes.io/projected/74d8307f-490d-46cc-b49f-986e78b1e89b-kube-api-access-zxrdw\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-bhcwm\" (UID: \"74d8307f-490d-46cc-b49f-986e78b1e89b\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-bhcwm"
Sep 17 18:22:30.111465 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:30.111443    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:30.177069 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:30.177046    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-bhcwm"
Sep 17 18:22:30.460466 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:30.460433    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-bhcwm"]
Sep 17 18:22:30.463455 ip-10-0-133-139 kubenswrapper[2585]: W0917 18:22:30.463428    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74d8307f_490d_46cc_b49f_986e78b1e89b.slice/crio-ffa9821c848f40929a4f4df0eea38a691121400903ac3d566f536494155b883f WatchSource:0}: Error finding container ffa9821c848f40929a4f4df0eea38a691121400903ac3d566f536494155b883f: Status 404 returned error can't find the container with id ffa9821c848f40929a4f4df0eea38a691121400903ac3d566f536494155b883f
Sep 17 18:22:30.655780 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:30.655743    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-bhcwm" event={"ID":"74d8307f-490d-46cc-b49f-986e78b1e89b","Type":"ContainerStarted","Data":"37d96de6c8d39fa09ab88654914b0f658aa13171c87ce2be99501545f17314f4"}
Sep 17 18:22:30.655969 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:30.655813    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-bhcwm" event={"ID":"74d8307f-490d-46cc-b49f-986e78b1e89b","Type":"ContainerStarted","Data":"ffa9821c848f40929a4f4df0eea38a691121400903ac3d566f536494155b883f"}
Sep 17 18:22:32.664564 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:32.664536    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-bhcwm_74d8307f-490d-46cc-b49f-986e78b1e89b/storage-initializer/0.log"
Sep 17 18:22:32.664971 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:32.664576    2585 generic.go:356] "Generic (PLEG): container finished" podID="74d8307f-490d-46cc-b49f-986e78b1e89b" containerID="37d96de6c8d39fa09ab88654914b0f658aa13171c87ce2be99501545f17314f4" exitCode=1
Sep 17 18:22:32.664971 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:32.664651    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-bhcwm" event={"ID":"74d8307f-490d-46cc-b49f-986e78b1e89b","Type":"ContainerDied","Data":"37d96de6c8d39fa09ab88654914b0f658aa13171c87ce2be99501545f17314f4"}
Sep 17 18:22:33.513152 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:33.513123    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z"
Sep 17 18:22:33.593614 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:33.593541    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f50ee9c-fa9c-4777-81bb-685444d46422-cabundle-cert\") pod \"9f50ee9c-fa9c-4777-81bb-685444d46422\" (UID: \"9f50ee9c-fa9c-4777-81bb-685444d46422\") "
Sep 17 18:22:33.593614 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:33.593576    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ggpwk\" (UniqueName: \"kubernetes.io/projected/9f50ee9c-fa9c-4777-81bb-685444d46422-kube-api-access-ggpwk\") pod \"9f50ee9c-fa9c-4777-81bb-685444d46422\" (UID: \"9f50ee9c-fa9c-4777-81bb-685444d46422\") "
Sep 17 18:22:33.593614 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:33.593598    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f50ee9c-fa9c-4777-81bb-685444d46422-proxy-tls\") pod \"9f50ee9c-fa9c-4777-81bb-685444d46422\" (UID: \"9f50ee9c-fa9c-4777-81bb-685444d46422\") "
Sep 17 18:22:33.593899 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:33.593633    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f50ee9c-fa9c-4777-81bb-685444d46422-kube-rbac-proxy-sar-config\") pod \"9f50ee9c-fa9c-4777-81bb-685444d46422\" (UID: \"9f50ee9c-fa9c-4777-81bb-685444d46422\") "
Sep 17 18:22:33.593899 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:33.593676    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f50ee9c-fa9c-4777-81bb-685444d46422-kserve-provision-location\") pod \"9f50ee9c-fa9c-4777-81bb-685444d46422\" (UID: \"9f50ee9c-fa9c-4777-81bb-685444d46422\") "
Sep 17 18:22:33.594010 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:33.593985    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f50ee9c-fa9c-4777-81bb-685444d46422-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9f50ee9c-fa9c-4777-81bb-685444d46422" (UID: "9f50ee9c-fa9c-4777-81bb-685444d46422"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:22:33.594061 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:33.594017    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f50ee9c-fa9c-4777-81bb-685444d46422-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9f50ee9c-fa9c-4777-81bb-685444d46422" (UID: "9f50ee9c-fa9c-4777-81bb-685444d46422"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:22:33.594061 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:33.594024    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f50ee9c-fa9c-4777-81bb-685444d46422-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9f50ee9c-fa9c-4777-81bb-685444d46422" (UID: "9f50ee9c-fa9c-4777-81bb-685444d46422"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:22:33.595707 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:33.595679    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f50ee9c-fa9c-4777-81bb-685444d46422-kube-api-access-ggpwk" (OuterVolumeSpecName: "kube-api-access-ggpwk") pod "9f50ee9c-fa9c-4777-81bb-685444d46422" (UID: "9f50ee9c-fa9c-4777-81bb-685444d46422"). InnerVolumeSpecName "kube-api-access-ggpwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:22:33.595707 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:33.595701    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f50ee9c-fa9c-4777-81bb-685444d46422-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9f50ee9c-fa9c-4777-81bb-685444d46422" (UID: "9f50ee9c-fa9c-4777-81bb-685444d46422"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:22:33.669876 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:33.669846    2585 generic.go:356] "Generic (PLEG): container finished" podID="9f50ee9c-fa9c-4777-81bb-685444d46422" containerID="06b6149245d3ce54aee36fc3c2d3b8d2e37eaa86de1a8a9e118484167f7e8e38" exitCode=0
Sep 17 18:22:33.670275 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:33.669901    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z"
Sep 17 18:22:33.670275 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:33.669932    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z" event={"ID":"9f50ee9c-fa9c-4777-81bb-685444d46422","Type":"ContainerDied","Data":"06b6149245d3ce54aee36fc3c2d3b8d2e37eaa86de1a8a9e118484167f7e8e38"}
Sep 17 18:22:33.670275 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:33.670011    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z" event={"ID":"9f50ee9c-fa9c-4777-81bb-685444d46422","Type":"ContainerDied","Data":"4af86305c559f20e3ab1747f8efdbdc024db9a85bb02d2e127de68dea2c3185a"}
Sep 17 18:22:33.670275 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:33.670046    2585 scope.go:117] "RemoveContainer" containerID="9c87438347b8de58f0c12bf7f65e729ffcff36c9fe4f4356d3e9a0ac5c48e7ab"
Sep 17 18:22:33.672229 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:33.672213    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-bhcwm_74d8307f-490d-46cc-b49f-986e78b1e89b/storage-initializer/0.log"
Sep 17 18:22:33.672336 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:33.672266    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-bhcwm" event={"ID":"74d8307f-490d-46cc-b49f-986e78b1e89b","Type":"ContainerStarted","Data":"1b4c15e339a91ec858673bc9483190bc74b754e52f8566b287d54294ae2a33c8"}
Sep 17 18:22:33.678070 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:33.677958    2585 scope.go:117] "RemoveContainer" containerID="06b6149245d3ce54aee36fc3c2d3b8d2e37eaa86de1a8a9e118484167f7e8e38"
Sep 17 18:22:33.690619 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:33.690601    2585 scope.go:117] "RemoveContainer" containerID="df594bc073987ede9e5e133bf877e74cc677e31e3b20edbfde0ef2c9ad723f52"
Sep 17 18:22:33.694620 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:33.694595    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggpwk\" (UniqueName: \"kubernetes.io/projected/9f50ee9c-fa9c-4777-81bb-685444d46422-kube-api-access-ggpwk\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:22:33.694745 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:33.694623    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f50ee9c-fa9c-4777-81bb-685444d46422-proxy-tls\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:22:33.694745 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:33.694638    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f50ee9c-fa9c-4777-81bb-685444d46422-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:22:33.694745 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:33.694652    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f50ee9c-fa9c-4777-81bb-685444d46422-kserve-provision-location\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:22:33.694745 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:33.694665    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f50ee9c-fa9c-4777-81bb-685444d46422-cabundle-cert\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:22:33.697271 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:33.697258    2585 scope.go:117] "RemoveContainer" containerID="9c87438347b8de58f0c12bf7f65e729ffcff36c9fe4f4356d3e9a0ac5c48e7ab"
Sep 17 18:22:33.697544 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:22:33.697526    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c87438347b8de58f0c12bf7f65e729ffcff36c9fe4f4356d3e9a0ac5c48e7ab\": container with ID starting with 9c87438347b8de58f0c12bf7f65e729ffcff36c9fe4f4356d3e9a0ac5c48e7ab not found: ID does not exist" containerID="9c87438347b8de58f0c12bf7f65e729ffcff36c9fe4f4356d3e9a0ac5c48e7ab"
Sep 17 18:22:33.697602 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:33.697550    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c87438347b8de58f0c12bf7f65e729ffcff36c9fe4f4356d3e9a0ac5c48e7ab"} err="failed to get container status \"9c87438347b8de58f0c12bf7f65e729ffcff36c9fe4f4356d3e9a0ac5c48e7ab\": rpc error: code = NotFound desc = could not find container \"9c87438347b8de58f0c12bf7f65e729ffcff36c9fe4f4356d3e9a0ac5c48e7ab\": container with ID starting with 9c87438347b8de58f0c12bf7f65e729ffcff36c9fe4f4356d3e9a0ac5c48e7ab not found: ID does not exist"
Sep 17 18:22:33.697602 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:33.697565    2585 scope.go:117] "RemoveContainer" containerID="06b6149245d3ce54aee36fc3c2d3b8d2e37eaa86de1a8a9e118484167f7e8e38"
Sep 17 18:22:33.697805 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:22:33.697786    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06b6149245d3ce54aee36fc3c2d3b8d2e37eaa86de1a8a9e118484167f7e8e38\": container with ID starting with 06b6149245d3ce54aee36fc3c2d3b8d2e37eaa86de1a8a9e118484167f7e8e38 not found: ID does not exist" containerID="06b6149245d3ce54aee36fc3c2d3b8d2e37eaa86de1a8a9e118484167f7e8e38"
Sep 17 18:22:33.697854 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:33.697811    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06b6149245d3ce54aee36fc3c2d3b8d2e37eaa86de1a8a9e118484167f7e8e38"} err="failed to get container status \"06b6149245d3ce54aee36fc3c2d3b8d2e37eaa86de1a8a9e118484167f7e8e38\": rpc error: code = NotFound desc = could not find container \"06b6149245d3ce54aee36fc3c2d3b8d2e37eaa86de1a8a9e118484167f7e8e38\": container with ID starting with 06b6149245d3ce54aee36fc3c2d3b8d2e37eaa86de1a8a9e118484167f7e8e38 not found: ID does not exist"
Sep 17 18:22:33.697854 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:33.697826    2585 scope.go:117] "RemoveContainer" containerID="df594bc073987ede9e5e133bf877e74cc677e31e3b20edbfde0ef2c9ad723f52"
Sep 17 18:22:33.698072 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:22:33.698050    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df594bc073987ede9e5e133bf877e74cc677e31e3b20edbfde0ef2c9ad723f52\": container with ID starting with df594bc073987ede9e5e133bf877e74cc677e31e3b20edbfde0ef2c9ad723f52 not found: ID does not exist" containerID="df594bc073987ede9e5e133bf877e74cc677e31e3b20edbfde0ef2c9ad723f52"
Sep 17 18:22:33.698153 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:33.698075    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df594bc073987ede9e5e133bf877e74cc677e31e3b20edbfde0ef2c9ad723f52"} err="failed to get container status \"df594bc073987ede9e5e133bf877e74cc677e31e3b20edbfde0ef2c9ad723f52\": rpc error: code = NotFound desc = could not find container \"df594bc073987ede9e5e133bf877e74cc677e31e3b20edbfde0ef2c9ad723f52\": container with ID starting with df594bc073987ede9e5e133bf877e74cc677e31e3b20edbfde0ef2c9ad723f52 not found: ID does not exist"
Sep 17 18:22:33.719067 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:33.719041    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z"]
Sep 17 18:22:33.729895 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:33.729871    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-7984f9dd45-xtc2z"]
Sep 17 18:22:34.103706 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:34.103675    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f50ee9c-fa9c-4777-81bb-685444d46422" path="/var/lib/kubelet/pods/9f50ee9c-fa9c-4777-81bb-685444d46422/volumes"
Sep 17 18:22:36.686605 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:36.686532    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-bhcwm_74d8307f-490d-46cc-b49f-986e78b1e89b/storage-initializer/1.log"
Sep 17 18:22:36.686998 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:36.686952    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-bhcwm_74d8307f-490d-46cc-b49f-986e78b1e89b/storage-initializer/0.log"
Sep 17 18:22:36.686998 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:36.686982    2585 generic.go:356] "Generic (PLEG): container finished" podID="74d8307f-490d-46cc-b49f-986e78b1e89b" containerID="1b4c15e339a91ec858673bc9483190bc74b754e52f8566b287d54294ae2a33c8" exitCode=1
Sep 17 18:22:36.687074 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:36.687053    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-bhcwm" event={"ID":"74d8307f-490d-46cc-b49f-986e78b1e89b","Type":"ContainerDied","Data":"1b4c15e339a91ec858673bc9483190bc74b754e52f8566b287d54294ae2a33c8"}
Sep 17 18:22:36.687126 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:36.687114    2585 scope.go:117] "RemoveContainer" containerID="37d96de6c8d39fa09ab88654914b0f658aa13171c87ce2be99501545f17314f4"
Sep 17 18:22:36.687437 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:36.687420    2585 scope.go:117] "RemoveContainer" containerID="37d96de6c8d39fa09ab88654914b0f658aa13171c87ce2be99501545f17314f4"
Sep 17 18:22:36.697069 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:22:36.697044    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-bhcwm_kserve-ci-e2e-test(74d8307f-490d-46cc-b49f-986e78b1e89b)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-bhcwm" podUID="74d8307f-490d-46cc-b49f-986e78b1e89b"
Sep 17 18:22:36.697157 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:22:36.697114    2585 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-bhcwm_kserve-ci-e2e-test_74d8307f-490d-46cc-b49f-986e78b1e89b_0 in pod sandbox ffa9821c848f40929a4f4df0eea38a691121400903ac3d566f536494155b883f from index: no such id: '37d96de6c8d39fa09ab88654914b0f658aa13171c87ce2be99501545f17314f4'" containerID="37d96de6c8d39fa09ab88654914b0f658aa13171c87ce2be99501545f17314f4"
Sep 17 18:22:36.697157 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:36.697139    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37d96de6c8d39fa09ab88654914b0f658aa13171c87ce2be99501545f17314f4"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-bhcwm_kserve-ci-e2e-test_74d8307f-490d-46cc-b49f-986e78b1e89b_0 in pod sandbox ffa9821c848f40929a4f4df0eea38a691121400903ac3d566f536494155b883f from index: no such id: '37d96de6c8d39fa09ab88654914b0f658aa13171c87ce2be99501545f17314f4'"
Sep 17 18:22:37.692390 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:37.692361    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-bhcwm_74d8307f-490d-46cc-b49f-986e78b1e89b/storage-initializer/1.log"
Sep 17 18:22:39.864101 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:39.864063    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-bhcwm"]
Sep 17 18:22:39.871983 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:39.871952    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:39.985855 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:39.985834    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-bhcwm_74d8307f-490d-46cc-b49f-986e78b1e89b/storage-initializer/1.log"
Sep 17 18:22:39.989223 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:39.989205    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-bhcwm"
Sep 17 18:22:40.038246 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:40.038221    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxrdw\" (UniqueName: \"kubernetes.io/projected/74d8307f-490d-46cc-b49f-986e78b1e89b-kube-api-access-zxrdw\") pod \"74d8307f-490d-46cc-b49f-986e78b1e89b\" (UID: \"74d8307f-490d-46cc-b49f-986e78b1e89b\") "
Sep 17 18:22:40.038362 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:40.038251    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74d8307f-490d-46cc-b49f-986e78b1e89b-kserve-provision-location\") pod \"74d8307f-490d-46cc-b49f-986e78b1e89b\" (UID: \"74d8307f-490d-46cc-b49f-986e78b1e89b\") "
Sep 17 18:22:40.038362 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:40.038278    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74d8307f-490d-46cc-b49f-986e78b1e89b-kube-rbac-proxy-sar-config\") pod \"74d8307f-490d-46cc-b49f-986e78b1e89b\" (UID: \"74d8307f-490d-46cc-b49f-986e78b1e89b\") "
Sep 17 18:22:40.038362 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:40.038307    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74d8307f-490d-46cc-b49f-986e78b1e89b-proxy-tls\") pod \"74d8307f-490d-46cc-b49f-986e78b1e89b\" (UID: \"74d8307f-490d-46cc-b49f-986e78b1e89b\") "
Sep 17 18:22:40.040731 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:40.038744    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74d8307f-490d-46cc-b49f-986e78b1e89b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "74d8307f-490d-46cc-b49f-986e78b1e89b" (UID: "74d8307f-490d-46cc-b49f-986e78b1e89b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:22:40.040731 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:40.038903    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74d8307f-490d-46cc-b49f-986e78b1e89b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "74d8307f-490d-46cc-b49f-986e78b1e89b" (UID: "74d8307f-490d-46cc-b49f-986e78b1e89b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:22:40.044138 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:40.044028    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74d8307f-490d-46cc-b49f-986e78b1e89b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "74d8307f-490d-46cc-b49f-986e78b1e89b" (UID: "74d8307f-490d-46cc-b49f-986e78b1e89b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:22:40.044138 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:40.044053    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74d8307f-490d-46cc-b49f-986e78b1e89b-kube-api-access-zxrdw" (OuterVolumeSpecName: "kube-api-access-zxrdw") pod "74d8307f-490d-46cc-b49f-986e78b1e89b" (UID: "74d8307f-490d-46cc-b49f-986e78b1e89b"). InnerVolumeSpecName "kube-api-access-zxrdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:22:40.139411 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:40.139360    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxrdw\" (UniqueName: \"kubernetes.io/projected/74d8307f-490d-46cc-b49f-986e78b1e89b-kube-api-access-zxrdw\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:22:40.139411 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:40.139381    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74d8307f-490d-46cc-b49f-986e78b1e89b-kserve-provision-location\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:22:40.139411 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:40.139392    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74d8307f-490d-46cc-b49f-986e78b1e89b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:22:40.139411 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:40.139401    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74d8307f-490d-46cc-b49f-986e78b1e89b-proxy-tls\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:22:40.704216 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:40.704188    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-bhcwm_74d8307f-490d-46cc-b49f-986e78b1e89b/storage-initializer/1.log"
Sep 17 18:22:40.704367 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:40.704276    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-bhcwm" event={"ID":"74d8307f-490d-46cc-b49f-986e78b1e89b","Type":"ContainerDied","Data":"ffa9821c848f40929a4f4df0eea38a691121400903ac3d566f536494155b883f"}
Sep 17 18:22:40.704367 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:40.704309    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-bhcwm"
Sep 17 18:22:40.704367 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:40.704338    2585 scope.go:117] "RemoveContainer" containerID="1b4c15e339a91ec858673bc9483190bc74b754e52f8566b287d54294ae2a33c8"
Sep 17 18:22:40.768288 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:40.768258    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-bhcwm"]
Sep 17 18:22:40.770794 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:40.770773    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5dc7bc7669-bhcwm"]
Sep 17 18:22:41.041824 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:41.041785    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:41.096329 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:41.096293    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2"]
Sep 17 18:22:41.096558 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:41.096544    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f50ee9c-fa9c-4777-81bb-685444d46422" containerName="kube-rbac-proxy"
Sep 17 18:22:41.096558 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:41.096558    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f50ee9c-fa9c-4777-81bb-685444d46422" containerName="kube-rbac-proxy"
Sep 17 18:22:41.096659 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:41.096566    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74d8307f-490d-46cc-b49f-986e78b1e89b" containerName="storage-initializer"
Sep 17 18:22:41.096659 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:41.096571    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="74d8307f-490d-46cc-b49f-986e78b1e89b" containerName="storage-initializer"
Sep 17 18:22:41.096659 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:41.096587    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f50ee9c-fa9c-4777-81bb-685444d46422" containerName="kserve-container"
Sep 17 18:22:41.096659 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:41.096608    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f50ee9c-fa9c-4777-81bb-685444d46422" containerName="kserve-container"
Sep 17 18:22:41.096659 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:41.096639    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f50ee9c-fa9c-4777-81bb-685444d46422" containerName="storage-initializer"
Sep 17 18:22:41.096659 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:41.096644    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f50ee9c-fa9c-4777-81bb-685444d46422" containerName="storage-initializer"
Sep 17 18:22:41.096857 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:41.096684    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="74d8307f-490d-46cc-b49f-986e78b1e89b" containerName="storage-initializer"
Sep 17 18:22:41.096857 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:41.096692    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="74d8307f-490d-46cc-b49f-986e78b1e89b" containerName="storage-initializer"
Sep 17 18:22:41.096857 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:41.096699    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f50ee9c-fa9c-4777-81bb-685444d46422" containerName="kserve-container"
Sep 17 18:22:41.096857 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:41.096704    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f50ee9c-fa9c-4777-81bb-685444d46422" containerName="kube-rbac-proxy"
Sep 17 18:22:41.096857 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:41.096755    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74d8307f-490d-46cc-b49f-986e78b1e89b" containerName="storage-initializer"
Sep 17 18:22:41.096857 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:41.096760    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="74d8307f-490d-46cc-b49f-986e78b1e89b" containerName="storage-initializer"
Sep 17 18:22:41.100841 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:41.100824    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2"
Sep 17 18:22:41.110831 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:41.110803    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:22:41.110831 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:41.110825    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
Sep 17 18:22:41.111021 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:41.110801    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:22:41.111117 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:41.111105    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lps5w\""
Sep 17 18:22:41.111310 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:41.111288    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
Sep 17 18:22:41.111383 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:41.111333    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 17 18:22:41.111383 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:41.111343    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:22:41.119993 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:41.119972    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2"]
Sep 17 18:22:41.146670 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:41.146630    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts7ps\" (UniqueName: \"kubernetes.io/projected/74231bb6-8130-445e-800d-4c0f5536b769-kube-api-access-ts7ps\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2\" (UID: \"74231bb6-8130-445e-800d-4c0f5536b769\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2"
Sep 17 18:22:41.146670 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:41.146673    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74231bb6-8130-445e-800d-4c0f5536b769-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2\" (UID: \"74231bb6-8130-445e-800d-4c0f5536b769\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2"
Sep 17 18:22:41.146857 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:41.146701    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74231bb6-8130-445e-800d-4c0f5536b769-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2\" (UID: \"74231bb6-8130-445e-800d-4c0f5536b769\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2"
Sep 17 18:22:41.146857 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:41.146787    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74231bb6-8130-445e-800d-4c0f5536b769-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2\" (UID: \"74231bb6-8130-445e-800d-4c0f5536b769\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2"
Sep 17 18:22:41.146857 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:41.146823    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74231bb6-8130-445e-800d-4c0f5536b769-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2\" (UID: \"74231bb6-8130-445e-800d-4c0f5536b769\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2"
Sep 17 18:22:41.195694 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:41.195659    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:41.248040 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:41.248014    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74231bb6-8130-445e-800d-4c0f5536b769-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2\" (UID: \"74231bb6-8130-445e-800d-4c0f5536b769\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2"
Sep 17 18:22:41.248173 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:41.248049    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74231bb6-8130-445e-800d-4c0f5536b769-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2\" (UID: \"74231bb6-8130-445e-800d-4c0f5536b769\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2"
Sep 17 18:22:41.248173 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:41.248066    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74231bb6-8130-445e-800d-4c0f5536b769-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2\" (UID: \"74231bb6-8130-445e-800d-4c0f5536b769\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2"
Sep 17 18:22:41.248285 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:41.248248    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts7ps\" (UniqueName: \"kubernetes.io/projected/74231bb6-8130-445e-800d-4c0f5536b769-kube-api-access-ts7ps\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2\" (UID: \"74231bb6-8130-445e-800d-4c0f5536b769\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2"
Sep 17 18:22:41.248319 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:41.248286    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74231bb6-8130-445e-800d-4c0f5536b769-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2\" (UID: \"74231bb6-8130-445e-800d-4c0f5536b769\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2"
Sep 17 18:22:41.248464 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:41.248425    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74231bb6-8130-445e-800d-4c0f5536b769-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2\" (UID: \"74231bb6-8130-445e-800d-4c0f5536b769\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2"
Sep 17 18:22:41.248786 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:41.248767    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74231bb6-8130-445e-800d-4c0f5536b769-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2\" (UID: \"74231bb6-8130-445e-800d-4c0f5536b769\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2"
Sep 17 18:22:41.248879 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:41.248867    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74231bb6-8130-445e-800d-4c0f5536b769-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2\" (UID: \"74231bb6-8130-445e-800d-4c0f5536b769\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2"
Sep 17 18:22:41.250575 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:41.250552    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74231bb6-8130-445e-800d-4c0f5536b769-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2\" (UID: \"74231bb6-8130-445e-800d-4c0f5536b769\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2"
Sep 17 18:22:41.260802 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:41.260781    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts7ps\" (UniqueName: \"kubernetes.io/projected/74231bb6-8130-445e-800d-4c0f5536b769-kube-api-access-ts7ps\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2\" (UID: \"74231bb6-8130-445e-800d-4c0f5536b769\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2"
Sep 17 18:22:41.409989 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:41.409915    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2"
Sep 17 18:22:41.690739 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:41.690682    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2"]
Sep 17 18:22:41.692470 ip-10-0-133-139 kubenswrapper[2585]: W0917 18:22:41.692436    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74231bb6_8130_445e_800d_4c0f5536b769.slice/crio-eebf83a140bdab908e8dd4867c972bfe792fb031e5ba827a87f35669a68cbb8c WatchSource:0}: Error finding container eebf83a140bdab908e8dd4867c972bfe792fb031e5ba827a87f35669a68cbb8c: Status 404 returned error can't find the container with id eebf83a140bdab908e8dd4867c972bfe792fb031e5ba827a87f35669a68cbb8c
Sep 17 18:22:41.710762 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:41.710735    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2" event={"ID":"74231bb6-8130-445e-800d-4c0f5536b769","Type":"ContainerStarted","Data":"eebf83a140bdab908e8dd4867c972bfe792fb031e5ba827a87f35669a68cbb8c"}
Sep 17 18:22:42.103850 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:42.103817    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74d8307f-490d-46cc-b49f-986e78b1e89b" path="/var/lib/kubelet/pods/74d8307f-490d-46cc-b49f-986e78b1e89b/volumes"
Sep 17 18:22:42.717233 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:42.717197    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2" event={"ID":"74231bb6-8130-445e-800d-4c0f5536b769","Type":"ContainerStarted","Data":"6124bbe4ecef325be21dcad416eab17c4163602d0ae7ae480ddb502b6a008da6"}
Sep 17 18:22:43.722867 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:43.722829    2585 generic.go:356] "Generic (PLEG): container finished" podID="74231bb6-8130-445e-800d-4c0f5536b769" containerID="6124bbe4ecef325be21dcad416eab17c4163602d0ae7ae480ddb502b6a008da6" exitCode=0
Sep 17 18:22:43.723388 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:43.722898    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2" event={"ID":"74231bb6-8130-445e-800d-4c0f5536b769","Type":"ContainerDied","Data":"6124bbe4ecef325be21dcad416eab17c4163602d0ae7ae480ddb502b6a008da6"}
Sep 17 18:22:44.728931 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:44.728895    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2" event={"ID":"74231bb6-8130-445e-800d-4c0f5536b769","Type":"ContainerStarted","Data":"eafc99c0e0971cc84248c14c0c71f40b0841c1ec254b8efd4b9cb759bb0bc48d"}
Sep 17 18:22:44.729380 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:44.728969    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2" event={"ID":"74231bb6-8130-445e-800d-4c0f5536b769","Type":"ContainerStarted","Data":"8bc6a491b843777be0d6c533a30fc63c33e603e3c5b120e1c3a3c0918dbe1033"}
Sep 17 18:22:44.729380 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:44.729004    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2"
Sep 17 18:22:44.729380 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:44.729033    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2"
Sep 17 18:22:44.730426 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:44.730399    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2" podUID="74231bb6-8130-445e-800d-4c0f5536b769" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 17 18:22:45.732970 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:45.732924    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2" podUID="74231bb6-8130-445e-800d-4c0f5536b769" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 17 18:22:50.737736 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:50.737708    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2"
Sep 17 18:22:50.738186 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:50.738160    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2" podUID="74231bb6-8130-445e-800d-4c0f5536b769" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 17 18:22:50.771779 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:22:50.771729    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2" podStartSLOduration=8.686984784 podStartE2EDuration="9.7717111s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.084726316s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:22:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:22:44.758775568 +0000 UTC m=+2901.172566006" watchObservedRunningTime="2026-09-17 18:22:50.7717111 +0000 UTC m=+2907.185501541"
Sep 17 18:23:00.738185 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:23:00.738148    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2" podUID="74231bb6-8130-445e-800d-4c0f5536b769" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 17 18:23:10.738239 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:23:10.738200    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2" podUID="74231bb6-8130-445e-800d-4c0f5536b769" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 17 18:23:20.738889 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:23:20.738850    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2" podUID="74231bb6-8130-445e-800d-4c0f5536b769" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 17 18:23:30.738245 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:23:30.738201    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2" podUID="74231bb6-8130-445e-800d-4c0f5536b769" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 17 18:23:39.977702 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:23:39.977658    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:40.738398 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:23:40.738356    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2" podUID="74231bb6-8130-445e-800d-4c0f5536b769" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 17 18:23:45.059222 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:23:45.059185    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:45.267693 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:23:45.267662    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:50.738378 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:23:50.738304    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2" podUID="74231bb6-8130-445e-800d-4c0f5536b769" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 17 18:24:00.739678 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:00.739650    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2"
Sep 17 18:24:01.034365 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:01.034336    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2"]
Sep 17 18:24:01.034578 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:01.034553    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2" podUID="74231bb6-8130-445e-800d-4c0f5536b769" containerName="kserve-container" containerID="cri-o://8bc6a491b843777be0d6c533a30fc63c33e603e3c5b120e1c3a3c0918dbe1033" gracePeriod=30
Sep 17 18:24:01.034672 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:01.034611    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2" podUID="74231bb6-8130-445e-800d-4c0f5536b769" containerName="kube-rbac-proxy" containerID="cri-o://eafc99c0e0971cc84248c14c0c71f40b0841c1ec254b8efd4b9cb759bb0bc48d" gracePeriod=30
Sep 17 18:24:01.057742 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:01.057712    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:02.031095 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:02.031057    2585 generic.go:356] "Generic (PLEG): container finished" podID="74231bb6-8130-445e-800d-4c0f5536b769" containerID="eafc99c0e0971cc84248c14c0c71f40b0841c1ec254b8efd4b9cb759bb0bc48d" exitCode=2
Sep 17 18:24:02.031528 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:02.031129    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2" event={"ID":"74231bb6-8130-445e-800d-4c0f5536b769","Type":"ContainerDied","Data":"eafc99c0e0971cc84248c14c0c71f40b0841c1ec254b8efd4b9cb759bb0bc48d"}
Sep 17 18:24:02.135685 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:02.135656    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-9h9pp"]
Sep 17 18:24:02.138940 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:02.138925    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-9h9pp"
Sep 17 18:24:02.141247 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:02.141225    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
Sep 17 18:24:02.141552 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:02.141533    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
Sep 17 18:24:02.149775 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:02.149752    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-9h9pp"]
Sep 17 18:24:02.163711 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:02.163687    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:02.327734 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:02.327644    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqwdw\" (UniqueName: \"kubernetes.io/projected/c448ef64-8a47-42a4-8d46-91cfd86248ab-kube-api-access-lqwdw\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-9h9pp\" (UID: \"c448ef64-8a47-42a4-8d46-91cfd86248ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-9h9pp"
Sep 17 18:24:02.327734 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:02.327691    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c448ef64-8a47-42a4-8d46-91cfd86248ab-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-9h9pp\" (UID: \"c448ef64-8a47-42a4-8d46-91cfd86248ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-9h9pp"
Sep 17 18:24:02.327930 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:02.327751    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c448ef64-8a47-42a4-8d46-91cfd86248ab-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-9h9pp\" (UID: \"c448ef64-8a47-42a4-8d46-91cfd86248ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-9h9pp"
Sep 17 18:24:02.327930 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:02.327795    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c448ef64-8a47-42a4-8d46-91cfd86248ab-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-9h9pp\" (UID: \"c448ef64-8a47-42a4-8d46-91cfd86248ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-9h9pp"
Sep 17 18:24:02.428547 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:02.428511    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqwdw\" (UniqueName: \"kubernetes.io/projected/c448ef64-8a47-42a4-8d46-91cfd86248ab-kube-api-access-lqwdw\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-9h9pp\" (UID: \"c448ef64-8a47-42a4-8d46-91cfd86248ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-9h9pp"
Sep 17 18:24:02.428722 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:02.428553    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c448ef64-8a47-42a4-8d46-91cfd86248ab-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-9h9pp\" (UID: \"c448ef64-8a47-42a4-8d46-91cfd86248ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-9h9pp"
Sep 17 18:24:02.428722 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:02.428576    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c448ef64-8a47-42a4-8d46-91cfd86248ab-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-9h9pp\" (UID: \"c448ef64-8a47-42a4-8d46-91cfd86248ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-9h9pp"
Sep 17 18:24:02.428722 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:02.428606    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c448ef64-8a47-42a4-8d46-91cfd86248ab-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-9h9pp\" (UID: \"c448ef64-8a47-42a4-8d46-91cfd86248ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-9h9pp"
Sep 17 18:24:02.428722 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:24:02.428697    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert" not found
Sep 17 18:24:02.428980 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:24:02.428780    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c448ef64-8a47-42a4-8d46-91cfd86248ab-proxy-tls podName:c448ef64-8a47-42a4-8d46-91cfd86248ab nodeName:}" failed. No retries permitted until 2026-09-17 18:24:02.92874396 +0000 UTC m=+2979.342534378 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c448ef64-8a47-42a4-8d46-91cfd86248ab-proxy-tls") pod "isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-9h9pp" (UID: "c448ef64-8a47-42a4-8d46-91cfd86248ab") : secret "isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert" not found
Sep 17 18:24:02.428980 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:02.428941    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c448ef64-8a47-42a4-8d46-91cfd86248ab-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-9h9pp\" (UID: \"c448ef64-8a47-42a4-8d46-91cfd86248ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-9h9pp"
Sep 17 18:24:02.429221 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:02.429200    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c448ef64-8a47-42a4-8d46-91cfd86248ab-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-9h9pp\" (UID: \"c448ef64-8a47-42a4-8d46-91cfd86248ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-9h9pp"
Sep 17 18:24:02.440275 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:02.440247    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqwdw\" (UniqueName: \"kubernetes.io/projected/c448ef64-8a47-42a4-8d46-91cfd86248ab-kube-api-access-lqwdw\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-9h9pp\" (UID: \"c448ef64-8a47-42a4-8d46-91cfd86248ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-9h9pp"
Sep 17 18:24:02.578864 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:02.578789    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:02.931685 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:02.931593    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c448ef64-8a47-42a4-8d46-91cfd86248ab-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-9h9pp\" (UID: \"c448ef64-8a47-42a4-8d46-91cfd86248ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-9h9pp"
Sep 17 18:24:02.934035 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:02.934007    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c448ef64-8a47-42a4-8d46-91cfd86248ab-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-9h9pp\" (UID: \"c448ef64-8a47-42a4-8d46-91cfd86248ab\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-9h9pp"
Sep 17 18:24:03.049544 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:03.049505    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-9h9pp"
Sep 17 18:24:03.318974 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:03.318949    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-9h9pp"]
Sep 17 18:24:03.321018 ip-10-0-133-139 kubenswrapper[2585]: W0917 18:24:03.320994    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc448ef64_8a47_42a4_8d46_91cfd86248ab.slice/crio-f794fbe8ecda54312851aa85ccc05d89b9e3e6cbcb89438baee67059934ebedf WatchSource:0}: Error finding container f794fbe8ecda54312851aa85ccc05d89b9e3e6cbcb89438baee67059934ebedf: Status 404 returned error can't find the container with id f794fbe8ecda54312851aa85ccc05d89b9e3e6cbcb89438baee67059934ebedf
Sep 17 18:24:04.039911 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:04.039874    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-9h9pp" event={"ID":"c448ef64-8a47-42a4-8d46-91cfd86248ab","Type":"ContainerStarted","Data":"a08f040af5b06169bfc88c48048ad7f65a268e4136bd166c8945bfbbae88dd59"}
Sep 17 18:24:04.040109 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:04.039950    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-9h9pp" event={"ID":"c448ef64-8a47-42a4-8d46-91cfd86248ab","Type":"ContainerStarted","Data":"f794fbe8ecda54312851aa85ccc05d89b9e3e6cbcb89438baee67059934ebedf"}
Sep 17 18:24:05.777986 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:05.777963    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2"
Sep 17 18:24:05.952666 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:05.952587    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74231bb6-8130-445e-800d-4c0f5536b769-cabundle-cert\") pod \"74231bb6-8130-445e-800d-4c0f5536b769\" (UID: \"74231bb6-8130-445e-800d-4c0f5536b769\") "
Sep 17 18:24:05.952666 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:05.952639    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74231bb6-8130-445e-800d-4c0f5536b769-kube-rbac-proxy-sar-config\") pod \"74231bb6-8130-445e-800d-4c0f5536b769\" (UID: \"74231bb6-8130-445e-800d-4c0f5536b769\") "
Sep 17 18:24:05.952865 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:05.952667    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74231bb6-8130-445e-800d-4c0f5536b769-proxy-tls\") pod \"74231bb6-8130-445e-800d-4c0f5536b769\" (UID: \"74231bb6-8130-445e-800d-4c0f5536b769\") "
Sep 17 18:24:05.952865 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:05.952764    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74231bb6-8130-445e-800d-4c0f5536b769-kserve-provision-location\") pod \"74231bb6-8130-445e-800d-4c0f5536b769\" (UID: \"74231bb6-8130-445e-800d-4c0f5536b769\") "
Sep 17 18:24:05.952865 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:05.952794    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ts7ps\" (UniqueName: \"kubernetes.io/projected/74231bb6-8130-445e-800d-4c0f5536b769-kube-api-access-ts7ps\") pod \"74231bb6-8130-445e-800d-4c0f5536b769\" (UID: \"74231bb6-8130-445e-800d-4c0f5536b769\") "
Sep 17 18:24:05.953051 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:05.953029    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74231bb6-8130-445e-800d-4c0f5536b769-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "74231bb6-8130-445e-800d-4c0f5536b769" (UID: "74231bb6-8130-445e-800d-4c0f5536b769"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:24:05.953123 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:05.953067    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74231bb6-8130-445e-800d-4c0f5536b769-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "74231bb6-8130-445e-800d-4c0f5536b769" (UID: "74231bb6-8130-445e-800d-4c0f5536b769"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:24:05.953123 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:05.953102    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74231bb6-8130-445e-800d-4c0f5536b769-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "74231bb6-8130-445e-800d-4c0f5536b769" (UID: "74231bb6-8130-445e-800d-4c0f5536b769"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:24:05.954861 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:05.954832    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74231bb6-8130-445e-800d-4c0f5536b769-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "74231bb6-8130-445e-800d-4c0f5536b769" (UID: "74231bb6-8130-445e-800d-4c0f5536b769"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:24:05.954974 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:05.954918    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74231bb6-8130-445e-800d-4c0f5536b769-kube-api-access-ts7ps" (OuterVolumeSpecName: "kube-api-access-ts7ps") pod "74231bb6-8130-445e-800d-4c0f5536b769" (UID: "74231bb6-8130-445e-800d-4c0f5536b769"). InnerVolumeSpecName "kube-api-access-ts7ps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:24:06.049293 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:06.049260    2585 generic.go:356] "Generic (PLEG): container finished" podID="74231bb6-8130-445e-800d-4c0f5536b769" containerID="8bc6a491b843777be0d6c533a30fc63c33e603e3c5b120e1c3a3c0918dbe1033" exitCode=0
Sep 17 18:24:06.049474 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:06.049320    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2" event={"ID":"74231bb6-8130-445e-800d-4c0f5536b769","Type":"ContainerDied","Data":"8bc6a491b843777be0d6c533a30fc63c33e603e3c5b120e1c3a3c0918dbe1033"}
Sep 17 18:24:06.049474 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:06.049323    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2"
Sep 17 18:24:06.049474 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:06.049367    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2" event={"ID":"74231bb6-8130-445e-800d-4c0f5536b769","Type":"ContainerDied","Data":"eebf83a140bdab908e8dd4867c972bfe792fb031e5ba827a87f35669a68cbb8c"}
Sep 17 18:24:06.049474 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:06.049393    2585 scope.go:117] "RemoveContainer" containerID="eafc99c0e0971cc84248c14c0c71f40b0841c1ec254b8efd4b9cb759bb0bc48d"
Sep 17 18:24:06.053365 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:06.053346    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74231bb6-8130-445e-800d-4c0f5536b769-kserve-provision-location\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:24:06.053365 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:06.053366    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts7ps\" (UniqueName: \"kubernetes.io/projected/74231bb6-8130-445e-800d-4c0f5536b769-kube-api-access-ts7ps\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:24:06.053548 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:06.053375    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74231bb6-8130-445e-800d-4c0f5536b769-cabundle-cert\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:24:06.053548 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:06.053384    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74231bb6-8130-445e-800d-4c0f5536b769-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:24:06.053548 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:06.053392    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74231bb6-8130-445e-800d-4c0f5536b769-proxy-tls\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:24:06.058388 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:06.058236    2585 scope.go:117] "RemoveContainer" containerID="8bc6a491b843777be0d6c533a30fc63c33e603e3c5b120e1c3a3c0918dbe1033"
Sep 17 18:24:06.065533 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:06.065519    2585 scope.go:117] "RemoveContainer" containerID="6124bbe4ecef325be21dcad416eab17c4163602d0ae7ae480ddb502b6a008da6"
Sep 17 18:24:06.072884 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:06.072864    2585 scope.go:117] "RemoveContainer" containerID="eafc99c0e0971cc84248c14c0c71f40b0841c1ec254b8efd4b9cb759bb0bc48d"
Sep 17 18:24:06.073195 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:24:06.073165    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eafc99c0e0971cc84248c14c0c71f40b0841c1ec254b8efd4b9cb759bb0bc48d\": container with ID starting with eafc99c0e0971cc84248c14c0c71f40b0841c1ec254b8efd4b9cb759bb0bc48d not found: ID does not exist" containerID="eafc99c0e0971cc84248c14c0c71f40b0841c1ec254b8efd4b9cb759bb0bc48d"
Sep 17 18:24:06.073287 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:06.073205    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eafc99c0e0971cc84248c14c0c71f40b0841c1ec254b8efd4b9cb759bb0bc48d"} err="failed to get container status \"eafc99c0e0971cc84248c14c0c71f40b0841c1ec254b8efd4b9cb759bb0bc48d\": rpc error: code = NotFound desc = could not find container \"eafc99c0e0971cc84248c14c0c71f40b0841c1ec254b8efd4b9cb759bb0bc48d\": container with ID starting with eafc99c0e0971cc84248c14c0c71f40b0841c1ec254b8efd4b9cb759bb0bc48d not found: ID does not exist"
Sep 17 18:24:06.073287 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:06.073227    2585 scope.go:117] "RemoveContainer" containerID="8bc6a491b843777be0d6c533a30fc63c33e603e3c5b120e1c3a3c0918dbe1033"
Sep 17 18:24:06.073383 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:06.073320    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2"]
Sep 17 18:24:06.073470 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:24:06.073444    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bc6a491b843777be0d6c533a30fc63c33e603e3c5b120e1c3a3c0918dbe1033\": container with ID starting with 8bc6a491b843777be0d6c533a30fc63c33e603e3c5b120e1c3a3c0918dbe1033 not found: ID does not exist" containerID="8bc6a491b843777be0d6c533a30fc63c33e603e3c5b120e1c3a3c0918dbe1033"
Sep 17 18:24:06.073548 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:06.073474    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bc6a491b843777be0d6c533a30fc63c33e603e3c5b120e1c3a3c0918dbe1033"} err="failed to get container status \"8bc6a491b843777be0d6c533a30fc63c33e603e3c5b120e1c3a3c0918dbe1033\": rpc error: code = NotFound desc = could not find container \"8bc6a491b843777be0d6c533a30fc63c33e603e3c5b120e1c3a3c0918dbe1033\": container with ID starting with 8bc6a491b843777be0d6c533a30fc63c33e603e3c5b120e1c3a3c0918dbe1033 not found: ID does not exist"
Sep 17 18:24:06.073548 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:06.073501    2585 scope.go:117] "RemoveContainer" containerID="6124bbe4ecef325be21dcad416eab17c4163602d0ae7ae480ddb502b6a008da6"
Sep 17 18:24:06.073744 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:24:06.073729    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6124bbe4ecef325be21dcad416eab17c4163602d0ae7ae480ddb502b6a008da6\": container with ID starting with 6124bbe4ecef325be21dcad416eab17c4163602d0ae7ae480ddb502b6a008da6 not found: ID does not exist" containerID="6124bbe4ecef325be21dcad416eab17c4163602d0ae7ae480ddb502b6a008da6"
Sep 17 18:24:06.073799 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:06.073746    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6124bbe4ecef325be21dcad416eab17c4163602d0ae7ae480ddb502b6a008da6"} err="failed to get container status \"6124bbe4ecef325be21dcad416eab17c4163602d0ae7ae480ddb502b6a008da6\": rpc error: code = NotFound desc = could not find container \"6124bbe4ecef325be21dcad416eab17c4163602d0ae7ae480ddb502b6a008da6\": container with ID starting with 6124bbe4ecef325be21dcad416eab17c4163602d0ae7ae480ddb502b6a008da6 not found: ID does not exist"
Sep 17 18:24:06.083337 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:06.079217    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2"]
Sep 17 18:24:06.103789 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:06.103763    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74231bb6-8130-445e-800d-4c0f5536b769" path="/var/lib/kubelet/pods/74231bb6-8130-445e-800d-4c0f5536b769/volumes"
Sep 17 18:24:06.734364 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:06.734325    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-55578787f7-j5nb2" podUID="74231bb6-8130-445e-800d-4c0f5536b769" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.47:8643/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 17 18:24:07.054199 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:07.054174    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-9h9pp_c448ef64-8a47-42a4-8d46-91cfd86248ab/storage-initializer/0.log"
Sep 17 18:24:07.054620 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:07.054213    2585 generic.go:356] "Generic (PLEG): container finished" podID="c448ef64-8a47-42a4-8d46-91cfd86248ab" containerID="a08f040af5b06169bfc88c48048ad7f65a268e4136bd166c8945bfbbae88dd59" exitCode=1
Sep 17 18:24:07.054620 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:07.054299    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-9h9pp" event={"ID":"c448ef64-8a47-42a4-8d46-91cfd86248ab","Type":"ContainerDied","Data":"a08f040af5b06169bfc88c48048ad7f65a268e4136bd166c8945bfbbae88dd59"}
Sep 17 18:24:08.061039 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:08.061006    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-9h9pp_c448ef64-8a47-42a4-8d46-91cfd86248ab/storage-initializer/0.log"
Sep 17 18:24:08.061442 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:08.061076    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-9h9pp" event={"ID":"c448ef64-8a47-42a4-8d46-91cfd86248ab","Type":"ContainerStarted","Data":"e433a711400fdd01c2478739ef0ae2d83f8150a6d794d680aa78fb3fc8d947bf"}
Sep 17 18:24:12.079603 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:12.079575    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-9h9pp_c448ef64-8a47-42a4-8d46-91cfd86248ab/storage-initializer/1.log"
Sep 17 18:24:12.080027 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:12.079917    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-9h9pp_c448ef64-8a47-42a4-8d46-91cfd86248ab/storage-initializer/0.log"
Sep 17 18:24:12.080027 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:12.079951    2585 generic.go:356] "Generic (PLEG): container finished" podID="c448ef64-8a47-42a4-8d46-91cfd86248ab" containerID="e433a711400fdd01c2478739ef0ae2d83f8150a6d794d680aa78fb3fc8d947bf" exitCode=1
Sep 17 18:24:12.080027 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:12.080001    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-9h9pp" event={"ID":"c448ef64-8a47-42a4-8d46-91cfd86248ab","Type":"ContainerDied","Data":"e433a711400fdd01c2478739ef0ae2d83f8150a6d794d680aa78fb3fc8d947bf"}
Sep 17 18:24:12.080132 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:12.080062    2585 scope.go:117] "RemoveContainer" containerID="a08f040af5b06169bfc88c48048ad7f65a268e4136bd166c8945bfbbae88dd59"
Sep 17 18:24:12.080414 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:12.080396    2585 scope.go:117] "RemoveContainer" containerID="a08f040af5b06169bfc88c48048ad7f65a268e4136bd166c8945bfbbae88dd59"
Sep 17 18:24:12.092632 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:24:12.092608    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-9h9pp_kserve-ci-e2e-test(c448ef64-8a47-42a4-8d46-91cfd86248ab)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-9h9pp" podUID="c448ef64-8a47-42a4-8d46-91cfd86248ab"
Sep 17 18:24:12.092907 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:24:12.092872    2585 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-9h9pp_kserve-ci-e2e-test_c448ef64-8a47-42a4-8d46-91cfd86248ab_0 in pod sandbox f794fbe8ecda54312851aa85ccc05d89b9e3e6cbcb89438baee67059934ebedf from index: no such id: 'a08f040af5b06169bfc88c48048ad7f65a268e4136bd166c8945bfbbae88dd59'" containerID="a08f040af5b06169bfc88c48048ad7f65a268e4136bd166c8945bfbbae88dd59"
Sep 17 18:24:12.092988 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:12.092912    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a08f040af5b06169bfc88c48048ad7f65a268e4136bd166c8945bfbbae88dd59"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-9h9pp_kserve-ci-e2e-test_c448ef64-8a47-42a4-8d46-91cfd86248ab_0 in pod sandbox f794fbe8ecda54312851aa85ccc05d89b9e3e6cbcb89438baee67059934ebedf from index: no such id: 'a08f040af5b06169bfc88c48048ad7f65a268e4136bd166c8945bfbbae88dd59'"
Sep 17 18:24:12.132759 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:12.132731    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-9h9pp"]
Sep 17 18:24:12.138963 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:12.138931    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:13.085849 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:13.085825    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-9h9pp_c448ef64-8a47-42a4-8d46-91cfd86248ab/storage-initializer/1.log"
Sep 17 18:24:13.221248 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:13.221226    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-9h9pp_c448ef64-8a47-42a4-8d46-91cfd86248ab/storage-initializer/1.log"
Sep 17 18:24:13.221466 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:13.221441    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:13.224718 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:13.224701    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-9h9pp"
Sep 17 18:24:13.251168 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:13.251144    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh"]
Sep 17 18:24:13.251461 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:13.251446    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74231bb6-8130-445e-800d-4c0f5536b769" containerName="storage-initializer"
Sep 17 18:24:13.251575 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:13.251464    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="74231bb6-8130-445e-800d-4c0f5536b769" containerName="storage-initializer"
Sep 17 18:24:13.251575 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:13.251481    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74231bb6-8130-445e-800d-4c0f5536b769" containerName="kserve-container"
Sep 17 18:24:13.251575 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:13.251507    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="74231bb6-8130-445e-800d-4c0f5536b769" containerName="kserve-container"
Sep 17 18:24:13.251575 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:13.251518    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c448ef64-8a47-42a4-8d46-91cfd86248ab" containerName="storage-initializer"
Sep 17 18:24:13.251575 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:13.251527    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c448ef64-8a47-42a4-8d46-91cfd86248ab" containerName="storage-initializer"
Sep 17 18:24:13.251575 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:13.251546    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74231bb6-8130-445e-800d-4c0f5536b769" containerName="kube-rbac-proxy"
Sep 17 18:24:13.251575 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:13.251554    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="74231bb6-8130-445e-800d-4c0f5536b769" containerName="kube-rbac-proxy"
Sep 17 18:24:13.251575 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:13.251563    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c448ef64-8a47-42a4-8d46-91cfd86248ab" containerName="storage-initializer"
Sep 17 18:24:13.251575 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:13.251571    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c448ef64-8a47-42a4-8d46-91cfd86248ab" containerName="storage-initializer"
Sep 17 18:24:13.252031 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:13.251644    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="74231bb6-8130-445e-800d-4c0f5536b769" containerName="kserve-container"
Sep 17 18:24:13.252031 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:13.251657    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="74231bb6-8130-445e-800d-4c0f5536b769" containerName="kube-rbac-proxy"
Sep 17 18:24:13.252031 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:13.251668    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c448ef64-8a47-42a4-8d46-91cfd86248ab" containerName="storage-initializer"
Sep 17 18:24:13.252031 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:13.251678    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c448ef64-8a47-42a4-8d46-91cfd86248ab" containerName="storage-initializer"
Sep 17 18:24:13.254624 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:13.254607    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh"
Sep 17 18:24:13.259886 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:13.259836    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
Sep 17 18:24:13.260152 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:13.260133    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:24:13.260346 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:13.260327    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
Sep 17 18:24:13.265195 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:13.265175    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh"]
Sep 17 18:24:13.305275 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:13.305251    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqwdw\" (UniqueName: \"kubernetes.io/projected/c448ef64-8a47-42a4-8d46-91cfd86248ab-kube-api-access-lqwdw\") pod \"c448ef64-8a47-42a4-8d46-91cfd86248ab\" (UID: \"c448ef64-8a47-42a4-8d46-91cfd86248ab\") "
Sep 17 18:24:13.305401 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:13.305279    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c448ef64-8a47-42a4-8d46-91cfd86248ab-kserve-provision-location\") pod \"c448ef64-8a47-42a4-8d46-91cfd86248ab\" (UID: \"c448ef64-8a47-42a4-8d46-91cfd86248ab\") "
Sep 17 18:24:13.305401 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:13.305310    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c448ef64-8a47-42a4-8d46-91cfd86248ab-kube-rbac-proxy-sar-config\") pod \"c448ef64-8a47-42a4-8d46-91cfd86248ab\" (UID: \"c448ef64-8a47-42a4-8d46-91cfd86248ab\") "
Sep 17 18:24:13.305401 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:13.305365    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c448ef64-8a47-42a4-8d46-91cfd86248ab-proxy-tls\") pod \"c448ef64-8a47-42a4-8d46-91cfd86248ab\" (UID: \"c448ef64-8a47-42a4-8d46-91cfd86248ab\") "
Sep 17 18:24:13.305593 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:13.305458    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mcbm\" (UniqueName: \"kubernetes.io/projected/1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd-kube-api-access-6mcbm\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh\" (UID: \"1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh"
Sep 17 18:24:13.305593 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:13.305509    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh\" (UID: \"1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh"
Sep 17 18:24:13.305593 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:13.305554    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh\" (UID: \"1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh"
Sep 17 18:24:13.305761 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:13.305607    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh\" (UID: \"1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh"
Sep 17 18:24:13.305761 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:13.305632    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c448ef64-8a47-42a4-8d46-91cfd86248ab-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c448ef64-8a47-42a4-8d46-91cfd86248ab" (UID: "c448ef64-8a47-42a4-8d46-91cfd86248ab"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:24:13.305761 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:13.305741    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh\" (UID: \"1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh"
Sep 17 18:24:13.305924 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:13.305759    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c448ef64-8a47-42a4-8d46-91cfd86248ab-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c448ef64-8a47-42a4-8d46-91cfd86248ab" (UID: "c448ef64-8a47-42a4-8d46-91cfd86248ab"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:24:13.305924 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:13.305794    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c448ef64-8a47-42a4-8d46-91cfd86248ab-kserve-provision-location\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:24:13.307477 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:13.307454    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c448ef64-8a47-42a4-8d46-91cfd86248ab-kube-api-access-lqwdw" (OuterVolumeSpecName: "kube-api-access-lqwdw") pod "c448ef64-8a47-42a4-8d46-91cfd86248ab" (UID: "c448ef64-8a47-42a4-8d46-91cfd86248ab"). InnerVolumeSpecName "kube-api-access-lqwdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:24:13.307599 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:13.307575    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c448ef64-8a47-42a4-8d46-91cfd86248ab-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c448ef64-8a47-42a4-8d46-91cfd86248ab" (UID: "c448ef64-8a47-42a4-8d46-91cfd86248ab"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:24:13.407057 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:13.406965    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh\" (UID: \"1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh"
Sep 17 18:24:13.407057 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:13.407021    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mcbm\" (UniqueName: \"kubernetes.io/projected/1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd-kube-api-access-6mcbm\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh\" (UID: \"1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh"
Sep 17 18:24:13.407057 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:13.407049    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh\" (UID: \"1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh"
Sep 17 18:24:13.407325 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:13.407086    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh\" (UID: \"1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh"
Sep 17 18:24:13.407325 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:13.407129    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh\" (UID: \"1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh"
Sep 17 18:24:13.407325 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:24:13.407130    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
Sep 17 18:24:13.407325 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:13.407167    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqwdw\" (UniqueName: \"kubernetes.io/projected/c448ef64-8a47-42a4-8d46-91cfd86248ab-kube-api-access-lqwdw\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:24:13.407325 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:13.407183    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c448ef64-8a47-42a4-8d46-91cfd86248ab-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:24:13.407325 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:13.407197    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c448ef64-8a47-42a4-8d46-91cfd86248ab-proxy-tls\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:24:13.407325 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:24:13.407217    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd-proxy-tls podName:1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd nodeName:}" failed. No retries permitted until 2026-09-17 18:24:13.907197088 +0000 UTC m=+2990.320987510 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd-proxy-tls") pod "isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh" (UID: "1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd") : secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
Sep 17 18:24:13.407616 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:13.407592    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh\" (UID: \"1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh"
Sep 17 18:24:13.407831 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:13.407816    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh\" (UID: \"1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh"
Sep 17 18:24:13.407867 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:13.407833    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh\" (UID: \"1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh"
Sep 17 18:24:13.415931 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:13.415911    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mcbm\" (UniqueName: \"kubernetes.io/projected/1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd-kube-api-access-6mcbm\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh\" (UID: \"1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh"
Sep 17 18:24:13.882152 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:13.882121    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:13.912087 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:13.912055    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh\" (UID: \"1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh"
Sep 17 18:24:13.914421 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:13.914405    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh\" (UID: \"1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh"
Sep 17 18:24:14.091174 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:14.091146    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-9h9pp_c448ef64-8a47-42a4-8d46-91cfd86248ab/storage-initializer/1.log"
Sep 17 18:24:14.091647 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:14.091200    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-9h9pp" event={"ID":"c448ef64-8a47-42a4-8d46-91cfd86248ab","Type":"ContainerDied","Data":"f794fbe8ecda54312851aa85ccc05d89b9e3e6cbcb89438baee67059934ebedf"}
Sep 17 18:24:14.091647 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:14.091244    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-9h9pp"
Sep 17 18:24:14.091647 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:14.091253    2585 scope.go:117] "RemoveContainer" containerID="e433a711400fdd01c2478739ef0ae2d83f8150a6d794d680aa78fb3fc8d947bf"
Sep 17 18:24:14.141376 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:14.141299    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-9h9pp"]
Sep 17 18:24:14.142507 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:14.142472    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-568dd8598f-9h9pp"]
Sep 17 18:24:14.169888 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:14.169864    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh"
Sep 17 18:24:14.446714 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:14.446632    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh"]
Sep 17 18:24:14.450581 ip-10-0-133-139 kubenswrapper[2585]: W0917 18:24:14.450548    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d76cc41_b9fe_4374_8ffb_c3f2dc5d50bd.slice/crio-a351f96aa74c5dd9ff9aefb44b877ed5dc10ecac9e814d74844a1822825fb6cd WatchSource:0}: Error finding container a351f96aa74c5dd9ff9aefb44b877ed5dc10ecac9e814d74844a1822825fb6cd: Status 404 returned error can't find the container with id a351f96aa74c5dd9ff9aefb44b877ed5dc10ecac9e814d74844a1822825fb6cd
Sep 17 18:24:15.096246 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:15.096207    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh" event={"ID":"1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd","Type":"ContainerStarted","Data":"ebdbd1523289c704c85dcfa8ca16a2072197302b5ba27460d1fa40a2db430c65"}
Sep 17 18:24:15.096695 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:15.096274    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh" event={"ID":"1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd","Type":"ContainerStarted","Data":"a351f96aa74c5dd9ff9aefb44b877ed5dc10ecac9e814d74844a1822825fb6cd"}
Sep 17 18:24:16.103555 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:16.103514    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c448ef64-8a47-42a4-8d46-91cfd86248ab" path="/var/lib/kubelet/pods/c448ef64-8a47-42a4-8d46-91cfd86248ab/volumes"
Sep 17 18:24:16.103979 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:16.103916    2585 generic.go:356] "Generic (PLEG): container finished" podID="1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd" containerID="ebdbd1523289c704c85dcfa8ca16a2072197302b5ba27460d1fa40a2db430c65" exitCode=0
Sep 17 18:24:16.103979 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:16.103961    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh" event={"ID":"1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd","Type":"ContainerDied","Data":"ebdbd1523289c704c85dcfa8ca16a2072197302b5ba27460d1fa40a2db430c65"}
Sep 17 18:24:17.113419 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:17.113383    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh" event={"ID":"1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd","Type":"ContainerStarted","Data":"168fe215a26bb8fa1b980e2d9804fb129256476e5c6c7e735efdd94925f90da1"}
Sep 17 18:24:17.113823 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:17.113452    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh" event={"ID":"1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd","Type":"ContainerStarted","Data":"fc307a1351156a1dcc0848195f04b7686bed01200fca229501b692648c943867"}
Sep 17 18:24:17.113823 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:17.113534    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh"
Sep 17 18:24:17.139625 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:17.139583    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh" podStartSLOduration=3.064644936 podStartE2EDuration="4.139567757s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.074922821s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:24:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:24:17.138082826 +0000 UTC m=+2993.551873256" watchObservedRunningTime="2026-09-17 18:24:17.139567757 +0000 UTC m=+2993.553358196"
Sep 17 18:24:18.117363 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:18.117325    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh"
Sep 17 18:24:18.118661 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:18.118633    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh" podUID="1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 17 18:24:19.121357 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:19.121314    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh" podUID="1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 17 18:24:24.125605 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:24.125576    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh"
Sep 17 18:24:24.126038 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:24.126011    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh" podUID="1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 17 18:24:28.518642 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:28.518612    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rrf79_beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5/ovn-acl-logging/0.log"
Sep 17 18:24:28.525754 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:28.525733    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rrf79_beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5/ovn-acl-logging/0.log"
Sep 17 18:24:30.148270 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:30.148239    2585 scope.go:117] "RemoveContainer" containerID="f7b5b4bdfc3f97dd00ef04aa2216e729ad5ea0aba56f563ba7571f30978ea6fb"
Sep 17 18:24:30.155777 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:30.155761    2585 scope.go:117] "RemoveContainer" containerID="e146d36fee9ba7b718cb99e6857827c46ff0c6ea39cd8cdd0d1e0b551486c760"
Sep 17 18:24:30.163391 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:30.163366    2585 scope.go:117] "RemoveContainer" containerID="f1a961fd18b45322a98b0068c19609d56ac4d00be236ad5e7b34bdf72f5deefe"
Sep 17 18:24:34.126026 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:34.125989    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh" podUID="1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 17 18:24:44.126968 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:44.126930    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh" podUID="1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 17 18:24:46.674338 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:46.674303    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:51.819677 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:51.819639    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:52.245160 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:52.245132    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:54.126636 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:24:54.126596    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh" podUID="1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 17 18:25:04.126731 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:04.126686    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh" podUID="1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 17 18:25:14.126456 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:14.126415    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh" podUID="1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 17 18:25:24.126344 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:24.126253    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh" podUID="1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 17 18:25:28.099876 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:28.099823    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh" podUID="1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.49:8080: connect: connection refused"
Sep 17 18:25:38.104113 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:38.104087    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh"
Sep 17 18:25:43.257824 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:43.257791    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh"]
Sep 17 18:25:43.258297 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:43.258114    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh" podUID="1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd" containerName="kserve-container" containerID="cri-o://fc307a1351156a1dcc0848195f04b7686bed01200fca229501b692648c943867" gracePeriod=30
Sep 17 18:25:43.258297 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:43.258148    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh" podUID="1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd" containerName="kube-rbac-proxy" containerID="cri-o://168fe215a26bb8fa1b980e2d9804fb129256476e5c6c7e735efdd94925f90da1" gracePeriod=30
Sep 17 18:25:43.279013 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:43.278992    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:25:43.419363 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:43.419334    2585 generic.go:356] "Generic (PLEG): container finished" podID="1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd" containerID="168fe215a26bb8fa1b980e2d9804fb129256476e5c6c7e735efdd94925f90da1" exitCode=2
Sep 17 18:25:43.419509 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:43.419406    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh" event={"ID":"1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd","Type":"ContainerDied","Data":"168fe215a26bb8fa1b980e2d9804fb129256476e5c6c7e735efdd94925f90da1"}
Sep 17 18:25:44.122224 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:44.122185    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh" podUID="1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.49:8643/healthz\": dial tcp 10.133.0.49:8643: connect: connection refused"
Sep 17 18:25:44.334269 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:44.334236    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:25:44.361192 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:44.361165    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-9jmws"]
Sep 17 18:25:44.364358 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:44.364337    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-9jmws"
Sep 17 18:25:44.366948 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:44.366929    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
Sep 17 18:25:44.367062 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:44.366958    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
Sep 17 18:25:44.371088 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:44.371066    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbff9\" (UniqueName: \"kubernetes.io/projected/fb30e986-fbdd-4d10-b5c1-808580a4c75d-kube-api-access-dbff9\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-9jmws\" (UID: \"fb30e986-fbdd-4d10-b5c1-808580a4c75d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-9jmws"
Sep 17 18:25:44.371187 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:44.371163    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb30e986-fbdd-4d10-b5c1-808580a4c75d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-9jmws\" (UID: \"fb30e986-fbdd-4d10-b5c1-808580a4c75d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-9jmws"
Sep 17 18:25:44.371252 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:44.371230    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb30e986-fbdd-4d10-b5c1-808580a4c75d-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-9jmws\" (UID: \"fb30e986-fbdd-4d10-b5c1-808580a4c75d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-9jmws"
Sep 17 18:25:44.371317 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:44.371289    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb30e986-fbdd-4d10-b5c1-808580a4c75d-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-9jmws\" (UID: \"fb30e986-fbdd-4d10-b5c1-808580a4c75d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-9jmws"
Sep 17 18:25:44.374365 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:44.374309    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-9jmws"]
Sep 17 18:25:44.471573 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:44.471543    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb30e986-fbdd-4d10-b5c1-808580a4c75d-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-9jmws\" (UID: \"fb30e986-fbdd-4d10-b5c1-808580a4c75d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-9jmws"
Sep 17 18:25:44.471739 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:44.471586    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbff9\" (UniqueName: \"kubernetes.io/projected/fb30e986-fbdd-4d10-b5c1-808580a4c75d-kube-api-access-dbff9\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-9jmws\" (UID: \"fb30e986-fbdd-4d10-b5c1-808580a4c75d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-9jmws"
Sep 17 18:25:44.471739 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:25:44.471683    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Sep 17 18:25:44.471739 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:44.471698    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb30e986-fbdd-4d10-b5c1-808580a4c75d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-9jmws\" (UID: \"fb30e986-fbdd-4d10-b5c1-808580a4c75d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-9jmws"
Sep 17 18:25:44.471887 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:25:44.471743    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb30e986-fbdd-4d10-b5c1-808580a4c75d-proxy-tls podName:fb30e986-fbdd-4d10-b5c1-808580a4c75d nodeName:}" failed. No retries permitted until 2026-09-17 18:25:44.971725484 +0000 UTC m=+3081.385515904 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fb30e986-fbdd-4d10-b5c1-808580a4c75d-proxy-tls") pod "isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-9jmws" (UID: "fb30e986-fbdd-4d10-b5c1-808580a4c75d") : secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Sep 17 18:25:44.471887 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:44.471779    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb30e986-fbdd-4d10-b5c1-808580a4c75d-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-9jmws\" (UID: \"fb30e986-fbdd-4d10-b5c1-808580a4c75d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-9jmws"
Sep 17 18:25:44.472124 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:44.472108    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb30e986-fbdd-4d10-b5c1-808580a4c75d-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-9jmws\" (UID: \"fb30e986-fbdd-4d10-b5c1-808580a4c75d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-9jmws"
Sep 17 18:25:44.472295 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:44.472278    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb30e986-fbdd-4d10-b5c1-808580a4c75d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-9jmws\" (UID: \"fb30e986-fbdd-4d10-b5c1-808580a4c75d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-9jmws"
Sep 17 18:25:44.481301 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:44.481274    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbff9\" (UniqueName: \"kubernetes.io/projected/fb30e986-fbdd-4d10-b5c1-808580a4c75d-kube-api-access-dbff9\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-9jmws\" (UID: \"fb30e986-fbdd-4d10-b5c1-808580a4c75d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-9jmws"
Sep 17 18:25:44.694931 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:44.694859    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:25:44.974444 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:44.974352    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb30e986-fbdd-4d10-b5c1-808580a4c75d-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-9jmws\" (UID: \"fb30e986-fbdd-4d10-b5c1-808580a4c75d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-9jmws"
Sep 17 18:25:44.976746 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:44.976726    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb30e986-fbdd-4d10-b5c1-808580a4c75d-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-9jmws\" (UID: \"fb30e986-fbdd-4d10-b5c1-808580a4c75d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-9jmws"
Sep 17 18:25:45.276110 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:45.276074    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-9jmws"
Sep 17 18:25:45.550440 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:45.550279    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-9jmws"]
Sep 17 18:25:45.553196 ip-10-0-133-139 kubenswrapper[2585]: W0917 18:25:45.553167    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb30e986_fbdd_4d10_b5c1_808580a4c75d.slice/crio-51e3947ccc76834217fed3c36ae8be9f3223381de634790138b4ab974bfa9715 WatchSource:0}: Error finding container 51e3947ccc76834217fed3c36ae8be9f3223381de634790138b4ab974bfa9715: Status 404 returned error can't find the container with id 51e3947ccc76834217fed3c36ae8be9f3223381de634790138b4ab974bfa9715
Sep 17 18:25:46.432266 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:46.432236    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-9jmws" event={"ID":"fb30e986-fbdd-4d10-b5c1-808580a4c75d","Type":"ContainerStarted","Data":"cdf933183b05f9135a0c8da06a13f92727e4232d900d7a0bef999a5a50dbd25b"}
Sep 17 18:25:46.432437 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:46.432289    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-9jmws" event={"ID":"fb30e986-fbdd-4d10-b5c1-808580a4c75d","Type":"ContainerStarted","Data":"51e3947ccc76834217fed3c36ae8be9f3223381de634790138b4ab974bfa9715"}
Sep 17 18:25:48.000603 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:48.000572    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh"
Sep 17 18:25:48.093933 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:48.093852    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd-kube-rbac-proxy-sar-config\") pod \"1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd\" (UID: \"1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd\") "
Sep 17 18:25:48.093933 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:48.093891    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd-kserve-provision-location\") pod \"1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd\" (UID: \"1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd\") "
Sep 17 18:25:48.093933 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:48.093915    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd-proxy-tls\") pod \"1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd\" (UID: \"1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd\") "
Sep 17 18:25:48.094223 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:48.093952    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6mcbm\" (UniqueName: \"kubernetes.io/projected/1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd-kube-api-access-6mcbm\") pod \"1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd\" (UID: \"1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd\") "
Sep 17 18:25:48.094223 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:48.093974    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd-cabundle-cert\") pod \"1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd\" (UID: \"1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd\") "
Sep 17 18:25:48.094223 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:48.094201    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd" (UID: "1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:25:48.094363 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:48.094240    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd" (UID: "1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:25:48.094418 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:48.094400    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd" (UID: "1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:25:48.096048 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:48.096024    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd" (UID: "1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:25:48.096140 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:48.096067    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd-kube-api-access-6mcbm" (OuterVolumeSpecName: "kube-api-access-6mcbm") pod "1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd" (UID: "1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd"). InnerVolumeSpecName "kube-api-access-6mcbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:25:48.194791 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:48.194766    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mcbm\" (UniqueName: \"kubernetes.io/projected/1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd-kube-api-access-6mcbm\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:25:48.194791 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:48.194787    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd-cabundle-cert\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:25:48.194791 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:48.194796    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:25:48.195013 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:48.194805    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd-kserve-provision-location\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:25:48.195013 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:48.194814    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd-proxy-tls\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:25:48.441697 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:48.441611    2585 generic.go:356] "Generic (PLEG): container finished" podID="1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd" containerID="fc307a1351156a1dcc0848195f04b7686bed01200fca229501b692648c943867" exitCode=0
Sep 17 18:25:48.441697 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:48.441678    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh"
Sep 17 18:25:48.441877 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:48.441680    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh" event={"ID":"1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd","Type":"ContainerDied","Data":"fc307a1351156a1dcc0848195f04b7686bed01200fca229501b692648c943867"}
Sep 17 18:25:48.441877 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:48.441803    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh" event={"ID":"1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd","Type":"ContainerDied","Data":"a351f96aa74c5dd9ff9aefb44b877ed5dc10ecac9e814d74844a1822825fb6cd"}
Sep 17 18:25:48.441877 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:48.441842    2585 scope.go:117] "RemoveContainer" containerID="168fe215a26bb8fa1b980e2d9804fb129256476e5c6c7e735efdd94925f90da1"
Sep 17 18:25:48.451982 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:48.451963    2585 scope.go:117] "RemoveContainer" containerID="fc307a1351156a1dcc0848195f04b7686bed01200fca229501b692648c943867"
Sep 17 18:25:48.459110 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:48.459042    2585 scope.go:117] "RemoveContainer" containerID="ebdbd1523289c704c85dcfa8ca16a2072197302b5ba27460d1fa40a2db430c65"
Sep 17 18:25:48.460822 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:48.460801    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh"]
Sep 17 18:25:48.466688 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:48.466668    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9b978f694-g4fkh"]
Sep 17 18:25:48.467229 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:48.467209    2585 scope.go:117] "RemoveContainer" containerID="168fe215a26bb8fa1b980e2d9804fb129256476e5c6c7e735efdd94925f90da1"
Sep 17 18:25:48.467519 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:25:48.467486    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"168fe215a26bb8fa1b980e2d9804fb129256476e5c6c7e735efdd94925f90da1\": container with ID starting with 168fe215a26bb8fa1b980e2d9804fb129256476e5c6c7e735efdd94925f90da1 not found: ID does not exist" containerID="168fe215a26bb8fa1b980e2d9804fb129256476e5c6c7e735efdd94925f90da1"
Sep 17 18:25:48.467565 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:48.467527    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"168fe215a26bb8fa1b980e2d9804fb129256476e5c6c7e735efdd94925f90da1"} err="failed to get container status \"168fe215a26bb8fa1b980e2d9804fb129256476e5c6c7e735efdd94925f90da1\": rpc error: code = NotFound desc = could not find container \"168fe215a26bb8fa1b980e2d9804fb129256476e5c6c7e735efdd94925f90da1\": container with ID starting with 168fe215a26bb8fa1b980e2d9804fb129256476e5c6c7e735efdd94925f90da1 not found: ID does not exist"
Sep 17 18:25:48.467565 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:48.467545    2585 scope.go:117] "RemoveContainer" containerID="fc307a1351156a1dcc0848195f04b7686bed01200fca229501b692648c943867"
Sep 17 18:25:48.467797 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:25:48.467779    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc307a1351156a1dcc0848195f04b7686bed01200fca229501b692648c943867\": container with ID starting with fc307a1351156a1dcc0848195f04b7686bed01200fca229501b692648c943867 not found: ID does not exist" containerID="fc307a1351156a1dcc0848195f04b7686bed01200fca229501b692648c943867"
Sep 17 18:25:48.467840 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:48.467803    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc307a1351156a1dcc0848195f04b7686bed01200fca229501b692648c943867"} err="failed to get container status \"fc307a1351156a1dcc0848195f04b7686bed01200fca229501b692648c943867\": rpc error: code = NotFound desc = could not find container \"fc307a1351156a1dcc0848195f04b7686bed01200fca229501b692648c943867\": container with ID starting with fc307a1351156a1dcc0848195f04b7686bed01200fca229501b692648c943867 not found: ID does not exist"
Sep 17 18:25:48.467840 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:48.467819    2585 scope.go:117] "RemoveContainer" containerID="ebdbd1523289c704c85dcfa8ca16a2072197302b5ba27460d1fa40a2db430c65"
Sep 17 18:25:48.468042 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:25:48.468024    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebdbd1523289c704c85dcfa8ca16a2072197302b5ba27460d1fa40a2db430c65\": container with ID starting with ebdbd1523289c704c85dcfa8ca16a2072197302b5ba27460d1fa40a2db430c65 not found: ID does not exist" containerID="ebdbd1523289c704c85dcfa8ca16a2072197302b5ba27460d1fa40a2db430c65"
Sep 17 18:25:48.468084 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:48.468047    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebdbd1523289c704c85dcfa8ca16a2072197302b5ba27460d1fa40a2db430c65"} err="failed to get container status \"ebdbd1523289c704c85dcfa8ca16a2072197302b5ba27460d1fa40a2db430c65\": rpc error: code = NotFound desc = could not find container \"ebdbd1523289c704c85dcfa8ca16a2072197302b5ba27460d1fa40a2db430c65\": container with ID starting with ebdbd1523289c704c85dcfa8ca16a2072197302b5ba27460d1fa40a2db430c65 not found: ID does not exist"
Sep 17 18:25:50.103638 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:50.103611    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd" path="/var/lib/kubelet/pods/1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd/volumes"
Sep 17 18:25:50.451244 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:50.451175    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-9jmws_fb30e986-fbdd-4d10-b5c1-808580a4c75d/storage-initializer/0.log"
Sep 17 18:25:50.451244 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:50.451211    2585 generic.go:356] "Generic (PLEG): container finished" podID="fb30e986-fbdd-4d10-b5c1-808580a4c75d" containerID="cdf933183b05f9135a0c8da06a13f92727e4232d900d7a0bef999a5a50dbd25b" exitCode=1
Sep 17 18:25:50.451420 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:50.451260    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-9jmws" event={"ID":"fb30e986-fbdd-4d10-b5c1-808580a4c75d","Type":"ContainerDied","Data":"cdf933183b05f9135a0c8da06a13f92727e4232d900d7a0bef999a5a50dbd25b"}
Sep 17 18:25:51.456147 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:51.456120    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-9jmws_fb30e986-fbdd-4d10-b5c1-808580a4c75d/storage-initializer/0.log"
Sep 17 18:25:51.456544 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:51.456194    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-9jmws" event={"ID":"fb30e986-fbdd-4d10-b5c1-808580a4c75d","Type":"ContainerStarted","Data":"65b9c4948b04946c1ce58a29dde3ef6fe99b1b4fc57e12d99d9d7058b3b4f466"}
Sep 17 18:25:54.371760 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:54.371731    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-9jmws"]
Sep 17 18:25:54.372170 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:54.371921    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-9jmws" podUID="fb30e986-fbdd-4d10-b5c1-808580a4c75d" containerName="storage-initializer" containerID="cri-o://65b9c4948b04946c1ce58a29dde3ef6fe99b1b4fc57e12d99d9d7058b3b4f466" gracePeriod=30
Sep 17 18:25:54.394042 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:54.394007    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:25:55.500262 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:55.500228    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:25:55.537875 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:55.537840    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr"]
Sep 17 18:25:55.538161 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:55.538145    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd" containerName="storage-initializer"
Sep 17 18:25:55.538242 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:55.538164    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd" containerName="storage-initializer"
Sep 17 18:25:55.538242 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:55.538179    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd" containerName="kube-rbac-proxy"
Sep 17 18:25:55.538242 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:55.538188    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd" containerName="kube-rbac-proxy"
Sep 17 18:25:55.538242 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:55.538209    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd" containerName="kserve-container"
Sep 17 18:25:55.538242 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:55.538218    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd" containerName="kserve-container"
Sep 17 18:25:55.538534 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:55.538282    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd" containerName="kserve-container"
Sep 17 18:25:55.538534 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:55.538295    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d76cc41-b9fe-4374-8ffb-c3f2dc5d50bd" containerName="kube-rbac-proxy"
Sep 17 18:25:55.540941 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:55.540921    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr"
Sep 17 18:25:55.543232 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:55.543202    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 17 18:25:55.543347 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:55.543312    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:25:55.543442 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:55.543423    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:25:55.543686 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:55.543663    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\""
Sep 17 18:25:55.550458 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:55.550438    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr"]
Sep 17 18:25:55.645790 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:55.645760    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr\" (UID: \"d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr"
Sep 17 18:25:55.645928 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:55.645794    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lplck\" (UniqueName: \"kubernetes.io/projected/d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c-kube-api-access-lplck\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr\" (UID: \"d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr"
Sep 17 18:25:55.645928 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:55.645870    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr\" (UID: \"d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr"
Sep 17 18:25:55.645928 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:55.645900    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr\" (UID: \"d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr"
Sep 17 18:25:55.646039 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:55.645939    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr\" (UID: \"d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr"
Sep 17 18:25:55.683739 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:55.683705    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:25:55.747057 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:55.747030    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr\" (UID: \"d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr"
Sep 17 18:25:55.747185 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:55.747065    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr\" (UID: \"d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr"
Sep 17 18:25:55.747185 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:55.747096    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lplck\" (UniqueName: \"kubernetes.io/projected/d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c-kube-api-access-lplck\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr\" (UID: \"d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr"
Sep 17 18:25:55.747185 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:55.747148    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr\" (UID: \"d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr"
Sep 17 18:25:55.747185 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:55.747174    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr\" (UID: \"d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr"
Sep 17 18:25:55.747540 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:55.747518    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr\" (UID: \"d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr"
Sep 17 18:25:55.747935 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:55.747909    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr\" (UID: \"d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr"
Sep 17 18:25:55.748045 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:55.747909    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr\" (UID: \"d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr"
Sep 17 18:25:55.749543 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:55.749527    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr\" (UID: \"d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr"
Sep 17 18:25:55.755255 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:55.755210    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lplck\" (UniqueName: \"kubernetes.io/projected/d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c-kube-api-access-lplck\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr\" (UID: \"d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr"
Sep 17 18:25:55.850854 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:55.850819    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr"
Sep 17 18:25:56.121625 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:56.121582    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr"]
Sep 17 18:25:56.123836 ip-10-0-133-139 kubenswrapper[2585]: W0917 18:25:56.123806    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0ae2cc6_2b4b_402c_bfb6_a55a2d71d23c.slice/crio-8308168f0acb0aeee9f730000c2949833a6a8a365ea576759ac4a241b411367f WatchSource:0}: Error finding container 8308168f0acb0aeee9f730000c2949833a6a8a365ea576759ac4a241b411367f: Status 404 returned error can't find the container with id 8308168f0acb0aeee9f730000c2949833a6a8a365ea576759ac4a241b411367f
Sep 17 18:25:56.202119 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:56.202091    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-9jmws_fb30e986-fbdd-4d10-b5c1-808580a4c75d/storage-initializer/1.log"
Sep 17 18:25:56.202553 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:56.202481    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-9jmws_fb30e986-fbdd-4d10-b5c1-808580a4c75d/storage-initializer/0.log"
Sep 17 18:25:56.204777 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:56.204763    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-9jmws"
Sep 17 18:25:56.250090 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:56.250046    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb30e986-fbdd-4d10-b5c1-808580a4c75d-proxy-tls\") pod \"fb30e986-fbdd-4d10-b5c1-808580a4c75d\" (UID: \"fb30e986-fbdd-4d10-b5c1-808580a4c75d\") "
Sep 17 18:25:56.250090 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:56.250103    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbff9\" (UniqueName: \"kubernetes.io/projected/fb30e986-fbdd-4d10-b5c1-808580a4c75d-kube-api-access-dbff9\") pod \"fb30e986-fbdd-4d10-b5c1-808580a4c75d\" (UID: \"fb30e986-fbdd-4d10-b5c1-808580a4c75d\") "
Sep 17 18:25:56.250308 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:56.250145    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb30e986-fbdd-4d10-b5c1-808580a4c75d-kserve-provision-location\") pod \"fb30e986-fbdd-4d10-b5c1-808580a4c75d\" (UID: \"fb30e986-fbdd-4d10-b5c1-808580a4c75d\") "
Sep 17 18:25:56.250308 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:56.250175    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb30e986-fbdd-4d10-b5c1-808580a4c75d-kube-rbac-proxy-sar-config\") pod \"fb30e986-fbdd-4d10-b5c1-808580a4c75d\" (UID: \"fb30e986-fbdd-4d10-b5c1-808580a4c75d\") "
Sep 17 18:25:56.250468 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:56.250442    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb30e986-fbdd-4d10-b5c1-808580a4c75d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fb30e986-fbdd-4d10-b5c1-808580a4c75d" (UID: "fb30e986-fbdd-4d10-b5c1-808580a4c75d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:25:56.250626 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:56.250603    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb30e986-fbdd-4d10-b5c1-808580a4c75d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fb30e986-fbdd-4d10-b5c1-808580a4c75d" (UID: "fb30e986-fbdd-4d10-b5c1-808580a4c75d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:25:56.252125 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:56.252106    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb30e986-fbdd-4d10-b5c1-808580a4c75d-kube-api-access-dbff9" (OuterVolumeSpecName: "kube-api-access-dbff9") pod "fb30e986-fbdd-4d10-b5c1-808580a4c75d" (UID: "fb30e986-fbdd-4d10-b5c1-808580a4c75d"). InnerVolumeSpecName "kube-api-access-dbff9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:25:56.252194 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:56.252131    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb30e986-fbdd-4d10-b5c1-808580a4c75d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fb30e986-fbdd-4d10-b5c1-808580a4c75d" (UID: "fb30e986-fbdd-4d10-b5c1-808580a4c75d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:25:56.350660 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:56.350594    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb30e986-fbdd-4d10-b5c1-808580a4c75d-kserve-provision-location\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:25:56.350660 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:56.350619    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb30e986-fbdd-4d10-b5c1-808580a4c75d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:25:56.350660 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:56.350631    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb30e986-fbdd-4d10-b5c1-808580a4c75d-proxy-tls\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:25:56.350660 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:56.350641    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbff9\" (UniqueName: \"kubernetes.io/projected/fb30e986-fbdd-4d10-b5c1-808580a4c75d-kube-api-access-dbff9\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:25:56.476578 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:56.476549    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-9jmws_fb30e986-fbdd-4d10-b5c1-808580a4c75d/storage-initializer/1.log"
Sep 17 18:25:56.476978 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:56.476963    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-9jmws_fb30e986-fbdd-4d10-b5c1-808580a4c75d/storage-initializer/0.log"
Sep 17 18:25:56.477047 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:56.476997    2585 generic.go:356] "Generic (PLEG): container finished" podID="fb30e986-fbdd-4d10-b5c1-808580a4c75d" containerID="65b9c4948b04946c1ce58a29dde3ef6fe99b1b4fc57e12d99d9d7058b3b4f466" exitCode=1
Sep 17 18:25:56.477096 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:56.477076    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-9jmws" event={"ID":"fb30e986-fbdd-4d10-b5c1-808580a4c75d","Type":"ContainerDied","Data":"65b9c4948b04946c1ce58a29dde3ef6fe99b1b4fc57e12d99d9d7058b3b4f466"}
Sep 17 18:25:56.477096 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:56.477081    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-9jmws"
Sep 17 18:25:56.477178 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:56.477130    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-9jmws" event={"ID":"fb30e986-fbdd-4d10-b5c1-808580a4c75d","Type":"ContainerDied","Data":"51e3947ccc76834217fed3c36ae8be9f3223381de634790138b4ab974bfa9715"}
Sep 17 18:25:56.477178 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:56.477158    2585 scope.go:117] "RemoveContainer" containerID="65b9c4948b04946c1ce58a29dde3ef6fe99b1b4fc57e12d99d9d7058b3b4f466"
Sep 17 18:25:56.478810 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:56.478783    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr" event={"ID":"d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c","Type":"ContainerStarted","Data":"6dd13d29bd284106e5ad0e3a547a750792be75a1e1377ffb21727f122c838d0e"}
Sep 17 18:25:56.478920 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:56.478860    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr" event={"ID":"d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c","Type":"ContainerStarted","Data":"8308168f0acb0aeee9f730000c2949833a6a8a365ea576759ac4a241b411367f"}
Sep 17 18:25:56.485453 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:56.485435    2585 scope.go:117] "RemoveContainer" containerID="cdf933183b05f9135a0c8da06a13f92727e4232d900d7a0bef999a5a50dbd25b"
Sep 17 18:25:56.493836 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:56.493820    2585 scope.go:117] "RemoveContainer" containerID="65b9c4948b04946c1ce58a29dde3ef6fe99b1b4fc57e12d99d9d7058b3b4f466"
Sep 17 18:25:56.494098 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:25:56.494082    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65b9c4948b04946c1ce58a29dde3ef6fe99b1b4fc57e12d99d9d7058b3b4f466\": container with ID starting with 65b9c4948b04946c1ce58a29dde3ef6fe99b1b4fc57e12d99d9d7058b3b4f466 not found: ID does not exist" containerID="65b9c4948b04946c1ce58a29dde3ef6fe99b1b4fc57e12d99d9d7058b3b4f466"
Sep 17 18:25:56.494159 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:56.494104    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65b9c4948b04946c1ce58a29dde3ef6fe99b1b4fc57e12d99d9d7058b3b4f466"} err="failed to get container status \"65b9c4948b04946c1ce58a29dde3ef6fe99b1b4fc57e12d99d9d7058b3b4f466\": rpc error: code = NotFound desc = could not find container \"65b9c4948b04946c1ce58a29dde3ef6fe99b1b4fc57e12d99d9d7058b3b4f466\": container with ID starting with 65b9c4948b04946c1ce58a29dde3ef6fe99b1b4fc57e12d99d9d7058b3b4f466 not found: ID does not exist"
Sep 17 18:25:56.494159 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:56.494119    2585 scope.go:117] "RemoveContainer" containerID="cdf933183b05f9135a0c8da06a13f92727e4232d900d7a0bef999a5a50dbd25b"
Sep 17 18:25:56.494337 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:25:56.494321    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdf933183b05f9135a0c8da06a13f92727e4232d900d7a0bef999a5a50dbd25b\": container with ID starting with cdf933183b05f9135a0c8da06a13f92727e4232d900d7a0bef999a5a50dbd25b not found: ID does not exist" containerID="cdf933183b05f9135a0c8da06a13f92727e4232d900d7a0bef999a5a50dbd25b"
Sep 17 18:25:56.494377 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:56.494341    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdf933183b05f9135a0c8da06a13f92727e4232d900d7a0bef999a5a50dbd25b"} err="failed to get container status \"cdf933183b05f9135a0c8da06a13f92727e4232d900d7a0bef999a5a50dbd25b\": rpc error: code = NotFound desc = could not find container \"cdf933183b05f9135a0c8da06a13f92727e4232d900d7a0bef999a5a50dbd25b\": container with ID starting with cdf933183b05f9135a0c8da06a13f92727e4232d900d7a0bef999a5a50dbd25b not found: ID does not exist"
Sep 17 18:25:56.524501 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:56.524458    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-9jmws"]
Sep 17 18:25:56.526894 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:56.526869    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-797b7ddc6d-9jmws"]
Sep 17 18:25:58.103680 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:25:58.103652    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb30e986-fbdd-4d10-b5c1-808580a4c75d" path="/var/lib/kubelet/pods/fb30e986-fbdd-4d10-b5c1-808580a4c75d/volumes"
Sep 17 18:26:00.495502 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:26:00.495466    2585 generic.go:356] "Generic (PLEG): container finished" podID="d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c" containerID="6dd13d29bd284106e5ad0e3a547a750792be75a1e1377ffb21727f122c838d0e" exitCode=0
Sep 17 18:26:00.495871 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:26:00.495538    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr" event={"ID":"d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c","Type":"ContainerDied","Data":"6dd13d29bd284106e5ad0e3a547a750792be75a1e1377ffb21727f122c838d0e"}
Sep 17 18:26:08.531598 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:26:08.531566    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr" event={"ID":"d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c","Type":"ContainerStarted","Data":"26ee40d5bf310310fbe1d4ed6686fc2fff53a1e65faf0dfa1ab674b558bc928b"}
Sep 17 18:26:08.532020 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:26:08.531626    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr" event={"ID":"d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c","Type":"ContainerStarted","Data":"c0d243d4de656e3e4c1d3533638e9e3454130bc3bd513c7424d440e2ca299857"}
Sep 17 18:26:08.532020 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:26:08.531903    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr"
Sep 17 18:26:08.532119 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:26:08.532056    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr"
Sep 17 18:26:08.533309 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:26:08.533283    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr" podUID="d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 17 18:26:08.560224 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:26:08.560180    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr" podStartSLOduration=2.561138596 podStartE2EDuration="13.560166391s" totalImagesPullingTime="7.231823772s" totalInitContainerRuntime="3.767204023s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:25:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:26:08.558165085 +0000 UTC m=+3104.971955515" watchObservedRunningTime="2026-09-17 18:26:08.560166391 +0000 UTC m=+3104.973956830"
Sep 17 18:26:09.535020 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:26:09.534975    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr" podUID="d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 17 18:26:10.537901 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:26:10.537859    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr" podUID="d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 17 18:26:15.543067 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:26:15.543038    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr"
Sep 17 18:26:15.543631 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:26:15.543605    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr" podUID="d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 17 18:26:18.985782 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:26:18.985748    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:25.544450 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:26:25.544391    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr" podUID="d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 17 18:26:35.544413 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:26:35.544374    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr" podUID="d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 17 18:26:43.281431 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:26:43.281381    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:45.543504 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:26:45.543462    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr" podUID="d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 17 18:26:48.418072 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:26:48.417971    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:48.471247 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:26:48.471212    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hrmkm"]
Sep 17 18:26:48.471579 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:26:48.471562    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb30e986-fbdd-4d10-b5c1-808580a4c75d" containerName="storage-initializer"
Sep 17 18:26:48.471670 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:26:48.471580    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb30e986-fbdd-4d10-b5c1-808580a4c75d" containerName="storage-initializer"
Sep 17 18:26:48.471670 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:26:48.471601    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb30e986-fbdd-4d10-b5c1-808580a4c75d" containerName="storage-initializer"
Sep 17 18:26:48.471670 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:26:48.471609    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb30e986-fbdd-4d10-b5c1-808580a4c75d" containerName="storage-initializer"
Sep 17 18:26:48.471836 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:26:48.471687    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb30e986-fbdd-4d10-b5c1-808580a4c75d" containerName="storage-initializer"
Sep 17 18:26:48.471836 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:26:48.471699    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb30e986-fbdd-4d10-b5c1-808580a4c75d" containerName="storage-initializer"
Sep 17 18:26:48.474644 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:26:48.474625    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hrmkm"
Sep 17 18:26:48.477076 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:26:48.477057    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-predictor-serving-cert\""
Sep 17 18:26:48.477195 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:26:48.477147    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
Sep 17 18:26:48.485692 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:26:48.485673    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hrmkm"]
Sep 17 18:26:48.540698 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:26:48.540665    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7px4\" (UniqueName: \"kubernetes.io/projected/18474a44-81ca-4e84-9b8a-dcaabd3490dc-kube-api-access-b7px4\") pod \"isvc-pmml-runtime-predictor-885fc8777-hrmkm\" (UID: \"18474a44-81ca-4e84-9b8a-dcaabd3490dc\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hrmkm"
Sep 17 18:26:48.540845 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:26:48.540708    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18474a44-81ca-4e84-9b8a-dcaabd3490dc-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-hrmkm\" (UID: \"18474a44-81ca-4e84-9b8a-dcaabd3490dc\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hrmkm"
Sep 17 18:26:48.540845 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:26:48.540736    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18474a44-81ca-4e84-9b8a-dcaabd3490dc-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-hrmkm\" (UID: \"18474a44-81ca-4e84-9b8a-dcaabd3490dc\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hrmkm"
Sep 17 18:26:48.540845 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:26:48.540809    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18474a44-81ca-4e84-9b8a-dcaabd3490dc-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-hrmkm\" (UID: \"18474a44-81ca-4e84-9b8a-dcaabd3490dc\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hrmkm"
Sep 17 18:26:48.540997 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:26:48.540870    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18474a44-81ca-4e84-9b8a-dcaabd3490dc-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-hrmkm\" (UID: \"18474a44-81ca-4e84-9b8a-dcaabd3490dc\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hrmkm"
Sep 17 18:26:48.607327 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:26:48.607284    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:48.642155 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:26:48.642129    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18474a44-81ca-4e84-9b8a-dcaabd3490dc-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-hrmkm\" (UID: \"18474a44-81ca-4e84-9b8a-dcaabd3490dc\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hrmkm"
Sep 17 18:26:48.642280 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:26:48.642163    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18474a44-81ca-4e84-9b8a-dcaabd3490dc-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-hrmkm\" (UID: \"18474a44-81ca-4e84-9b8a-dcaabd3490dc\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hrmkm"
Sep 17 18:26:48.642280 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:26:48.642215    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18474a44-81ca-4e84-9b8a-dcaabd3490dc-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-hrmkm\" (UID: \"18474a44-81ca-4e84-9b8a-dcaabd3490dc\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hrmkm"
Sep 17 18:26:48.642280 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:26:48.642241    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18474a44-81ca-4e84-9b8a-dcaabd3490dc-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-hrmkm\" (UID: \"18474a44-81ca-4e84-9b8a-dcaabd3490dc\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hrmkm"
Sep 17 18:26:48.642447 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:26:48.642377    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7px4\" (UniqueName: \"kubernetes.io/projected/18474a44-81ca-4e84-9b8a-dcaabd3490dc-kube-api-access-b7px4\") pod \"isvc-pmml-runtime-predictor-885fc8777-hrmkm\" (UID: \"18474a44-81ca-4e84-9b8a-dcaabd3490dc\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hrmkm"
Sep 17 18:26:48.642709 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:26:48.642690    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18474a44-81ca-4e84-9b8a-dcaabd3490dc-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-hrmkm\" (UID: \"18474a44-81ca-4e84-9b8a-dcaabd3490dc\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hrmkm"
Sep 17 18:26:48.642909 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:26:48.642889    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18474a44-81ca-4e84-9b8a-dcaabd3490dc-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-hrmkm\" (UID: \"18474a44-81ca-4e84-9b8a-dcaabd3490dc\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hrmkm"
Sep 17 18:26:48.642952 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:26:48.642928    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18474a44-81ca-4e84-9b8a-dcaabd3490dc-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-hrmkm\" (UID: \"18474a44-81ca-4e84-9b8a-dcaabd3490dc\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hrmkm"
Sep 17 18:26:48.644753 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:26:48.644737    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18474a44-81ca-4e84-9b8a-dcaabd3490dc-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-hrmkm\" (UID: \"18474a44-81ca-4e84-9b8a-dcaabd3490dc\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hrmkm"
Sep 17 18:26:48.651902 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:26:48.651883    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7px4\" (UniqueName: \"kubernetes.io/projected/18474a44-81ca-4e84-9b8a-dcaabd3490dc-kube-api-access-b7px4\") pod \"isvc-pmml-runtime-predictor-885fc8777-hrmkm\" (UID: \"18474a44-81ca-4e84-9b8a-dcaabd3490dc\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hrmkm"
Sep 17 18:26:48.785669 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:26:48.785636    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hrmkm"
Sep 17 18:26:49.059435 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:26:49.059368    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hrmkm"]
Sep 17 18:26:49.062850 ip-10-0-133-139 kubenswrapper[2585]: W0917 18:26:49.062820    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18474a44_81ca_4e84_9b8a_dcaabd3490dc.slice/crio-1a87a579ef97299dd98632d03dd067ed19d177c645f840326218ea5625960ce4 WatchSource:0}: Error finding container 1a87a579ef97299dd98632d03dd067ed19d177c645f840326218ea5625960ce4: Status 404 returned error can't find the container with id 1a87a579ef97299dd98632d03dd067ed19d177c645f840326218ea5625960ce4
Sep 17 18:26:49.680941 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:26:49.680903    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hrmkm" event={"ID":"18474a44-81ca-4e84-9b8a-dcaabd3490dc","Type":"ContainerStarted","Data":"0be9c50786ca722e654d51d0bdab8f01495395cc3db0132ff83df87feee60d9f"}
Sep 17 18:26:49.681315 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:26:49.680979    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hrmkm" event={"ID":"18474a44-81ca-4e84-9b8a-dcaabd3490dc","Type":"ContainerStarted","Data":"1a87a579ef97299dd98632d03dd067ed19d177c645f840326218ea5625960ce4"}
Sep 17 18:26:53.696963 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:26:53.696931    2585 generic.go:356] "Generic (PLEG): container finished" podID="18474a44-81ca-4e84-9b8a-dcaabd3490dc" containerID="0be9c50786ca722e654d51d0bdab8f01495395cc3db0132ff83df87feee60d9f" exitCode=0
Sep 17 18:26:53.697351 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:26:53.697010    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hrmkm" event={"ID":"18474a44-81ca-4e84-9b8a-dcaabd3490dc","Type":"ContainerDied","Data":"0be9c50786ca722e654d51d0bdab8f01495395cc3db0132ff83df87feee60d9f"}
Sep 17 18:26:54.703777 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:26:54.703737    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hrmkm" event={"ID":"18474a44-81ca-4e84-9b8a-dcaabd3490dc","Type":"ContainerStarted","Data":"0ff903f85fe6fb8e24c3d00bac169b8ea58d1470e1fe8d535bcb8f88086e404c"}
Sep 17 18:26:54.704258 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:26:54.703797    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hrmkm" event={"ID":"18474a44-81ca-4e84-9b8a-dcaabd3490dc","Type":"ContainerStarted","Data":"2623d494c4a0e1422215332c9edd320182246ddd5fb27a8c04cb6260a6ed750e"}
Sep 17 18:26:54.704258 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:26:54.703989    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hrmkm"
Sep 17 18:26:54.722769 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:26:54.722704    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hrmkm" podStartSLOduration=2.921919539 podStartE2EDuration="6.722683115s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.800763576s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:26:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:26:54.722336258 +0000 UTC m=+3151.136126701" watchObservedRunningTime="2026-09-17 18:26:54.722683115 +0000 UTC m=+3151.136473568"
Sep 17 18:26:55.544093 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:26:55.544054    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr" podUID="d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 17 18:26:55.709126 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:26:55.709093    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hrmkm"
Sep 17 18:26:55.710385 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:26:55.710358    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hrmkm" podUID="18474a44-81ca-4e84-9b8a-dcaabd3490dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 18:26:56.713510 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:26:56.713461    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hrmkm" podUID="18474a44-81ca-4e84-9b8a-dcaabd3490dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 18:27:01.718281 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:27:01.718244    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hrmkm"
Sep 17 18:27:01.718770 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:27:01.718744    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hrmkm" podUID="18474a44-81ca-4e84-9b8a-dcaabd3490dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 18:27:05.544332 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:27:05.544296    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr" podUID="d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 17 18:27:11.719751 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:27:11.719710    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hrmkm" podUID="18474a44-81ca-4e84-9b8a-dcaabd3490dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 18:27:15.544394 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:27:15.544356    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr" podUID="d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 17 18:27:21.718864 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:27:21.718826    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hrmkm" podUID="18474a44-81ca-4e84-9b8a-dcaabd3490dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 18:27:25.544020 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:27:25.543976    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr" podUID="d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 17 18:27:31.719087 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:27:31.719040    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hrmkm" podUID="18474a44-81ca-4e84-9b8a-dcaabd3490dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 18:27:35.543807 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:27:35.543774    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr" podUID="d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.51:8080: connect: connection refused"
Sep 17 18:27:41.719267 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:27:41.719230    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hrmkm" podUID="18474a44-81ca-4e84-9b8a-dcaabd3490dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 18:27:43.101136 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:27:43.101106    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr"
Sep 17 18:27:47.218553 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:27:47.218516    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr"]
Sep 17 18:27:47.218959 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:27:47.218750    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr" podUID="d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c" containerName="kserve-container" containerID="cri-o://c0d243d4de656e3e4c1d3533638e9e3454130bc3bd513c7424d440e2ca299857" gracePeriod=30
Sep 17 18:27:47.218959 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:27:47.218828    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr" podUID="d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c" containerName="kube-rbac-proxy" containerID="cri-o://26ee40d5bf310310fbe1d4ed6686fc2fff53a1e65faf0dfa1ab674b558bc928b" gracePeriod=30
Sep 17 18:27:47.243951 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:27:47.243909    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:27:47.914618 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:27:47.914587    2585 generic.go:356] "Generic (PLEG): container finished" podID="d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c" containerID="26ee40d5bf310310fbe1d4ed6686fc2fff53a1e65faf0dfa1ab674b558bc928b" exitCode=2
Sep 17 18:27:47.914791 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:27:47.914638    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr" event={"ID":"d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c","Type":"ContainerDied","Data":"26ee40d5bf310310fbe1d4ed6686fc2fff53a1e65faf0dfa1ab674b558bc928b"}
Sep 17 18:27:50.538917 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:27:50.538878    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr" podUID="d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.51:8643/healthz\": dial tcp 10.133.0.51:8643: connect: connection refused"
Sep 17 18:27:51.365272 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:27:51.365246    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr"
Sep 17 18:27:51.402744 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:27:51.402716    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c-cabundle-cert\") pod \"d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c\" (UID: \"d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c\") "
Sep 17 18:27:51.402931 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:27:51.402760    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lplck\" (UniqueName: \"kubernetes.io/projected/d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c-kube-api-access-lplck\") pod \"d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c\" (UID: \"d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c\") "
Sep 17 18:27:51.402931 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:27:51.402788    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c-proxy-tls\") pod \"d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c\" (UID: \"d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c\") "
Sep 17 18:27:51.402931 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:27:51.402824    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c-kserve-provision-location\") pod \"d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c\" (UID: \"d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c\") "
Sep 17 18:27:51.402931 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:27:51.402875    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c-kube-rbac-proxy-sar-config\") pod \"d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c\" (UID: \"d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c\") "
Sep 17 18:27:51.403218 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:27:51.403183    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c" (UID: "d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:27:51.403308 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:27:51.403193    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c" (UID: "d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:27:51.403308 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:27:51.403283    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c" (UID: "d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:27:51.405032 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:27:51.405007    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c-kube-api-access-lplck" (OuterVolumeSpecName: "kube-api-access-lplck") pod "d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c" (UID: "d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c"). InnerVolumeSpecName "kube-api-access-lplck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:27:51.405130 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:27:51.405006    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c" (UID: "d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:27:51.504125 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:27:51.504094    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lplck\" (UniqueName: \"kubernetes.io/projected/d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c-kube-api-access-lplck\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:27:51.504274 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:27:51.504130    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c-proxy-tls\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:27:51.504274 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:27:51.504145    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c-kserve-provision-location\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:27:51.504274 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:27:51.504159    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:27:51.504274 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:27:51.504173    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c-cabundle-cert\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:27:51.719386 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:27:51.719350    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hrmkm" podUID="18474a44-81ca-4e84-9b8a-dcaabd3490dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 18:27:51.931865 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:27:51.931762    2585 generic.go:356] "Generic (PLEG): container finished" podID="d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c" containerID="c0d243d4de656e3e4c1d3533638e9e3454130bc3bd513c7424d440e2ca299857" exitCode=0
Sep 17 18:27:51.931865 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:27:51.931832    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr"
Sep 17 18:27:51.932094 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:27:51.931832    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr" event={"ID":"d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c","Type":"ContainerDied","Data":"c0d243d4de656e3e4c1d3533638e9e3454130bc3bd513c7424d440e2ca299857"}
Sep 17 18:27:51.932094 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:27:51.931999    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr" event={"ID":"d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c","Type":"ContainerDied","Data":"8308168f0acb0aeee9f730000c2949833a6a8a365ea576759ac4a241b411367f"}
Sep 17 18:27:51.932094 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:27:51.932038    2585 scope.go:117] "RemoveContainer" containerID="26ee40d5bf310310fbe1d4ed6686fc2fff53a1e65faf0dfa1ab674b558bc928b"
Sep 17 18:27:51.940244 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:27:51.940230    2585 scope.go:117] "RemoveContainer" containerID="c0d243d4de656e3e4c1d3533638e9e3454130bc3bd513c7424d440e2ca299857"
Sep 17 18:27:51.947499 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:27:51.947472    2585 scope.go:117] "RemoveContainer" containerID="6dd13d29bd284106e5ad0e3a547a750792be75a1e1377ffb21727f122c838d0e"
Sep 17 18:27:51.954610 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:27:51.954593    2585 scope.go:117] "RemoveContainer" containerID="26ee40d5bf310310fbe1d4ed6686fc2fff53a1e65faf0dfa1ab674b558bc928b"
Sep 17 18:27:51.955014 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:27:51.954956    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26ee40d5bf310310fbe1d4ed6686fc2fff53a1e65faf0dfa1ab674b558bc928b\": container with ID starting with 26ee40d5bf310310fbe1d4ed6686fc2fff53a1e65faf0dfa1ab674b558bc928b not found: ID does not exist" containerID="26ee40d5bf310310fbe1d4ed6686fc2fff53a1e65faf0dfa1ab674b558bc928b"
Sep 17 18:27:51.955014 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:27:51.954998    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26ee40d5bf310310fbe1d4ed6686fc2fff53a1e65faf0dfa1ab674b558bc928b"} err="failed to get container status \"26ee40d5bf310310fbe1d4ed6686fc2fff53a1e65faf0dfa1ab674b558bc928b\": rpc error: code = NotFound desc = could not find container \"26ee40d5bf310310fbe1d4ed6686fc2fff53a1e65faf0dfa1ab674b558bc928b\": container with ID starting with 26ee40d5bf310310fbe1d4ed6686fc2fff53a1e65faf0dfa1ab674b558bc928b not found: ID does not exist"
Sep 17 18:27:51.955251 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:27:51.955021    2585 scope.go:117] "RemoveContainer" containerID="c0d243d4de656e3e4c1d3533638e9e3454130bc3bd513c7424d440e2ca299857"
Sep 17 18:27:51.955465 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:27:51.955388    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0d243d4de656e3e4c1d3533638e9e3454130bc3bd513c7424d440e2ca299857\": container with ID starting with c0d243d4de656e3e4c1d3533638e9e3454130bc3bd513c7424d440e2ca299857 not found: ID does not exist" containerID="c0d243d4de656e3e4c1d3533638e9e3454130bc3bd513c7424d440e2ca299857"
Sep 17 18:27:51.955465 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:27:51.955438    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0d243d4de656e3e4c1d3533638e9e3454130bc3bd513c7424d440e2ca299857"} err="failed to get container status \"c0d243d4de656e3e4c1d3533638e9e3454130bc3bd513c7424d440e2ca299857\": rpc error: code = NotFound desc = could not find container \"c0d243d4de656e3e4c1d3533638e9e3454130bc3bd513c7424d440e2ca299857\": container with ID starting with c0d243d4de656e3e4c1d3533638e9e3454130bc3bd513c7424d440e2ca299857 not found: ID does not exist"
Sep 17 18:27:51.955465 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:27:51.955460    2585 scope.go:117] "RemoveContainer" containerID="6dd13d29bd284106e5ad0e3a547a750792be75a1e1377ffb21727f122c838d0e"
Sep 17 18:27:51.955798 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:27:51.955779    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dd13d29bd284106e5ad0e3a547a750792be75a1e1377ffb21727f122c838d0e\": container with ID starting with 6dd13d29bd284106e5ad0e3a547a750792be75a1e1377ffb21727f122c838d0e not found: ID does not exist" containerID="6dd13d29bd284106e5ad0e3a547a750792be75a1e1377ffb21727f122c838d0e"
Sep 17 18:27:51.955876 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:27:51.955806    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dd13d29bd284106e5ad0e3a547a750792be75a1e1377ffb21727f122c838d0e"} err="failed to get container status \"6dd13d29bd284106e5ad0e3a547a750792be75a1e1377ffb21727f122c838d0e\": rpc error: code = NotFound desc = could not find container \"6dd13d29bd284106e5ad0e3a547a750792be75a1e1377ffb21727f122c838d0e\": container with ID starting with 6dd13d29bd284106e5ad0e3a547a750792be75a1e1377ffb21727f122c838d0e not found: ID does not exist"
Sep 17 18:27:51.956955 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:27:51.956933    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr"]
Sep 17 18:27:51.958857 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:27:51.958838    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-hc4jr"]
Sep 17 18:27:52.103721 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:27:52.103682    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c" path="/var/lib/kubelet/pods/d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c/volumes"
Sep 17 18:28:01.719103 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:01.719064    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hrmkm" podUID="18474a44-81ca-4e84-9b8a-dcaabd3490dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 18:28:11.719039 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:11.718997    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hrmkm" podUID="18474a44-81ca-4e84-9b8a-dcaabd3490dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 18:28:18.100340 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:18.100249    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hrmkm" podUID="18474a44-81ca-4e84-9b8a-dcaabd3490dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 18:28:28.103357 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:28.103320    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hrmkm"
Sep 17 18:28:30.138767 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:30.138736    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hrmkm"]
Sep 17 18:28:30.139174 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:30.138980    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hrmkm" podUID="18474a44-81ca-4e84-9b8a-dcaabd3490dc" containerName="kserve-container" containerID="cri-o://2623d494c4a0e1422215332c9edd320182246ddd5fb27a8c04cb6260a6ed750e" gracePeriod=30
Sep 17 18:28:30.139174 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:30.138994    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hrmkm" podUID="18474a44-81ca-4e84-9b8a-dcaabd3490dc" containerName="kube-rbac-proxy" containerID="cri-o://0ff903f85fe6fb8e24c3d00bac169b8ea58d1470e1fe8d535bcb8f88086e404c" gracePeriod=30
Sep 17 18:28:30.169628 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:30.169606    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:28:31.069881 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:31.069845    2585 generic.go:356] "Generic (PLEG): container finished" podID="18474a44-81ca-4e84-9b8a-dcaabd3490dc" containerID="0ff903f85fe6fb8e24c3d00bac169b8ea58d1470e1fe8d535bcb8f88086e404c" exitCode=2
Sep 17 18:28:31.070053 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:31.069910    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hrmkm" event={"ID":"18474a44-81ca-4e84-9b8a-dcaabd3490dc","Type":"ContainerDied","Data":"0ff903f85fe6fb8e24c3d00bac169b8ea58d1470e1fe8d535bcb8f88086e404c"}
Sep 17 18:28:31.714456 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:31.714419    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hrmkm" podUID="18474a44-81ca-4e84-9b8a-dcaabd3490dc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.52:8643/healthz\": dial tcp 10.133.0.52:8643: connect: connection refused"
Sep 17 18:28:34.476404 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:34.476379    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hrmkm"
Sep 17 18:28:34.631450 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:34.631371    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7px4\" (UniqueName: \"kubernetes.io/projected/18474a44-81ca-4e84-9b8a-dcaabd3490dc-kube-api-access-b7px4\") pod \"18474a44-81ca-4e84-9b8a-dcaabd3490dc\" (UID: \"18474a44-81ca-4e84-9b8a-dcaabd3490dc\") "
Sep 17 18:28:34.631450 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:34.631432    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18474a44-81ca-4e84-9b8a-dcaabd3490dc-cabundle-cert\") pod \"18474a44-81ca-4e84-9b8a-dcaabd3490dc\" (UID: \"18474a44-81ca-4e84-9b8a-dcaabd3490dc\") "
Sep 17 18:28:34.631695 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:34.631454    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18474a44-81ca-4e84-9b8a-dcaabd3490dc-kserve-provision-location\") pod \"18474a44-81ca-4e84-9b8a-dcaabd3490dc\" (UID: \"18474a44-81ca-4e84-9b8a-dcaabd3490dc\") "
Sep 17 18:28:34.631695 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:34.631485    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18474a44-81ca-4e84-9b8a-dcaabd3490dc-kube-rbac-proxy-sar-config\") pod \"18474a44-81ca-4e84-9b8a-dcaabd3490dc\" (UID: \"18474a44-81ca-4e84-9b8a-dcaabd3490dc\") "
Sep 17 18:28:34.631695 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:34.631531    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18474a44-81ca-4e84-9b8a-dcaabd3490dc-proxy-tls\") pod \"18474a44-81ca-4e84-9b8a-dcaabd3490dc\" (UID: \"18474a44-81ca-4e84-9b8a-dcaabd3490dc\") "
Sep 17 18:28:34.631860 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:34.631810    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18474a44-81ca-4e84-9b8a-dcaabd3490dc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "18474a44-81ca-4e84-9b8a-dcaabd3490dc" (UID: "18474a44-81ca-4e84-9b8a-dcaabd3490dc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:28:34.631860 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:34.631833    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18474a44-81ca-4e84-9b8a-dcaabd3490dc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "18474a44-81ca-4e84-9b8a-dcaabd3490dc" (UID: "18474a44-81ca-4e84-9b8a-dcaabd3490dc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:28:34.631931 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:34.631851    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18474a44-81ca-4e84-9b8a-dcaabd3490dc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "18474a44-81ca-4e84-9b8a-dcaabd3490dc" (UID: "18474a44-81ca-4e84-9b8a-dcaabd3490dc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:28:34.633456 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:34.633434    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18474a44-81ca-4e84-9b8a-dcaabd3490dc-kube-api-access-b7px4" (OuterVolumeSpecName: "kube-api-access-b7px4") pod "18474a44-81ca-4e84-9b8a-dcaabd3490dc" (UID: "18474a44-81ca-4e84-9b8a-dcaabd3490dc"). InnerVolumeSpecName "kube-api-access-b7px4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:28:34.633568 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:34.633452    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18474a44-81ca-4e84-9b8a-dcaabd3490dc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "18474a44-81ca-4e84-9b8a-dcaabd3490dc" (UID: "18474a44-81ca-4e84-9b8a-dcaabd3490dc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:28:34.732078 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:34.732050    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7px4\" (UniqueName: \"kubernetes.io/projected/18474a44-81ca-4e84-9b8a-dcaabd3490dc-kube-api-access-b7px4\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:28:34.732078 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:34.732074    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18474a44-81ca-4e84-9b8a-dcaabd3490dc-cabundle-cert\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:28:34.732231 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:34.732084    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18474a44-81ca-4e84-9b8a-dcaabd3490dc-kserve-provision-location\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:28:34.732231 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:34.732092    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18474a44-81ca-4e84-9b8a-dcaabd3490dc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:28:34.732231 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:34.732102    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18474a44-81ca-4e84-9b8a-dcaabd3490dc-proxy-tls\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:28:35.085065 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:35.085028    2585 generic.go:356] "Generic (PLEG): container finished" podID="18474a44-81ca-4e84-9b8a-dcaabd3490dc" containerID="2623d494c4a0e1422215332c9edd320182246ddd5fb27a8c04cb6260a6ed750e" exitCode=0
Sep 17 18:28:35.085248 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:35.085074    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hrmkm" event={"ID":"18474a44-81ca-4e84-9b8a-dcaabd3490dc","Type":"ContainerDied","Data":"2623d494c4a0e1422215332c9edd320182246ddd5fb27a8c04cb6260a6ed750e"}
Sep 17 18:28:35.085248 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:35.085086    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hrmkm"
Sep 17 18:28:35.085248 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:35.085134    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hrmkm" event={"ID":"18474a44-81ca-4e84-9b8a-dcaabd3490dc","Type":"ContainerDied","Data":"1a87a579ef97299dd98632d03dd067ed19d177c645f840326218ea5625960ce4"}
Sep 17 18:28:35.085248 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:35.085170    2585 scope.go:117] "RemoveContainer" containerID="0ff903f85fe6fb8e24c3d00bac169b8ea58d1470e1fe8d535bcb8f88086e404c"
Sep 17 18:28:35.093335 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:35.093315    2585 scope.go:117] "RemoveContainer" containerID="2623d494c4a0e1422215332c9edd320182246ddd5fb27a8c04cb6260a6ed750e"
Sep 17 18:28:35.100345 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:35.100327    2585 scope.go:117] "RemoveContainer" containerID="0be9c50786ca722e654d51d0bdab8f01495395cc3db0132ff83df87feee60d9f"
Sep 17 18:28:35.107178 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:35.107161    2585 scope.go:117] "RemoveContainer" containerID="0ff903f85fe6fb8e24c3d00bac169b8ea58d1470e1fe8d535bcb8f88086e404c"
Sep 17 18:28:35.107460 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:28:35.107438    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ff903f85fe6fb8e24c3d00bac169b8ea58d1470e1fe8d535bcb8f88086e404c\": container with ID starting with 0ff903f85fe6fb8e24c3d00bac169b8ea58d1470e1fe8d535bcb8f88086e404c not found: ID does not exist" containerID="0ff903f85fe6fb8e24c3d00bac169b8ea58d1470e1fe8d535bcb8f88086e404c"
Sep 17 18:28:35.107563 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:35.107472    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ff903f85fe6fb8e24c3d00bac169b8ea58d1470e1fe8d535bcb8f88086e404c"} err="failed to get container status \"0ff903f85fe6fb8e24c3d00bac169b8ea58d1470e1fe8d535bcb8f88086e404c\": rpc error: code = NotFound desc = could not find container \"0ff903f85fe6fb8e24c3d00bac169b8ea58d1470e1fe8d535bcb8f88086e404c\": container with ID starting with 0ff903f85fe6fb8e24c3d00bac169b8ea58d1470e1fe8d535bcb8f88086e404c not found: ID does not exist"
Sep 17 18:28:35.107563 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:35.107519    2585 scope.go:117] "RemoveContainer" containerID="2623d494c4a0e1422215332c9edd320182246ddd5fb27a8c04cb6260a6ed750e"
Sep 17 18:28:35.107808 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:28:35.107790    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2623d494c4a0e1422215332c9edd320182246ddd5fb27a8c04cb6260a6ed750e\": container with ID starting with 2623d494c4a0e1422215332c9edd320182246ddd5fb27a8c04cb6260a6ed750e not found: ID does not exist" containerID="2623d494c4a0e1422215332c9edd320182246ddd5fb27a8c04cb6260a6ed750e"
Sep 17 18:28:35.107861 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:35.107817    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2623d494c4a0e1422215332c9edd320182246ddd5fb27a8c04cb6260a6ed750e"} err="failed to get container status \"2623d494c4a0e1422215332c9edd320182246ddd5fb27a8c04cb6260a6ed750e\": rpc error: code = NotFound desc = could not find container \"2623d494c4a0e1422215332c9edd320182246ddd5fb27a8c04cb6260a6ed750e\": container with ID starting with 2623d494c4a0e1422215332c9edd320182246ddd5fb27a8c04cb6260a6ed750e not found: ID does not exist"
Sep 17 18:28:35.107861 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:35.107831    2585 scope.go:117] "RemoveContainer" containerID="0be9c50786ca722e654d51d0bdab8f01495395cc3db0132ff83df87feee60d9f"
Sep 17 18:28:35.108071 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:28:35.108055    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0be9c50786ca722e654d51d0bdab8f01495395cc3db0132ff83df87feee60d9f\": container with ID starting with 0be9c50786ca722e654d51d0bdab8f01495395cc3db0132ff83df87feee60d9f not found: ID does not exist" containerID="0be9c50786ca722e654d51d0bdab8f01495395cc3db0132ff83df87feee60d9f"
Sep 17 18:28:35.108123 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:35.108077    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0be9c50786ca722e654d51d0bdab8f01495395cc3db0132ff83df87feee60d9f"} err="failed to get container status \"0be9c50786ca722e654d51d0bdab8f01495395cc3db0132ff83df87feee60d9f\": rpc error: code = NotFound desc = could not find container \"0be9c50786ca722e654d51d0bdab8f01495395cc3db0132ff83df87feee60d9f\": container with ID starting with 0be9c50786ca722e654d51d0bdab8f01495395cc3db0132ff83df87feee60d9f not found: ID does not exist"
Sep 17 18:28:35.108213 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:35.108196    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hrmkm"]
Sep 17 18:28:35.112206 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:35.112187    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-hrmkm"]
Sep 17 18:28:35.294404 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:35.294360    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:28:35.355299 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:35.355223    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2"]
Sep 17 18:28:35.355470 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:35.355458    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c" containerName="kserve-container"
Sep 17 18:28:35.355538 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:35.355471    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c" containerName="kserve-container"
Sep 17 18:28:35.355538 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:35.355482    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18474a44-81ca-4e84-9b8a-dcaabd3490dc" containerName="storage-initializer"
Sep 17 18:28:35.355538 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:35.355501    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="18474a44-81ca-4e84-9b8a-dcaabd3490dc" containerName="storage-initializer"
Sep 17 18:28:35.355538 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:35.355507    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c" containerName="storage-initializer"
Sep 17 18:28:35.355538 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:35.355513    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c" containerName="storage-initializer"
Sep 17 18:28:35.355538 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:35.355518    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18474a44-81ca-4e84-9b8a-dcaabd3490dc" containerName="kserve-container"
Sep 17 18:28:35.355538 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:35.355523    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="18474a44-81ca-4e84-9b8a-dcaabd3490dc" containerName="kserve-container"
Sep 17 18:28:35.355538 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:35.355533    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c" containerName="kube-rbac-proxy"
Sep 17 18:28:35.355538 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:35.355538    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c" containerName="kube-rbac-proxy"
Sep 17 18:28:35.355813 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:35.355547    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18474a44-81ca-4e84-9b8a-dcaabd3490dc" containerName="kube-rbac-proxy"
Sep 17 18:28:35.355813 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:35.355552    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="18474a44-81ca-4e84-9b8a-dcaabd3490dc" containerName="kube-rbac-proxy"
Sep 17 18:28:35.355813 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:35.355597    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="18474a44-81ca-4e84-9b8a-dcaabd3490dc" containerName="kserve-container"
Sep 17 18:28:35.355813 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:35.355606    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c" containerName="kserve-container"
Sep 17 18:28:35.355813 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:35.355612    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="18474a44-81ca-4e84-9b8a-dcaabd3490dc" containerName="kube-rbac-proxy"
Sep 17 18:28:35.355813 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:35.355618    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0ae2cc6-2b4b-402c-bfb6-a55a2d71d23c" containerName="kube-rbac-proxy"
Sep 17 18:28:35.359965 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:35.359947    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2"
Sep 17 18:28:35.362250 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:35.362225    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 18:28:35.362416 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:35.362310    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 18:28:35.362482 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:35.362475    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-a52b29-kube-rbac-proxy-sar-config\""
Sep 17 18:28:35.362671 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:35.362637    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 18:28:35.362793 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:35.362714    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 18:28:35.362793 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:35.362738    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-a52b29-predictor-serving-cert\""
Sep 17 18:28:35.362793 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:35.362785    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lps5w\""
Sep 17 18:28:35.369889 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:35.369868    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2"]
Sep 17 18:28:35.391120 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:35.391100    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:28:35.439118 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:35.439092    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkpq8\" (UniqueName: \"kubernetes.io/projected/0a40eca7-1139-45a0-80e6-1f6e054be941-kube-api-access-qkpq8\") pod \"isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2\" (UID: \"0a40eca7-1139-45a0-80e6-1f6e054be941\") " pod="kserve-ci-e2e-test/isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2"
Sep 17 18:28:35.439227 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:35.439134    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a40eca7-1139-45a0-80e6-1f6e054be941-proxy-tls\") pod \"isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2\" (UID: \"0a40eca7-1139-45a0-80e6-1f6e054be941\") " pod="kserve-ci-e2e-test/isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2"
Sep 17 18:28:35.439227 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:35.439160    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a40eca7-1139-45a0-80e6-1f6e054be941-kserve-provision-location\") pod \"isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2\" (UID: \"0a40eca7-1139-45a0-80e6-1f6e054be941\") " pod="kserve-ci-e2e-test/isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2"
Sep 17 18:28:35.439227 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:35.439220    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0a40eca7-1139-45a0-80e6-1f6e054be941-kube-rbac-proxy-sar-config\") pod \"isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2\" (UID: \"0a40eca7-1139-45a0-80e6-1f6e054be941\") " pod="kserve-ci-e2e-test/isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2"
Sep 17 18:28:35.439361 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:35.439243    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0a40eca7-1139-45a0-80e6-1f6e054be941-cabundle-cert\") pod \"isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2\" (UID: \"0a40eca7-1139-45a0-80e6-1f6e054be941\") " pod="kserve-ci-e2e-test/isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2"
Sep 17 18:28:35.540486 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:35.540452    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a40eca7-1139-45a0-80e6-1f6e054be941-kserve-provision-location\") pod \"isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2\" (UID: \"0a40eca7-1139-45a0-80e6-1f6e054be941\") " pod="kserve-ci-e2e-test/isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2"
Sep 17 18:28:35.540942 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:35.540520    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0a40eca7-1139-45a0-80e6-1f6e054be941-kube-rbac-proxy-sar-config\") pod \"isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2\" (UID: \"0a40eca7-1139-45a0-80e6-1f6e054be941\") " pod="kserve-ci-e2e-test/isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2"
Sep 17 18:28:35.540942 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:35.540552    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0a40eca7-1139-45a0-80e6-1f6e054be941-cabundle-cert\") pod \"isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2\" (UID: \"0a40eca7-1139-45a0-80e6-1f6e054be941\") " pod="kserve-ci-e2e-test/isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2"
Sep 17 18:28:35.540942 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:35.540693    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkpq8\" (UniqueName: \"kubernetes.io/projected/0a40eca7-1139-45a0-80e6-1f6e054be941-kube-api-access-qkpq8\") pod \"isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2\" (UID: \"0a40eca7-1139-45a0-80e6-1f6e054be941\") " pod="kserve-ci-e2e-test/isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2"
Sep 17 18:28:35.540942 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:35.540764    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a40eca7-1139-45a0-80e6-1f6e054be941-proxy-tls\") pod \"isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2\" (UID: \"0a40eca7-1139-45a0-80e6-1f6e054be941\") " pod="kserve-ci-e2e-test/isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2"
Sep 17 18:28:35.540942 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:35.540866    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a40eca7-1139-45a0-80e6-1f6e054be941-kserve-provision-location\") pod \"isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2\" (UID: \"0a40eca7-1139-45a0-80e6-1f6e054be941\") " pod="kserve-ci-e2e-test/isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2"
Sep 17 18:28:35.541234 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:35.541141    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0a40eca7-1139-45a0-80e6-1f6e054be941-kube-rbac-proxy-sar-config\") pod \"isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2\" (UID: \"0a40eca7-1139-45a0-80e6-1f6e054be941\") " pod="kserve-ci-e2e-test/isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2"
Sep 17 18:28:35.541293 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:35.541254    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0a40eca7-1139-45a0-80e6-1f6e054be941-cabundle-cert\") pod \"isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2\" (UID: \"0a40eca7-1139-45a0-80e6-1f6e054be941\") " pod="kserve-ci-e2e-test/isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2"
Sep 17 18:28:35.543102 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:35.543084    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a40eca7-1139-45a0-80e6-1f6e054be941-proxy-tls\") pod \"isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2\" (UID: \"0a40eca7-1139-45a0-80e6-1f6e054be941\") " pod="kserve-ci-e2e-test/isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2"
Sep 17 18:28:35.549449 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:35.549429    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkpq8\" (UniqueName: \"kubernetes.io/projected/0a40eca7-1139-45a0-80e6-1f6e054be941-kube-api-access-qkpq8\") pod \"isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2\" (UID: \"0a40eca7-1139-45a0-80e6-1f6e054be941\") " pod="kserve-ci-e2e-test/isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2"
Sep 17 18:28:35.670801 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:35.670722    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2"
Sep 17 18:28:35.942905 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:35.942865    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2"]
Sep 17 18:28:35.945445 ip-10-0-133-139 kubenswrapper[2585]: W0917 18:28:35.945423    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a40eca7_1139_45a0_80e6_1f6e054be941.slice/crio-7d04159788aa68d7809508f67ac699a97964aa7759fb7a4121b70e5fc723c2d3 WatchSource:0}: Error finding container 7d04159788aa68d7809508f67ac699a97964aa7759fb7a4121b70e5fc723c2d3: Status 404 returned error can't find the container with id 7d04159788aa68d7809508f67ac699a97964aa7759fb7a4121b70e5fc723c2d3
Sep 17 18:28:36.090642 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:36.090608    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2" event={"ID":"0a40eca7-1139-45a0-80e6-1f6e054be941","Type":"ContainerStarted","Data":"af32eab28a78e53a055555a135b4f729a313c03e50d5e584ce439c04adddf574"}
Sep 17 18:28:36.090840 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:36.090667    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2" event={"ID":"0a40eca7-1139-45a0-80e6-1f6e054be941","Type":"ContainerStarted","Data":"7d04159788aa68d7809508f67ac699a97964aa7759fb7a4121b70e5fc723c2d3"}
Sep 17 18:28:36.103998 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:36.103970    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18474a44-81ca-4e84-9b8a-dcaabd3490dc" path="/var/lib/kubelet/pods/18474a44-81ca-4e84-9b8a-dcaabd3490dc/volumes"
Sep 17 18:28:40.108506 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:40.108460    2585 generic.go:356] "Generic (PLEG): container finished" podID="0a40eca7-1139-45a0-80e6-1f6e054be941" containerID="af32eab28a78e53a055555a135b4f729a313c03e50d5e584ce439c04adddf574" exitCode=0
Sep 17 18:28:40.108896 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:40.108523    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2" event={"ID":"0a40eca7-1139-45a0-80e6-1f6e054be941","Type":"ContainerDied","Data":"af32eab28a78e53a055555a135b4f729a313c03e50d5e584ce439c04adddf574"}
Sep 17 18:28:41.113649 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:41.113613    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2" event={"ID":"0a40eca7-1139-45a0-80e6-1f6e054be941","Type":"ContainerStarted","Data":"49d7ecce11f6c2e80811a708ff37895101a4e70a0d23eaee2e94381b4efb9307"}
Sep 17 18:28:41.114078 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:41.113671    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2" event={"ID":"0a40eca7-1139-45a0-80e6-1f6e054be941","Type":"ContainerStarted","Data":"f160b14e12aa6f5a4cb653d604257e172a39d4b0abca647296fec74ca93f697f"}
Sep 17 18:28:41.114078 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:41.113923    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2"
Sep 17 18:28:41.114078 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:41.114067    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2"
Sep 17 18:28:41.115531 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:41.115505    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2" podUID="0a40eca7-1139-45a0-80e6-1f6e054be941" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 17 18:28:41.134334 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:41.134293    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2" podStartSLOduration=2.452033958 podStartE2EDuration="6.134279148s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.68224519s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:28:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:28:41.132220196 +0000 UTC m=+3257.546010627" watchObservedRunningTime="2026-09-17 18:28:41.134279148 +0000 UTC m=+3257.548069586"
Sep 17 18:28:42.116965 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:42.116922    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2" podUID="0a40eca7-1139-45a0-80e6-1f6e054be941" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 17 18:28:47.121788 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:47.121759    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2"
Sep 17 18:28:47.122436 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:47.122405    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2" podUID="0a40eca7-1139-45a0-80e6-1f6e054be941" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 17 18:28:57.122290 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:28:57.122246    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2" podUID="0a40eca7-1139-45a0-80e6-1f6e054be941" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 17 18:29:07.122551 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:29:07.122487    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2" podUID="0a40eca7-1139-45a0-80e6-1f6e054be941" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 17 18:29:17.122821 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:29:17.122782    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2" podUID="0a40eca7-1139-45a0-80e6-1f6e054be941" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 17 18:29:27.123252 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:29:27.123210    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2" podUID="0a40eca7-1139-45a0-80e6-1f6e054be941" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 17 18:29:28.542779 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:29:28.542745    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rrf79_beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5/ovn-acl-logging/0.log"
Sep 17 18:29:28.553581 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:29:28.553548    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rrf79_beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5/ovn-acl-logging/0.log"
Sep 17 18:29:37.122379 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:29:37.122337    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2" podUID="0a40eca7-1139-45a0-80e6-1f6e054be941" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 17 18:29:47.123056 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:29:47.122972    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2" podUID="0a40eca7-1139-45a0-80e6-1f6e054be941" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 17 18:29:49.826523 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:29:49.826448    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:55.936843 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:29:55.936806    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:57.123515 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:29:57.123468    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2"
Sep 17 18:30:20.492536 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:20.492456    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:20.573767 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:20.573737    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-a52b29-predictor-6bf9d85478-l5x8b"]
Sep 17 18:30:20.576903 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:20.576886    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-a52b29-predictor-6bf9d85478-l5x8b"
Sep 17 18:30:20.580094 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:20.580074    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-a52b29-dockercfg-z9msx\""
Sep 17 18:30:20.580323 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:20.580307    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-a52b29-predictor-serving-cert\""
Sep 17 18:30:20.580859 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:20.580843    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-a52b29-kube-rbac-proxy-sar-config\""
Sep 17 18:30:20.581431 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:20.581415    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-a52b29\""
Sep 17 18:30:20.591535 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:20.591513    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-a52b29-predictor-6bf9d85478-l5x8b"]
Sep 17 18:30:20.634255 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:20.634232    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:20.668247 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:20.668220    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e20058de-e3e6-40f3-830b-44c19095bb0a-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-a52b29-predictor-6bf9d85478-l5x8b\" (UID: \"e20058de-e3e6-40f3-830b-44c19095bb0a\") " pod="kserve-ci-e2e-test/isvc-secondary-a52b29-predictor-6bf9d85478-l5x8b"
Sep 17 18:30:20.668358 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:20.668261    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e20058de-e3e6-40f3-830b-44c19095bb0a-cabundle-cert\") pod \"isvc-secondary-a52b29-predictor-6bf9d85478-l5x8b\" (UID: \"e20058de-e3e6-40f3-830b-44c19095bb0a\") " pod="kserve-ci-e2e-test/isvc-secondary-a52b29-predictor-6bf9d85478-l5x8b"
Sep 17 18:30:20.668358 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:20.668330    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e20058de-e3e6-40f3-830b-44c19095bb0a-proxy-tls\") pod \"isvc-secondary-a52b29-predictor-6bf9d85478-l5x8b\" (UID: \"e20058de-e3e6-40f3-830b-44c19095bb0a\") " pod="kserve-ci-e2e-test/isvc-secondary-a52b29-predictor-6bf9d85478-l5x8b"
Sep 17 18:30:20.668438 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:20.668357    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk2vk\" (UniqueName: \"kubernetes.io/projected/e20058de-e3e6-40f3-830b-44c19095bb0a-kube-api-access-hk2vk\") pod \"isvc-secondary-a52b29-predictor-6bf9d85478-l5x8b\" (UID: \"e20058de-e3e6-40f3-830b-44c19095bb0a\") " pod="kserve-ci-e2e-test/isvc-secondary-a52b29-predictor-6bf9d85478-l5x8b"
Sep 17 18:30:20.668438 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:20.668383    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e20058de-e3e6-40f3-830b-44c19095bb0a-kserve-provision-location\") pod \"isvc-secondary-a52b29-predictor-6bf9d85478-l5x8b\" (UID: \"e20058de-e3e6-40f3-830b-44c19095bb0a\") " pod="kserve-ci-e2e-test/isvc-secondary-a52b29-predictor-6bf9d85478-l5x8b"
Sep 17 18:30:20.769547 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:20.769516    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e20058de-e3e6-40f3-830b-44c19095bb0a-proxy-tls\") pod \"isvc-secondary-a52b29-predictor-6bf9d85478-l5x8b\" (UID: \"e20058de-e3e6-40f3-830b-44c19095bb0a\") " pod="kserve-ci-e2e-test/isvc-secondary-a52b29-predictor-6bf9d85478-l5x8b"
Sep 17 18:30:20.769707 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:20.769557    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk2vk\" (UniqueName: \"kubernetes.io/projected/e20058de-e3e6-40f3-830b-44c19095bb0a-kube-api-access-hk2vk\") pod \"isvc-secondary-a52b29-predictor-6bf9d85478-l5x8b\" (UID: \"e20058de-e3e6-40f3-830b-44c19095bb0a\") " pod="kserve-ci-e2e-test/isvc-secondary-a52b29-predictor-6bf9d85478-l5x8b"
Sep 17 18:30:20.769707 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:20.769580    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e20058de-e3e6-40f3-830b-44c19095bb0a-kserve-provision-location\") pod \"isvc-secondary-a52b29-predictor-6bf9d85478-l5x8b\" (UID: \"e20058de-e3e6-40f3-830b-44c19095bb0a\") " pod="kserve-ci-e2e-test/isvc-secondary-a52b29-predictor-6bf9d85478-l5x8b"
Sep 17 18:30:20.769707 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:20.769614    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e20058de-e3e6-40f3-830b-44c19095bb0a-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-a52b29-predictor-6bf9d85478-l5x8b\" (UID: \"e20058de-e3e6-40f3-830b-44c19095bb0a\") " pod="kserve-ci-e2e-test/isvc-secondary-a52b29-predictor-6bf9d85478-l5x8b"
Sep 17 18:30:20.769707 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:20.769658    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e20058de-e3e6-40f3-830b-44c19095bb0a-cabundle-cert\") pod \"isvc-secondary-a52b29-predictor-6bf9d85478-l5x8b\" (UID: \"e20058de-e3e6-40f3-830b-44c19095bb0a\") " pod="kserve-ci-e2e-test/isvc-secondary-a52b29-predictor-6bf9d85478-l5x8b"
Sep 17 18:30:20.770131 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:20.770112    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e20058de-e3e6-40f3-830b-44c19095bb0a-kserve-provision-location\") pod \"isvc-secondary-a52b29-predictor-6bf9d85478-l5x8b\" (UID: \"e20058de-e3e6-40f3-830b-44c19095bb0a\") " pod="kserve-ci-e2e-test/isvc-secondary-a52b29-predictor-6bf9d85478-l5x8b"
Sep 17 18:30:20.770406 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:20.770389    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e20058de-e3e6-40f3-830b-44c19095bb0a-cabundle-cert\") pod \"isvc-secondary-a52b29-predictor-6bf9d85478-l5x8b\" (UID: \"e20058de-e3e6-40f3-830b-44c19095bb0a\") " pod="kserve-ci-e2e-test/isvc-secondary-a52b29-predictor-6bf9d85478-l5x8b"
Sep 17 18:30:20.770447 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:20.770409    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e20058de-e3e6-40f3-830b-44c19095bb0a-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-a52b29-predictor-6bf9d85478-l5x8b\" (UID: \"e20058de-e3e6-40f3-830b-44c19095bb0a\") " pod="kserve-ci-e2e-test/isvc-secondary-a52b29-predictor-6bf9d85478-l5x8b"
Sep 17 18:30:20.771958 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:20.771940    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e20058de-e3e6-40f3-830b-44c19095bb0a-proxy-tls\") pod \"isvc-secondary-a52b29-predictor-6bf9d85478-l5x8b\" (UID: \"e20058de-e3e6-40f3-830b-44c19095bb0a\") " pod="kserve-ci-e2e-test/isvc-secondary-a52b29-predictor-6bf9d85478-l5x8b"
Sep 17 18:30:20.780274 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:20.780253    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk2vk\" (UniqueName: \"kubernetes.io/projected/e20058de-e3e6-40f3-830b-44c19095bb0a-kube-api-access-hk2vk\") pod \"isvc-secondary-a52b29-predictor-6bf9d85478-l5x8b\" (UID: \"e20058de-e3e6-40f3-830b-44c19095bb0a\") " pod="kserve-ci-e2e-test/isvc-secondary-a52b29-predictor-6bf9d85478-l5x8b"
Sep 17 18:30:20.886895 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:20.886851    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-a52b29-predictor-6bf9d85478-l5x8b"
Sep 17 18:30:21.161276 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:21.161244    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-a52b29-predictor-6bf9d85478-l5x8b"]
Sep 17 18:30:21.163505 ip-10-0-133-139 kubenswrapper[2585]: W0917 18:30:21.163463    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode20058de_e3e6_40f3_830b_44c19095bb0a.slice/crio-e84c949530ffeae8e29dc7d0d00ea438adb51827d04ec65458dd01383a431dfe WatchSource:0}: Error finding container e84c949530ffeae8e29dc7d0d00ea438adb51827d04ec65458dd01383a431dfe: Status 404 returned error can't find the container with id e84c949530ffeae8e29dc7d0d00ea438adb51827d04ec65458dd01383a431dfe
Sep 17 18:30:21.469442 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:21.469352    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-a52b29-predictor-6bf9d85478-l5x8b" event={"ID":"e20058de-e3e6-40f3-830b-44c19095bb0a","Type":"ContainerStarted","Data":"2571f90ce759d5fb59a43270a46e625ace797220d107829b68d1ef975828a143"}
Sep 17 18:30:21.469442 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:21.469415    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-a52b29-predictor-6bf9d85478-l5x8b" event={"ID":"e20058de-e3e6-40f3-830b-44c19095bb0a","Type":"ContainerStarted","Data":"e84c949530ffeae8e29dc7d0d00ea438adb51827d04ec65458dd01383a431dfe"}
Sep 17 18:30:23.478269 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:23.478244    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a52b29-predictor-6bf9d85478-l5x8b_e20058de-e3e6-40f3-830b-44c19095bb0a/storage-initializer/0.log"
Sep 17 18:30:23.478727 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:23.478280    2585 generic.go:356] "Generic (PLEG): container finished" podID="e20058de-e3e6-40f3-830b-44c19095bb0a" containerID="2571f90ce759d5fb59a43270a46e625ace797220d107829b68d1ef975828a143" exitCode=1
Sep 17 18:30:23.478727 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:23.478336    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-a52b29-predictor-6bf9d85478-l5x8b" event={"ID":"e20058de-e3e6-40f3-830b-44c19095bb0a","Type":"ContainerDied","Data":"2571f90ce759d5fb59a43270a46e625ace797220d107829b68d1ef975828a143"}
Sep 17 18:30:24.484186 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:24.484155    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a52b29-predictor-6bf9d85478-l5x8b_e20058de-e3e6-40f3-830b-44c19095bb0a/storage-initializer/0.log"
Sep 17 18:30:24.484580 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:24.484271    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-a52b29-predictor-6bf9d85478-l5x8b" event={"ID":"e20058de-e3e6-40f3-830b-44c19095bb0a","Type":"ContainerStarted","Data":"b4c30425af71a1dfd2165b0d7e7791ad85319e7189caa5d79f0ae5959c88fbf3"}
Sep 17 18:30:29.504057 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:29.504033    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a52b29-predictor-6bf9d85478-l5x8b_e20058de-e3e6-40f3-830b-44c19095bb0a/storage-initializer/1.log"
Sep 17 18:30:29.504529 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:29.504408    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a52b29-predictor-6bf9d85478-l5x8b_e20058de-e3e6-40f3-830b-44c19095bb0a/storage-initializer/0.log"
Sep 17 18:30:29.504529 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:29.504441    2585 generic.go:356] "Generic (PLEG): container finished" podID="e20058de-e3e6-40f3-830b-44c19095bb0a" containerID="b4c30425af71a1dfd2165b0d7e7791ad85319e7189caa5d79f0ae5959c88fbf3" exitCode=1
Sep 17 18:30:29.504648 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:29.504523    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-a52b29-predictor-6bf9d85478-l5x8b" event={"ID":"e20058de-e3e6-40f3-830b-44c19095bb0a","Type":"ContainerDied","Data":"b4c30425af71a1dfd2165b0d7e7791ad85319e7189caa5d79f0ae5959c88fbf3"}
Sep 17 18:30:29.504648 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:29.504607    2585 scope.go:117] "RemoveContainer" containerID="2571f90ce759d5fb59a43270a46e625ace797220d107829b68d1ef975828a143"
Sep 17 18:30:29.505012 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:29.504993    2585 scope.go:117] "RemoveContainer" containerID="2571f90ce759d5fb59a43270a46e625ace797220d107829b68d1ef975828a143"
Sep 17 18:30:29.515365 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:30:29.515342    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-a52b29-predictor-6bf9d85478-l5x8b_kserve-ci-e2e-test(e20058de-e3e6-40f3-830b-44c19095bb0a)\"" pod="kserve-ci-e2e-test/isvc-secondary-a52b29-predictor-6bf9d85478-l5x8b" podUID="e20058de-e3e6-40f3-830b-44c19095bb0a"
Sep 17 18:30:29.515643 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:30:29.515613    2585 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-a52b29-predictor-6bf9d85478-l5x8b_kserve-ci-e2e-test_e20058de-e3e6-40f3-830b-44c19095bb0a_0 in pod sandbox e84c949530ffeae8e29dc7d0d00ea438adb51827d04ec65458dd01383a431dfe from index: no such id: '2571f90ce759d5fb59a43270a46e625ace797220d107829b68d1ef975828a143'" containerID="2571f90ce759d5fb59a43270a46e625ace797220d107829b68d1ef975828a143"
Sep 17 18:30:29.515729 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:29.515654    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2571f90ce759d5fb59a43270a46e625ace797220d107829b68d1ef975828a143"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-a52b29-predictor-6bf9d85478-l5x8b_kserve-ci-e2e-test_e20058de-e3e6-40f3-830b-44c19095bb0a_0 in pod sandbox e84c949530ffeae8e29dc7d0d00ea438adb51827d04ec65458dd01383a431dfe from index: no such id: '2571f90ce759d5fb59a43270a46e625ace797220d107829b68d1ef975828a143'"
Sep 17 18:30:30.509689 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:30.509662    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a52b29-predictor-6bf9d85478-l5x8b_e20058de-e3e6-40f3-830b-44c19095bb0a/storage-initializer/1.log"
Sep 17 18:30:34.636689 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:34.636654    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-a52b29-predictor-6bf9d85478-l5x8b"]
Sep 17 18:30:34.651522 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:34.651474    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:34.657486 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:34.657447    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2"]
Sep 17 18:30:34.657782 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:34.657747    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2" podUID="0a40eca7-1139-45a0-80e6-1f6e054be941" containerName="kserve-container" containerID="cri-o://f160b14e12aa6f5a4cb653d604257e172a39d4b0abca647296fec74ca93f697f" gracePeriod=30
Sep 17 18:30:34.657894 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:34.657803    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2" podUID="0a40eca7-1139-45a0-80e6-1f6e054be941" containerName="kube-rbac-proxy" containerID="cri-o://49d7ecce11f6c2e80811a708ff37895101a4e70a0d23eaee2e94381b4efb9307" gracePeriod=30
Sep 17 18:30:34.752579 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:34.752543    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:34.775318 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:34.775288    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a52b29-predictor-6bf9d85478-l5x8b_e20058de-e3e6-40f3-830b-44c19095bb0a/storage-initializer/1.log"
Sep 17 18:30:34.777411 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:34.777393    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-a52b29-predictor-6bf9d85478-l5x8b"
Sep 17 18:30:34.881055 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:34.881031    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e20058de-e3e6-40f3-830b-44c19095bb0a-kserve-provision-location\") pod \"e20058de-e3e6-40f3-830b-44c19095bb0a\" (UID: \"e20058de-e3e6-40f3-830b-44c19095bb0a\") "
Sep 17 18:30:34.881239 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:34.881062    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e20058de-e3e6-40f3-830b-44c19095bb0a-cabundle-cert\") pod \"e20058de-e3e6-40f3-830b-44c19095bb0a\" (UID: \"e20058de-e3e6-40f3-830b-44c19095bb0a\") "
Sep 17 18:30:34.881239 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:34.881093    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e20058de-e3e6-40f3-830b-44c19095bb0a-proxy-tls\") pod \"e20058de-e3e6-40f3-830b-44c19095bb0a\" (UID: \"e20058de-e3e6-40f3-830b-44c19095bb0a\") "
Sep 17 18:30:34.881239 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:34.881152    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hk2vk\" (UniqueName: \"kubernetes.io/projected/e20058de-e3e6-40f3-830b-44c19095bb0a-kube-api-access-hk2vk\") pod \"e20058de-e3e6-40f3-830b-44c19095bb0a\" (UID: \"e20058de-e3e6-40f3-830b-44c19095bb0a\") "
Sep 17 18:30:34.881239 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:34.881181    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e20058de-e3e6-40f3-830b-44c19095bb0a-kube-rbac-proxy-sar-config\") pod \"e20058de-e3e6-40f3-830b-44c19095bb0a\" (UID: \"e20058de-e3e6-40f3-830b-44c19095bb0a\") "
Sep 17 18:30:34.881468 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:34.881323    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e20058de-e3e6-40f3-830b-44c19095bb0a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e20058de-e3e6-40f3-830b-44c19095bb0a" (UID: "e20058de-e3e6-40f3-830b-44c19095bb0a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:30:34.881468 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:34.881397    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e20058de-e3e6-40f3-830b-44c19095bb0a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e20058de-e3e6-40f3-830b-44c19095bb0a" (UID: "e20058de-e3e6-40f3-830b-44c19095bb0a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:30:34.881619 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:34.881591    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e20058de-e3e6-40f3-830b-44c19095bb0a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e20058de-e3e6-40f3-830b-44c19095bb0a" (UID: "e20058de-e3e6-40f3-830b-44c19095bb0a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:30:34.883271 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:34.883244    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e20058de-e3e6-40f3-830b-44c19095bb0a-kube-api-access-hk2vk" (OuterVolumeSpecName: "kube-api-access-hk2vk") pod "e20058de-e3e6-40f3-830b-44c19095bb0a" (UID: "e20058de-e3e6-40f3-830b-44c19095bb0a"). InnerVolumeSpecName "kube-api-access-hk2vk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:30:34.883386 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:34.883266    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e20058de-e3e6-40f3-830b-44c19095bb0a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e20058de-e3e6-40f3-830b-44c19095bb0a" (UID: "e20058de-e3e6-40f3-830b-44c19095bb0a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:30:34.981735 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:34.981652    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hk2vk\" (UniqueName: \"kubernetes.io/projected/e20058de-e3e6-40f3-830b-44c19095bb0a-kube-api-access-hk2vk\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:30:34.981735 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:34.981683    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e20058de-e3e6-40f3-830b-44c19095bb0a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:30:34.981735 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:34.981693    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e20058de-e3e6-40f3-830b-44c19095bb0a-kserve-provision-location\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:30:34.981735 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:34.981702    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e20058de-e3e6-40f3-830b-44c19095bb0a-cabundle-cert\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:30:34.981735 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:34.981712    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e20058de-e3e6-40f3-830b-44c19095bb0a-proxy-tls\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:30:35.530980 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:35.530943    2585 generic.go:356] "Generic (PLEG): container finished" podID="0a40eca7-1139-45a0-80e6-1f6e054be941" containerID="49d7ecce11f6c2e80811a708ff37895101a4e70a0d23eaee2e94381b4efb9307" exitCode=2
Sep 17 18:30:35.531167 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:35.531016    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2" event={"ID":"0a40eca7-1139-45a0-80e6-1f6e054be941","Type":"ContainerDied","Data":"49d7ecce11f6c2e80811a708ff37895101a4e70a0d23eaee2e94381b4efb9307"}
Sep 17 18:30:35.532125 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:35.532111    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-a52b29-predictor-6bf9d85478-l5x8b_e20058de-e3e6-40f3-830b-44c19095bb0a/storage-initializer/1.log"
Sep 17 18:30:35.532199 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:35.532173    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-a52b29-predictor-6bf9d85478-l5x8b" event={"ID":"e20058de-e3e6-40f3-830b-44c19095bb0a","Type":"ContainerDied","Data":"e84c949530ffeae8e29dc7d0d00ea438adb51827d04ec65458dd01383a431dfe"}
Sep 17 18:30:35.532237 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:35.532207    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-a52b29-predictor-6bf9d85478-l5x8b"
Sep 17 18:30:35.532237 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:35.532232    2585 scope.go:117] "RemoveContainer" containerID="b4c30425af71a1dfd2165b0d7e7791ad85319e7189caa5d79f0ae5959c88fbf3"
Sep 17 18:30:35.571065 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:35.571039    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-a52b29-predictor-6bf9d85478-l5x8b"]
Sep 17 18:30:35.575980 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:35.575958    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-a52b29-predictor-6bf9d85478-l5x8b"]
Sep 17 18:30:36.109390 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:36.109358    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e20058de-e3e6-40f3-830b-44c19095bb0a" path="/var/lib/kubelet/pods/e20058de-e3e6-40f3-830b-44c19095bb0a/volumes"
Sep 17 18:30:37.118094 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:37.118048    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2" podUID="0a40eca7-1139-45a0-80e6-1f6e054be941" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.53:8643/healthz\": dial tcp 10.133.0.53:8643: connect: connection refused"
Sep 17 18:30:37.122399 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:37.122376    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2" podUID="0a40eca7-1139-45a0-80e6-1f6e054be941" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 17 18:30:39.206792 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:39.206768    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2"
Sep 17 18:30:39.315371 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:39.315305    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qkpq8\" (UniqueName: \"kubernetes.io/projected/0a40eca7-1139-45a0-80e6-1f6e054be941-kube-api-access-qkpq8\") pod \"0a40eca7-1139-45a0-80e6-1f6e054be941\" (UID: \"0a40eca7-1139-45a0-80e6-1f6e054be941\") "
Sep 17 18:30:39.315371 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:39.315358    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0a40eca7-1139-45a0-80e6-1f6e054be941-kube-rbac-proxy-sar-config\") pod \"0a40eca7-1139-45a0-80e6-1f6e054be941\" (UID: \"0a40eca7-1139-45a0-80e6-1f6e054be941\") "
Sep 17 18:30:39.315551 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:39.315531    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a40eca7-1139-45a0-80e6-1f6e054be941-proxy-tls\") pod \"0a40eca7-1139-45a0-80e6-1f6e054be941\" (UID: \"0a40eca7-1139-45a0-80e6-1f6e054be941\") "
Sep 17 18:30:39.315603 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:39.315585    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0a40eca7-1139-45a0-80e6-1f6e054be941-cabundle-cert\") pod \"0a40eca7-1139-45a0-80e6-1f6e054be941\" (UID: \"0a40eca7-1139-45a0-80e6-1f6e054be941\") "
Sep 17 18:30:39.315653 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:39.315617    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a40eca7-1139-45a0-80e6-1f6e054be941-kserve-provision-location\") pod \"0a40eca7-1139-45a0-80e6-1f6e054be941\" (UID: \"0a40eca7-1139-45a0-80e6-1f6e054be941\") "
Sep 17 18:30:39.315722 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:39.315699    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a40eca7-1139-45a0-80e6-1f6e054be941-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0a40eca7-1139-45a0-80e6-1f6e054be941" (UID: "0a40eca7-1139-45a0-80e6-1f6e054be941"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:30:39.315919 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:39.315813    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0a40eca7-1139-45a0-80e6-1f6e054be941-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:30:39.316010 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:39.315948    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a40eca7-1139-45a0-80e6-1f6e054be941-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0a40eca7-1139-45a0-80e6-1f6e054be941" (UID: "0a40eca7-1139-45a0-80e6-1f6e054be941"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:30:39.316010 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:39.315979    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a40eca7-1139-45a0-80e6-1f6e054be941-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0a40eca7-1139-45a0-80e6-1f6e054be941" (UID: "0a40eca7-1139-45a0-80e6-1f6e054be941"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:30:39.317434 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:39.317401    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a40eca7-1139-45a0-80e6-1f6e054be941-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0a40eca7-1139-45a0-80e6-1f6e054be941" (UID: "0a40eca7-1139-45a0-80e6-1f6e054be941"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:30:39.317540 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:39.317443    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a40eca7-1139-45a0-80e6-1f6e054be941-kube-api-access-qkpq8" (OuterVolumeSpecName: "kube-api-access-qkpq8") pod "0a40eca7-1139-45a0-80e6-1f6e054be941" (UID: "0a40eca7-1139-45a0-80e6-1f6e054be941"). InnerVolumeSpecName "kube-api-access-qkpq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:30:39.416814 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:39.416786    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a40eca7-1139-45a0-80e6-1f6e054be941-proxy-tls\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:30:39.416814 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:39.416809    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0a40eca7-1139-45a0-80e6-1f6e054be941-cabundle-cert\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:30:39.416953 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:39.416822    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a40eca7-1139-45a0-80e6-1f6e054be941-kserve-provision-location\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:30:39.416953 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:39.416835    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkpq8\" (UniqueName: \"kubernetes.io/projected/0a40eca7-1139-45a0-80e6-1f6e054be941-kube-api-access-qkpq8\") on node \"ip-10-0-133-139.ec2.internal\" DevicePath \"\""
Sep 17 18:30:39.549687 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:39.549656    2585 generic.go:356] "Generic (PLEG): container finished" podID="0a40eca7-1139-45a0-80e6-1f6e054be941" containerID="f160b14e12aa6f5a4cb653d604257e172a39d4b0abca647296fec74ca93f697f" exitCode=0
Sep 17 18:30:39.549805 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:39.549712    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2"
Sep 17 18:30:39.549805 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:39.549734    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2" event={"ID":"0a40eca7-1139-45a0-80e6-1f6e054be941","Type":"ContainerDied","Data":"f160b14e12aa6f5a4cb653d604257e172a39d4b0abca647296fec74ca93f697f"}
Sep 17 18:30:39.549805 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:39.549781    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2" event={"ID":"0a40eca7-1139-45a0-80e6-1f6e054be941","Type":"ContainerDied","Data":"7d04159788aa68d7809508f67ac699a97964aa7759fb7a4121b70e5fc723c2d3"}
Sep 17 18:30:39.549913 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:39.549806    2585 scope.go:117] "RemoveContainer" containerID="49d7ecce11f6c2e80811a708ff37895101a4e70a0d23eaee2e94381b4efb9307"
Sep 17 18:30:39.558432 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:39.558409    2585 scope.go:117] "RemoveContainer" containerID="f160b14e12aa6f5a4cb653d604257e172a39d4b0abca647296fec74ca93f697f"
Sep 17 18:30:39.565544 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:39.565522    2585 scope.go:117] "RemoveContainer" containerID="af32eab28a78e53a055555a135b4f729a313c03e50d5e584ce439c04adddf574"
Sep 17 18:30:39.572317 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:39.572276    2585 scope.go:117] "RemoveContainer" containerID="49d7ecce11f6c2e80811a708ff37895101a4e70a0d23eaee2e94381b4efb9307"
Sep 17 18:30:39.572554 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:30:39.572535    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49d7ecce11f6c2e80811a708ff37895101a4e70a0d23eaee2e94381b4efb9307\": container with ID starting with 49d7ecce11f6c2e80811a708ff37895101a4e70a0d23eaee2e94381b4efb9307 not found: ID does not exist" containerID="49d7ecce11f6c2e80811a708ff37895101a4e70a0d23eaee2e94381b4efb9307"
Sep 17 18:30:39.572640 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:39.572561    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49d7ecce11f6c2e80811a708ff37895101a4e70a0d23eaee2e94381b4efb9307"} err="failed to get container status \"49d7ecce11f6c2e80811a708ff37895101a4e70a0d23eaee2e94381b4efb9307\": rpc error: code = NotFound desc = could not find container \"49d7ecce11f6c2e80811a708ff37895101a4e70a0d23eaee2e94381b4efb9307\": container with ID starting with 49d7ecce11f6c2e80811a708ff37895101a4e70a0d23eaee2e94381b4efb9307 not found: ID does not exist"
Sep 17 18:30:39.572640 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:39.572577    2585 scope.go:117] "RemoveContainer" containerID="f160b14e12aa6f5a4cb653d604257e172a39d4b0abca647296fec74ca93f697f"
Sep 17 18:30:39.572802 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:30:39.572787    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f160b14e12aa6f5a4cb653d604257e172a39d4b0abca647296fec74ca93f697f\": container with ID starting with f160b14e12aa6f5a4cb653d604257e172a39d4b0abca647296fec74ca93f697f not found: ID does not exist" containerID="f160b14e12aa6f5a4cb653d604257e172a39d4b0abca647296fec74ca93f697f"
Sep 17 18:30:39.572849 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:39.572807    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f160b14e12aa6f5a4cb653d604257e172a39d4b0abca647296fec74ca93f697f"} err="failed to get container status \"f160b14e12aa6f5a4cb653d604257e172a39d4b0abca647296fec74ca93f697f\": rpc error: code = NotFound desc = could not find container \"f160b14e12aa6f5a4cb653d604257e172a39d4b0abca647296fec74ca93f697f\": container with ID starting with f160b14e12aa6f5a4cb653d604257e172a39d4b0abca647296fec74ca93f697f not found: ID does not exist"
Sep 17 18:30:39.572849 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:39.572819    2585 scope.go:117] "RemoveContainer" containerID="af32eab28a78e53a055555a135b4f729a313c03e50d5e584ce439c04adddf574"
Sep 17 18:30:39.572990 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:39.572975    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2"]
Sep 17 18:30:39.573031 ip-10-0-133-139 kubenswrapper[2585]: E0917 18:30:39.573003    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af32eab28a78e53a055555a135b4f729a313c03e50d5e584ce439c04adddf574\": container with ID starting with af32eab28a78e53a055555a135b4f729a313c03e50d5e584ce439c04adddf574 not found: ID does not exist" containerID="af32eab28a78e53a055555a135b4f729a313c03e50d5e584ce439c04adddf574"
Sep 17 18:30:39.573066 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:39.573025    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af32eab28a78e53a055555a135b4f729a313c03e50d5e584ce439c04adddf574"} err="failed to get container status \"af32eab28a78e53a055555a135b4f729a313c03e50d5e584ce439c04adddf574\": rpc error: code = NotFound desc = could not find container \"af32eab28a78e53a055555a135b4f729a313c03e50d5e584ce439c04adddf574\": container with ID starting with af32eab28a78e53a055555a135b4f729a313c03e50d5e584ce439c04adddf574 not found: ID does not exist"
Sep 17 18:30:39.576481 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:39.576460    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-a52b29-predictor-6d78fb99f7-dsrc2"]
Sep 17 18:30:40.103282 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:40.103252    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a40eca7-1139-45a0-80e6-1f6e054be941" path="/var/lib/kubelet/pods/0a40eca7-1139-45a0-80e6-1f6e054be941/volumes"
Sep 17 18:30:40.611385 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:40.611355    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:40.706108 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:30:40.706064    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:18.735147 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:18.735064    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:18.836376 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:18.836335    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:20.919697 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:20.919667    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-7spmx_dbb6518b-5a9b-4a33-89a1-71c9e4db70a5/global-pull-secret-syncer/0.log"
Sep 17 18:31:21.088114 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:21.088081    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log"
Sep 17 18:31:21.182544 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:21.182455    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-139.ec2.internal_aba6d40ed12a7528810e76c18b9ddea9/haproxy/0.log"
Sep 17 18:31:23.880723 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:23.880687    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:23.980337 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:23.980301    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jpg4z_856535bd-6884-4e98-8089-852deae228aa/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:25.772116 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:25.772082    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wp9c7_e29f50f9-0bcf-4660-8348-914273f06fc5/node-exporter/0.log"
Sep 17 18:31:25.796445 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:25.796423    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wp9c7_e29f50f9-0bcf-4660-8348-914273f06fc5/kube-rbac-proxy/0.log"
Sep 17 18:31:25.822869 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:25.822847    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wp9c7_e29f50f9-0bcf-4660-8348-914273f06fc5/init-textfile/0.log"
Sep 17 18:31:29.540702 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:29.540663    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5s492/perf-node-gather-daemonset-jpwhl"]
Sep 17 18:31:29.541079 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:29.540890    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e20058de-e3e6-40f3-830b-44c19095bb0a" containerName="storage-initializer"
Sep 17 18:31:29.541079 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:29.540900    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e20058de-e3e6-40f3-830b-44c19095bb0a" containerName="storage-initializer"
Sep 17 18:31:29.541079 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:29.540914    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a40eca7-1139-45a0-80e6-1f6e054be941" containerName="kube-rbac-proxy"
Sep 17 18:31:29.541079 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:29.540920    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a40eca7-1139-45a0-80e6-1f6e054be941" containerName="kube-rbac-proxy"
Sep 17 18:31:29.541079 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:29.540930    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e20058de-e3e6-40f3-830b-44c19095bb0a" containerName="storage-initializer"
Sep 17 18:31:29.541079 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:29.540935    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e20058de-e3e6-40f3-830b-44c19095bb0a" containerName="storage-initializer"
Sep 17 18:31:29.541079 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:29.540943    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a40eca7-1139-45a0-80e6-1f6e054be941" containerName="storage-initializer"
Sep 17 18:31:29.541079 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:29.540948    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a40eca7-1139-45a0-80e6-1f6e054be941" containerName="storage-initializer"
Sep 17 18:31:29.541079 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:29.540956    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a40eca7-1139-45a0-80e6-1f6e054be941" containerName="kserve-container"
Sep 17 18:31:29.541079 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:29.540962    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a40eca7-1139-45a0-80e6-1f6e054be941" containerName="kserve-container"
Sep 17 18:31:29.541079 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:29.540999    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="e20058de-e3e6-40f3-830b-44c19095bb0a" containerName="storage-initializer"
Sep 17 18:31:29.541079 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:29.541007    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="e20058de-e3e6-40f3-830b-44c19095bb0a" containerName="storage-initializer"
Sep 17 18:31:29.541079 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:29.541013    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a40eca7-1139-45a0-80e6-1f6e054be941" containerName="kserve-container"
Sep 17 18:31:29.541079 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:29.541020    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a40eca7-1139-45a0-80e6-1f6e054be941" containerName="kube-rbac-proxy"
Sep 17 18:31:29.543798 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:29.543780    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5s492/perf-node-gather-daemonset-jpwhl"
Sep 17 18:31:29.546378 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:29.546356    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5s492\"/\"default-dockercfg-lrg5v\""
Sep 17 18:31:29.547309 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:29.547293    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5s492\"/\"kube-root-ca.crt\""
Sep 17 18:31:29.547384 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:29.547313    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5s492\"/\"openshift-service-ca.crt\""
Sep 17 18:31:29.553349 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:29.553331    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5s492/perf-node-gather-daemonset-jpwhl"]
Sep 17 18:31:29.654845 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:29.654820    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b792cdf0-3b33-48e0-ad01-d4f2ce2d5645-proc\") pod \"perf-node-gather-daemonset-jpwhl\" (UID: \"b792cdf0-3b33-48e0-ad01-d4f2ce2d5645\") " pod="openshift-must-gather-5s492/perf-node-gather-daemonset-jpwhl"
Sep 17 18:31:29.654992 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:29.654864    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b792cdf0-3b33-48e0-ad01-d4f2ce2d5645-lib-modules\") pod \"perf-node-gather-daemonset-jpwhl\" (UID: \"b792cdf0-3b33-48e0-ad01-d4f2ce2d5645\") " pod="openshift-must-gather-5s492/perf-node-gather-daemonset-jpwhl"
Sep 17 18:31:29.654992 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:29.654880    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b792cdf0-3b33-48e0-ad01-d4f2ce2d5645-podres\") pod \"perf-node-gather-daemonset-jpwhl\" (UID: \"b792cdf0-3b33-48e0-ad01-d4f2ce2d5645\") " pod="openshift-must-gather-5s492/perf-node-gather-daemonset-jpwhl"
Sep 17 18:31:29.654992 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:29.654931    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b792cdf0-3b33-48e0-ad01-d4f2ce2d5645-sys\") pod \"perf-node-gather-daemonset-jpwhl\" (UID: \"b792cdf0-3b33-48e0-ad01-d4f2ce2d5645\") " pod="openshift-must-gather-5s492/perf-node-gather-daemonset-jpwhl"
Sep 17 18:31:29.655106 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:29.655008    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgd7j\" (UniqueName: \"kubernetes.io/projected/b792cdf0-3b33-48e0-ad01-d4f2ce2d5645-kube-api-access-pgd7j\") pod \"perf-node-gather-daemonset-jpwhl\" (UID: \"b792cdf0-3b33-48e0-ad01-d4f2ce2d5645\") " pod="openshift-must-gather-5s492/perf-node-gather-daemonset-jpwhl"
Sep 17 18:31:29.756212 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:29.756187    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b792cdf0-3b33-48e0-ad01-d4f2ce2d5645-lib-modules\") pod \"perf-node-gather-daemonset-jpwhl\" (UID: \"b792cdf0-3b33-48e0-ad01-d4f2ce2d5645\") " pod="openshift-must-gather-5s492/perf-node-gather-daemonset-jpwhl"
Sep 17 18:31:29.756381 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:29.756217    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b792cdf0-3b33-48e0-ad01-d4f2ce2d5645-podres\") pod \"perf-node-gather-daemonset-jpwhl\" (UID: \"b792cdf0-3b33-48e0-ad01-d4f2ce2d5645\") " pod="openshift-must-gather-5s492/perf-node-gather-daemonset-jpwhl"
Sep 17 18:31:29.756381 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:29.756232    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b792cdf0-3b33-48e0-ad01-d4f2ce2d5645-sys\") pod \"perf-node-gather-daemonset-jpwhl\" (UID: \"b792cdf0-3b33-48e0-ad01-d4f2ce2d5645\") " pod="openshift-must-gather-5s492/perf-node-gather-daemonset-jpwhl"
Sep 17 18:31:29.756381 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:29.756271    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgd7j\" (UniqueName: \"kubernetes.io/projected/b792cdf0-3b33-48e0-ad01-d4f2ce2d5645-kube-api-access-pgd7j\") pod \"perf-node-gather-daemonset-jpwhl\" (UID: \"b792cdf0-3b33-48e0-ad01-d4f2ce2d5645\") " pod="openshift-must-gather-5s492/perf-node-gather-daemonset-jpwhl"
Sep 17 18:31:29.756381 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:29.756306    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b792cdf0-3b33-48e0-ad01-d4f2ce2d5645-proc\") pod \"perf-node-gather-daemonset-jpwhl\" (UID: \"b792cdf0-3b33-48e0-ad01-d4f2ce2d5645\") " pod="openshift-must-gather-5s492/perf-node-gather-daemonset-jpwhl"
Sep 17 18:31:29.756381 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:29.756339    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b792cdf0-3b33-48e0-ad01-d4f2ce2d5645-sys\") pod \"perf-node-gather-daemonset-jpwhl\" (UID: \"b792cdf0-3b33-48e0-ad01-d4f2ce2d5645\") " pod="openshift-must-gather-5s492/perf-node-gather-daemonset-jpwhl"
Sep 17 18:31:29.756381 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:29.756373    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b792cdf0-3b33-48e0-ad01-d4f2ce2d5645-lib-modules\") pod \"perf-node-gather-daemonset-jpwhl\" (UID: \"b792cdf0-3b33-48e0-ad01-d4f2ce2d5645\") " pod="openshift-must-gather-5s492/perf-node-gather-daemonset-jpwhl"
Sep 17 18:31:29.756650 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:29.756384    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b792cdf0-3b33-48e0-ad01-d4f2ce2d5645-proc\") pod \"perf-node-gather-daemonset-jpwhl\" (UID: \"b792cdf0-3b33-48e0-ad01-d4f2ce2d5645\") " pod="openshift-must-gather-5s492/perf-node-gather-daemonset-jpwhl"
Sep 17 18:31:29.756650 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:29.756373    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b792cdf0-3b33-48e0-ad01-d4f2ce2d5645-podres\") pod \"perf-node-gather-daemonset-jpwhl\" (UID: \"b792cdf0-3b33-48e0-ad01-d4f2ce2d5645\") " pod="openshift-must-gather-5s492/perf-node-gather-daemonset-jpwhl"
Sep 17 18:31:29.765164 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:29.765146    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgd7j\" (UniqueName: \"kubernetes.io/projected/b792cdf0-3b33-48e0-ad01-d4f2ce2d5645-kube-api-access-pgd7j\") pod \"perf-node-gather-daemonset-jpwhl\" (UID: \"b792cdf0-3b33-48e0-ad01-d4f2ce2d5645\") " pod="openshift-must-gather-5s492/perf-node-gather-daemonset-jpwhl"
Sep 17 18:31:29.853270 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:29.853201    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5s492/perf-node-gather-daemonset-jpwhl"
Sep 17 18:31:30.121505 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:30.121468    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5s492/perf-node-gather-daemonset-jpwhl"]
Sep 17 18:31:30.123635 ip-10-0-133-139 kubenswrapper[2585]: W0917 18:31:30.123607    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb792cdf0_3b33_48e0_ad01_d4f2ce2d5645.slice/crio-f33534ad288b30d95b549f368ca78de8e895fcbbf24f3cd9b2faeec3f29293af WatchSource:0}: Error finding container f33534ad288b30d95b549f368ca78de8e895fcbbf24f3cd9b2faeec3f29293af: Status 404 returned error can't find the container with id f33534ad288b30d95b549f368ca78de8e895fcbbf24f3cd9b2faeec3f29293af
Sep 17 18:31:30.641605 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:30.641580    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-t7fvf_3973eda4-9544-4956-980a-4c11ba7700c1/dns/0.log"
Sep 17 18:31:30.669029 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:30.669006    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-t7fvf_3973eda4-9544-4956-980a-4c11ba7700c1/kube-rbac-proxy/0.log"
Sep 17 18:31:30.723221 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:30.723194    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5s492/perf-node-gather-daemonset-jpwhl" event={"ID":"b792cdf0-3b33-48e0-ad01-d4f2ce2d5645","Type":"ContainerStarted","Data":"778ee814c9f33aa60cdf14f9b3ec2f1dc783d5a6554a209092eec842145965a0"}
Sep 17 18:31:30.723378 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:30.723240    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5s492/perf-node-gather-daemonset-jpwhl" event={"ID":"b792cdf0-3b33-48e0-ad01-d4f2ce2d5645","Type":"ContainerStarted","Data":"f33534ad288b30d95b549f368ca78de8e895fcbbf24f3cd9b2faeec3f29293af"}
Sep 17 18:31:30.723378 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:30.723271    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5s492/perf-node-gather-daemonset-jpwhl"
Sep 17 18:31:30.741563 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:30.741519    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5s492/perf-node-gather-daemonset-jpwhl" podStartSLOduration=1.741503784 podStartE2EDuration="1.741503784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:31:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:31:30.739717001 +0000 UTC m=+3427.153507439" watchObservedRunningTime="2026-09-17 18:31:30.741503784 +0000 UTC m=+3427.155294214"
Sep 17 18:31:30.756443 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:30.756419    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kgbmb_6c27507b-19a0-4c12-a379-10474f260f84/dns-node-resolver/0.log"
Sep 17 18:31:31.375466 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:31.375440    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mglss_e8bda048-e7cf-4118-82c4-bd1d987e4566/node-ca/0.log"
Sep 17 18:31:32.538713 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:32.538684    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jvtw5_98e53603-643c-4852-b805-5e07c9016190/serve-healthcheck-canary/0.log"
Sep 17 18:31:33.012999 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:33.012971    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hjpz9_7c0b81ec-0283-421e-ae34-032edea2eeb8/kube-rbac-proxy/0.log"
Sep 17 18:31:33.035383 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:33.035359    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hjpz9_7c0b81ec-0283-421e-ae34-032edea2eeb8/exporter/0.log"
Sep 17 18:31:33.059425 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:33.059404    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hjpz9_7c0b81ec-0283-421e-ae34-032edea2eeb8/extractor/0.log"
Sep 17 18:31:35.756008 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:35.755968    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-7f654675b-7vhmc_e8ed622f-3f48-4e28-af3b-fc634c8e52dd/manager/0.log"
Sep 17 18:31:36.736767 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:36.736739    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5s492/perf-node-gather-daemonset-jpwhl"
Sep 17 18:31:42.506688 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:42.506656    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9tfrw_751c4ff3-f343-4280-b970-3583c2a4c28b/kube-multus-additional-cni-plugins/0.log"
Sep 17 18:31:42.533088 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:42.533061    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9tfrw_751c4ff3-f343-4280-b970-3583c2a4c28b/egress-router-binary-copy/0.log"
Sep 17 18:31:42.555963 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:42.555937    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9tfrw_751c4ff3-f343-4280-b970-3583c2a4c28b/cni-plugins/0.log"
Sep 17 18:31:42.582067 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:42.582042    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9tfrw_751c4ff3-f343-4280-b970-3583c2a4c28b/bond-cni-plugin/0.log"
Sep 17 18:31:42.607393 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:42.607366    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9tfrw_751c4ff3-f343-4280-b970-3583c2a4c28b/routeoverride-cni/0.log"
Sep 17 18:31:42.637575 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:42.637553    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9tfrw_751c4ff3-f343-4280-b970-3583c2a4c28b/whereabouts-cni-bincopy/0.log"
Sep 17 18:31:42.670554 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:42.670524    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9tfrw_751c4ff3-f343-4280-b970-3583c2a4c28b/whereabouts-cni/0.log"
Sep 17 18:31:42.940229 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:42.940156    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-d972p_9ef20182-de84-4cba-9eb4-1fcfdfbd1abe/kube-multus/0.log"
Sep 17 18:31:43.093061 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:43.093031    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-p7vh6_39af04b6-713b-406b-952d-7b2ef2b2b952/network-metrics-daemon/0.log"
Sep 17 18:31:43.114431 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:43.114408    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-p7vh6_39af04b6-713b-406b-952d-7b2ef2b2b952/kube-rbac-proxy/0.log"
Sep 17 18:31:44.462877 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:44.462853    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rrf79_beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5/ovn-controller/0.log"
Sep 17 18:31:44.481265 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:44.481240    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rrf79_beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5/ovn-acl-logging/1.log"
Sep 17 18:31:44.481406 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:44.481348    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rrf79_beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5/ovn-acl-logging/0.log"
Sep 17 18:31:44.505370 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:44.505348    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rrf79_beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5/kube-rbac-proxy-node/0.log"
Sep 17 18:31:44.529757 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:44.529737    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rrf79_beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 18:31:44.547999 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:44.547975    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rrf79_beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5/northd/0.log"
Sep 17 18:31:44.569556 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:44.569536    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rrf79_beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5/nbdb/0.log"
Sep 17 18:31:44.595413 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:44.595396    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rrf79_beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5/sbdb/0.log"
Sep 17 18:31:44.657537 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:44.657507    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rrf79_beff8ae2-1a86-49b3-a9bc-d9d2e1b99bf5/ovnkube-controller/0.log"
Sep 17 18:31:45.768628 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:45.768599    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-z8nx5_7bcabfd1-e72e-4866-9190-6bcce1f7ce64/network-check-target-container/0.log"
Sep 17 18:31:46.772423 ip-10-0-133-139 kubenswrapper[2585]: I0917 18:31:46.772385    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-lt9dx_a40b9f88-37de-405c-9702-3a73548b4f63/iptables-alerter/0.log"