Sep 17 15:51:45.794993 ip-10-0-135-43 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 17 15:51:45.795002 ip-10-0-135-43 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 17 15:51:45.795009 ip-10-0-135-43 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 17 15:51:45.795157 ip-10-0-135-43 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 17 15:51:55.912608 ip-10-0-135-43 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 17 15:51:55.912625 ip-10-0-135-43 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 9b1a432796494954aeede4daaf40025f --
Sep 17 15:54:02.802392 ip-10-0-135-43 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 15:54:06.274386 ip-10-0-135-43 kubenswrapper[2585]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 15:54:06.274386 ip-10-0-135-43 kubenswrapper[2585]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 15:54:06.274386 ip-10-0-135-43 kubenswrapper[2585]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 15:54:06.274386 ip-10-0-135-43 kubenswrapper[2585]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 15:54:06.274386 ip-10-0-135-43 kubenswrapper[2585]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 15:54:06.275259 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.274471    2585 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 15:54:06.278455 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278438    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:54:06.278455 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278453    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:54:06.278455 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278457    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:54:06.278553 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278460    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:54:06.278553 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278463    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:54:06.278553 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278466    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:54:06.278553 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278469    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:54:06.278553 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278471    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:54:06.278553 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278474    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:54:06.278553 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278477    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:54:06.278553 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278479    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:54:06.278553 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278482    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:54:06.278553 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278484    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:54:06.278553 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278488    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:54:06.278553 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278491    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:54:06.278553 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278493    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:54:06.278553 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278496    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:54:06.278553 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278499    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:54:06.278553 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278502    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:54:06.278553 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278505    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:54:06.278553 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278507    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:54:06.278553 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278510    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:54:06.278553 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278512    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:54:06.279038 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278515    2585 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:54:06.279038 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278518    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:54:06.279038 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278526    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:54:06.279038 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278529    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:54:06.279038 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278531    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:54:06.279038 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278534    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:54:06.279038 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278536    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:54:06.279038 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278539    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:54:06.279038 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278541    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:54:06.279038 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278544    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:54:06.279038 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278547    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:54:06.279038 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278550    2585 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:54:06.279038 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278552    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:54:06.279038 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278557    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:54:06.279038 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278560    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:54:06.279038 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278563    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:54:06.279038 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278565    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:54:06.279038 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278568    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:54:06.279038 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278571    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:54:06.279038 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278574    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:54:06.279570 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278578    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:54:06.279570 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278582    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:54:06.279570 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278585    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:54:06.279570 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278588    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:54:06.279570 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278590    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:54:06.279570 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278593    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:54:06.279570 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278596    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:54:06.279570 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278599    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:54:06.279570 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278601    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:54:06.279570 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278604    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:54:06.279570 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278607    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:54:06.279570 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278609    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:54:06.279570 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278612    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:54:06.279570 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278615    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:54:06.279570 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278617    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:54:06.279570 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278625    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:54:06.279570 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278628    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:54:06.279570 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278631    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:54:06.279570 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278634    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:54:06.279570 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278636    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:54:06.280060 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278641    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:54:06.280060 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278644    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:54:06.280060 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278647    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:54:06.280060 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278649    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:54:06.280060 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278652    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:54:06.280060 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278655    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:54:06.280060 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278658    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:54:06.280060 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278661    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:54:06.280060 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278664    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:54:06.280060 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278667    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:54:06.280060 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278670    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:54:06.280060 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278672    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:54:06.280060 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278676    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:54:06.280060 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278680    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:54:06.280060 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278683    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:54:06.280060 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278685    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:54:06.280060 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278689    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:54:06.280060 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278692    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:54:06.280510 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278695    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:54:06.280510 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278697    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:54:06.280510 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278700    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:54:06.280510 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278702    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:54:06.280510 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278705    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:54:06.280510 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278707    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:54:06.280510 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278710    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:54:06.280510 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278712    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:54:06.280510 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278715    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:54:06.280510 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278718    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:54:06.280510 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278721    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:54:06.280510 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278723    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:54:06.280510 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278726    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:54:06.280510 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278728    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:54:06.280510 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278731    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:54:06.280510 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278733    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:54:06.280510 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278736    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:54:06.280510 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.278739    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:54:06.280510 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279397    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:54:06.280510 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279404    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:54:06.281027 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279407    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:54:06.281027 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279410    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:54:06.281027 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279413    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:54:06.281027 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279430    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:54:06.281027 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279433    2585 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:54:06.281027 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279436    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:54:06.281027 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279439    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:54:06.281027 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279442    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:54:06.281027 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279444    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:54:06.281027 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279447    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:54:06.281027 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279450    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:54:06.281027 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279452    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:54:06.281027 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279455    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:54:06.281027 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279458    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:54:06.281027 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279460    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:54:06.281027 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279463    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:54:06.281027 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279466    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:54:06.281027 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279468    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:54:06.281027 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279471    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:54:06.281027 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279475    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:54:06.281554 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279479    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:54:06.281554 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279482    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:54:06.281554 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279485    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:54:06.281554 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279489    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:54:06.281554 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279491    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:54:06.281554 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279494    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:54:06.281554 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279497    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:54:06.281554 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279499    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:54:06.281554 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279502    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:54:06.281554 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279505    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:54:06.281554 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279507    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:54:06.281554 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279511    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:54:06.281554 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279514    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:54:06.281554 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279517    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:54:06.281554 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279520    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:54:06.281554 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279522    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:54:06.281554 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279525    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:54:06.281554 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279527    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:54:06.281554 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279530    2585 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:54:06.282038 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279533    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:54:06.282038 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279535    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:54:06.282038 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279538    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:54:06.282038 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279540    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:54:06.282038 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279543    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:54:06.282038 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279545    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:54:06.282038 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279549    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:54:06.282038 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279553    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:54:06.282038 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279556    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:54:06.282038 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279559    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:54:06.282038 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279561    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:54:06.282038 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279564    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:54:06.282038 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279568    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:54:06.282038 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279571    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:54:06.282038 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279574    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:54:06.282038 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279578    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:54:06.282038 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279581    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:54:06.282038 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279584    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:54:06.282038 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279587    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:54:06.282524 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279590    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:54:06.282524 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279592    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:54:06.282524 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279594    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:54:06.282524 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279597    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:54:06.282524 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279600    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:54:06.282524 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279603    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:54:06.282524 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279606    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:54:06.282524 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279608    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:54:06.282524 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279611    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:54:06.282524 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279613    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:54:06.282524 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279616    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:54:06.282524 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279619    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:54:06.282524 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279621    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:54:06.282524 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279624    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:54:06.282524 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279626    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:54:06.282524 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279629    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:54:06.282524 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279632    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:54:06.282524 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279634    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:54:06.282524 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279637    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:54:06.282524 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279639    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:54:06.283021 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279642    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:54:06.283021 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279645    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:54:06.283021 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279647    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:54:06.283021 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279650    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:54:06.283021 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279652    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:54:06.283021 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279655    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:54:06.283021 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279657    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:54:06.283021 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279659    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:54:06.283021 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279662    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:54:06.283021 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279664    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:54:06.283021 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279667    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:54:06.283021 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279669    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:54:06.283021 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279672    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:54:06.283021 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279675    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:54:06.283021 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279677    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:54:06.283021 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279684    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:54:06.283021 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279686    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:54:06.283021 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279689    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:54:06.283021 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.279692    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:54:06.283021 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280041    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:54:06.283526 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280045    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:54:06.283526 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280048    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:54:06.283526 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280050    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:54:06.283526 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280053    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:54:06.283526 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280056    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:54:06.283526 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280059    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:54:06.283526 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280062    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:54:06.283526 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280064    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:54:06.283526 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280067    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:54:06.283526 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280070    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:54:06.283526 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280073    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:54:06.283526 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280076    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:54:06.283526 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280078    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:54:06.283526 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280081    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:54:06.283526 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280084    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:54:06.283526 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280087    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:54:06.283526 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280089    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:54:06.283526 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280092    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:54:06.283526 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280095    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:54:06.283982 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280097    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:54:06.283982 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280100    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:54:06.283982 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280103    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:54:06.283982 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280107    2585 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:54:06.283982 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280109    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:54:06.283982 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280112    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:54:06.283982 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280114    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:54:06.283982 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280117    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:54:06.283982 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280123    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:54:06.283982 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280127    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:54:06.283982 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280131    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:54:06.283982 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280134    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:54:06.283982 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280137    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:54:06.283982 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280139    2585 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:54:06.283982 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280142    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:54:06.283982 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280145    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:54:06.283982 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280147    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:54:06.283982 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280150    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:54:06.283982 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280152    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:54:06.284498 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280156    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:54:06.284498 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280159    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:54:06.284498 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280161    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:54:06.284498 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280164    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:54:06.284498 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280166    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:54:06.284498 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280169    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:54:06.284498 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280171    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:54:06.284498 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280174    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:54:06.284498 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280176    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:54:06.284498 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280179    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:54:06.284498 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280183    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:54:06.284498 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280186    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:54:06.284498 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280189    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:54:06.284498 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280192    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:54:06.284498 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280194    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:54:06.284498 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280197    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:54:06.284498 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280200    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:54:06.284498 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280202    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:54:06.284498 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280205    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:54:06.284962 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280208    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:54:06.284962 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280210    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:54:06.284962 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280214    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:54:06.284962 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280217    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:54:06.284962 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280222    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:54:06.284962 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280224    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:54:06.284962 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280227    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:54:06.284962 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280230    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:54:06.284962 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280232    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:54:06.284962 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280235    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:54:06.284962 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280237    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:54:06.284962 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280240    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:54:06.284962 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280242    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:54:06.284962 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280245    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:54:06.284962 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280248    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:54:06.284962 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280251    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:54:06.284962 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280253    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:54:06.284962 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280256    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:54:06.284962 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280258    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:54:06.284962 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280261    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:54:06.285463 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280264    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:54:06.285463 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280266    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:54:06.285463 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280269    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:54:06.285463 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280272    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:54:06.285463 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280274    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:54:06.285463 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280277    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:54:06.285463 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280279    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:54:06.285463 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280282    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:54:06.285463 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280284    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:54:06.285463 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280287    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:54:06.285463 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280290    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:54:06.285463 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280292    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:54:06.285463 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280295    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:54:06.285463 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280297    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:54:06.285463 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280301    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:54:06.285463 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280304    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:54:06.285463 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280307    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:54:06.285463 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280310    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:54:06.285463 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280313    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:54:06.285463 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280315    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:54:06.285948 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.280317    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:54:06.285948 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280392    2585 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 15:54:06.285948 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280454    2585 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 15:54:06.285948 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280473    2585 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 15:54:06.285948 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280487    2585 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 15:54:06.285948 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280491    2585 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 15:54:06.285948 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280494    2585 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 15:54:06.285948 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280499    2585 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 15:54:06.285948 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280503    2585 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 15:54:06.285948 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280506    2585 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 15:54:06.285948 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280509    2585 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 15:54:06.285948 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280513    2585 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 15:54:06.285948 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280516    2585 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 15:54:06.285948 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280520    2585 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 15:54:06.285948 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280523    2585 flags.go:64] FLAG: --cgroup-root=""
Sep 17 15:54:06.285948 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280526    2585 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 15:54:06.285948 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280529    2585 flags.go:64] FLAG: --client-ca-file=""
Sep 17 15:54:06.285948 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280532    2585 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 15:54:06.285948 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280535    2585 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 15:54:06.285948 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280539    2585 flags.go:64] FLAG: --cluster-domain=""
Sep 17 15:54:06.285948 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280542    2585 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 15:54:06.285948 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280545    2585 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 15:54:06.285948 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280548    2585 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 15:54:06.285948 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280551    2585 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 15:54:06.286574 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280555    2585 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 15:54:06.286574 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280558    2585 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 15:54:06.286574 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280561    2585 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 15:54:06.286574 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280565    2585 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 15:54:06.286574 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280568    2585 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 15:54:06.286574 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280571    2585 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 15:54:06.286574 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280575    2585 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 15:54:06.286574 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280578    2585 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 15:54:06.286574 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280581    2585 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 15:54:06.286574 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280585    2585 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 15:54:06.286574 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280588    2585 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 15:54:06.286574 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280591    2585 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 15:54:06.286574 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280594    2585 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 15:54:06.286574 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280596    2585 flags.go:64] FLAG: --enable-server="true"
Sep 17 15:54:06.286574 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280599    2585 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 15:54:06.286574 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280604    2585 flags.go:64] FLAG: --event-burst="100"
Sep 17 15:54:06.286574 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280607    2585 flags.go:64] FLAG: --event-qps="50"
Sep 17 15:54:06.286574 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280610    2585 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 15:54:06.286574 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280613    2585 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 15:54:06.286574 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280616    2585 flags.go:64] FLAG: --eviction-hard=""
Sep 17 15:54:06.286574 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280619    2585 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 15:54:06.286574 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280622    2585 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 15:54:06.286574 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280625    2585 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 15:54:06.286574 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280628    2585 flags.go:64] FLAG: --eviction-soft=""
Sep 17 15:54:06.286574 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280631    2585 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 15:54:06.287201 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280633    2585 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 15:54:06.287201 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280637    2585 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 15:54:06.287201 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280639    2585 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 15:54:06.287201 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280642    2585 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 15:54:06.287201 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280645    2585 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 15:54:06.287201 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280648    2585 flags.go:64] FLAG: --feature-gates=""
Sep 17 15:54:06.287201 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280651    2585 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 15:54:06.287201 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280654    2585 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 15:54:06.287201 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280658    2585 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 15:54:06.287201 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280661    2585 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 15:54:06.287201 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280664    2585 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 15:54:06.287201 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280667    2585 flags.go:64] FLAG: --help="false"
Sep 17 15:54:06.287201 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280670    2585 flags.go:64] FLAG: --hostname-override="ip-10-0-135-43.ec2.internal"
Sep 17 15:54:06.287201 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280673    2585 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 15:54:06.287201 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280676    2585 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 15:54:06.287201 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280679    2585 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 15:54:06.287201 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280682    2585 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 15:54:06.287201 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280686    2585 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 15:54:06.287201 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280689    2585 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 15:54:06.287201 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280691    2585 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 15:54:06.287201 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280694    2585 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 15:54:06.287201 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280697    2585 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 15:54:06.287201 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280700    2585 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 15:54:06.287866 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280703    2585 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 15:54:06.287866 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280706    2585 flags.go:64] FLAG: --kube-reserved=""
Sep 17 15:54:06.287866 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280709    2585 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 15:54:06.287866 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280711    2585 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 15:54:06.287866 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280714    2585 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 15:54:06.287866 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280717    2585 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 15:54:06.287866 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280720    2585 flags.go:64] FLAG: --lock-file=""
Sep 17 15:54:06.287866 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280723    2585 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 15:54:06.287866 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280726    2585 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 15:54:06.287866 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280729    2585 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 15:54:06.287866 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280734    2585 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 15:54:06.287866 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280737    2585 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 15:54:06.287866 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280739    2585 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 15:54:06.287866 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280742    2585 flags.go:64] FLAG: --logging-format="text"
Sep 17 15:54:06.287866 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280745    2585 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 15:54:06.287866 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280748    2585 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 15:54:06.287866 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280751    2585 flags.go:64] FLAG: --manifest-url=""
Sep 17 15:54:06.287866 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280754    2585 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 15:54:06.287866 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280759    2585 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 15:54:06.287866 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280762    2585 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 15:54:06.287866 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280766    2585 flags.go:64] FLAG: --max-pods="110"
Sep 17 15:54:06.287866 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280769    2585 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 15:54:06.287866 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280772    2585 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 15:54:06.287866 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280775    2585 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 15:54:06.287866 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280778    2585 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 15:54:06.288539 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280780    2585 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 15:54:06.288539 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280783    2585 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 15:54:06.288539 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280786    2585 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 15:54:06.288539 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280793    2585 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 15:54:06.288539 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280796    2585 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 15:54:06.288539 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280799    2585 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 15:54:06.288539 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280802    2585 flags.go:64] FLAG: --pod-cidr=""
Sep 17 15:54:06.288539 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280805    2585 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 15:54:06.288539 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280810    2585 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 15:54:06.288539 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280814    2585 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 15:54:06.288539 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280817    2585 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 15:54:06.288539 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280820    2585 flags.go:64] FLAG: --port="10250"
Sep 17 15:54:06.288539 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280823    2585 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 15:54:06.288539 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280826    2585 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04ab6ebf0f0ec3754"
Sep 17 15:54:06.288539 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280829    2585 flags.go:64] FLAG: --qos-reserved=""
Sep 17 15:54:06.288539 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280831    2585 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 15:54:06.288539 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280834    2585 flags.go:64] FLAG: --register-node="true"
Sep 17 15:54:06.288539 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280837    2585 flags.go:64] FLAG: --register-with-taints=""
Sep 17 15:54:06.288539 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280841    2585 flags.go:64] FLAG: --registry-burst="10"
Sep 17 15:54:06.288539 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280843    2585 flags.go:64] FLAG: --registry-qps="5"
Sep 17 15:54:06.288539 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280846    2585 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 15:54:06.288539 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280849    2585 flags.go:64] FLAG: --reserved-memory=""
Sep 17 15:54:06.288539 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280852    2585 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 15:54:06.288539 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280855    2585 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 15:54:06.288539 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280858    2585 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 15:54:06.289149 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280861    2585 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 15:54:06.289149 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280864    2585 flags.go:64] FLAG: --runonce="false"
Sep 17 15:54:06.289149 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280867    2585 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 15:54:06.289149 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280870    2585 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 15:54:06.289149 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280872    2585 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 15:54:06.289149 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280875    2585 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 15:54:06.289149 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280878    2585 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 15:54:06.289149 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280881    2585 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 15:54:06.289149 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280884    2585 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 15:54:06.289149 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280887    2585 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 15:54:06.289149 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280889    2585 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 15:54:06.289149 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280892    2585 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 15:54:06.289149 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280895    2585 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 15:54:06.289149 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280898    2585 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 15:54:06.289149 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280901    2585 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 15:54:06.289149 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280903    2585 flags.go:64] FLAG: --system-cgroups=""
Sep 17 15:54:06.289149 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280906    2585 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 15:54:06.289149 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280912    2585 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 15:54:06.289149 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280914    2585 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 15:54:06.289149 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280917    2585 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 15:54:06.289149 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280921    2585 flags.go:64] FLAG: --tls-min-version=""
Sep 17 15:54:06.289149 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280923    2585 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 15:54:06.289149 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280926    2585 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 15:54:06.289149 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280929    2585 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 15:54:06.289149 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280932    2585 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 15:54:06.289784 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280935    2585 flags.go:64] FLAG: --v="2"
Sep 17 15:54:06.289784 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280938    2585 flags.go:64] FLAG: --version="false"
Sep 17 15:54:06.289784 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280942    2585 flags.go:64] FLAG: --vmodule=""
Sep 17 15:54:06.289784 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280946    2585 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 15:54:06.289784 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.280949    2585 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 15:54:06.289784 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281057    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:54:06.289784 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281063    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:54:06.289784 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281067    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:54:06.289784 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281070    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:54:06.289784 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281073    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:54:06.289784 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281077    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:54:06.289784 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281081    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:54:06.289784 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281085    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:54:06.289784 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281088    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:54:06.289784 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281091    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:54:06.289784 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281094    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:54:06.289784 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281097    2585 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:54:06.289784 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281100    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:54:06.289784 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281103    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:54:06.289784 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281105    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:54:06.289784 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281108    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:54:06.290305 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281111    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:54:06.290305 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281114    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:54:06.290305 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281117    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:54:06.290305 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281119    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:54:06.290305 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281122    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:54:06.290305 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281125    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:54:06.290305 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281128    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:54:06.290305 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281130    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:54:06.290305 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281133    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:54:06.290305 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281135    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:54:06.290305 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281139    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:54:06.290305 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281143    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:54:06.290305 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281146    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:54:06.290305 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281149    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:54:06.290305 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281151    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:54:06.290305 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281154    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:54:06.290305 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281157    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:54:06.290305 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281161    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:54:06.290305 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281165    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:54:06.290791 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281168    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:54:06.290791 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281171    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:54:06.290791 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281176    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:54:06.290791 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281179    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:54:06.290791 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281182    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:54:06.290791 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281185    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:54:06.290791 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281188    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:54:06.290791 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281191    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:54:06.290791 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281194    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:54:06.290791 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281197    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:54:06.290791 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281200    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:54:06.290791 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281202    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:54:06.290791 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281205    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:54:06.290791 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281208    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:54:06.290791 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281211    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:54:06.290791 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281213    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:54:06.290791 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281216    2585 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:54:06.290791 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281219    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:54:06.290791 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281222    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:54:06.290791 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281225    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:54:06.291326 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281227    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:54:06.291326 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281230    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:54:06.291326 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281232    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:54:06.291326 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281235    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:54:06.291326 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281238    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:54:06.291326 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281240    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:54:06.291326 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281243    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:54:06.291326 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281245    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:54:06.291326 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281248    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:54:06.291326 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281250    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:54:06.291326 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281255    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:54:06.291326 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281257    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:54:06.291326 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281260    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:54:06.291326 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281263    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:54:06.291326 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281267    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:54:06.291326 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281269    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:54:06.291326 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281272    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:54:06.291326 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281275    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:54:06.291326 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281277    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:54:06.291326 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281280    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:54:06.291841 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281282    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:54:06.291841 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281285    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:54:06.291841 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281288    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:54:06.291841 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281290    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:54:06.291841 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281293    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:54:06.291841 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281295    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:54:06.291841 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281298    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:54:06.291841 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281300    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:54:06.291841 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281303    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:54:06.291841 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281305    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:54:06.291841 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281308    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:54:06.291841 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281311    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:54:06.291841 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281314    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:54:06.291841 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281317    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:54:06.291841 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281319    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:54:06.291841 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281322    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:54:06.291841 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281325    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:54:06.291841 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281328    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:54:06.291841 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281330    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:54:06.291841 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281333    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:54:06.292350 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281336    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:54:06.292350 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281338    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:54:06.292350 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281342    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:54:06.292350 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.281345    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:54:06.292350 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.281917    2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 15:54:06.292350 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.289173    2585 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 15:54:06.292350 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.289188    2585 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 15:54:06.292350 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.289208    2585 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 15:54:06.292350 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.289217    2585 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 15:54:06.292350 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289290    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:54:06.292350 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289295    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:54:06.292350 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289298    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:54:06.292350 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289301    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:54:06.292350 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289305    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:54:06.292350 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289308    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:54:06.292745 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289311    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:54:06.292745 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289314    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:54:06.292745 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289318    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:54:06.292745 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289321    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:54:06.292745 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289324    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:54:06.292745 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289326    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:54:06.292745 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289330    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:54:06.292745 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289332    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:54:06.292745 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289341    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:54:06.292745 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289344    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:54:06.292745 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289347    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:54:06.292745 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289349    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:54:06.292745 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289352    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:54:06.292745 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289354    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:54:06.292745 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289357    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:54:06.292745 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289360    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:54:06.292745 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289363    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:54:06.292745 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289368    2585 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:54:06.292745 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289371    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:54:06.292745 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289373    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:54:06.293233 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289376    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:54:06.293233 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289386    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:54:06.293233 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289389    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:54:06.293233 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289392    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:54:06.293233 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289395    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:54:06.293233 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289398    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:54:06.293233 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289400    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:54:06.293233 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289403    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:54:06.293233 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289406    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:54:06.293233 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289409    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:54:06.293233 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289411    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:54:06.293233 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289414    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:54:06.293233 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289431    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:54:06.293233 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289435    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:54:06.293233 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289437    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:54:06.293233 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289440    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:54:06.293233 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289447    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:54:06.293233 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289450    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:54:06.293233 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289452    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:54:06.293719 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289455    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:54:06.293719 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289458    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:54:06.293719 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289468    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:54:06.293719 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289471    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:54:06.293719 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289475    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:54:06.293719 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289479    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:54:06.293719 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289482    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:54:06.293719 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289485    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:54:06.293719 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289488    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:54:06.293719 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289491    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:54:06.293719 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289494    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:54:06.293719 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289497    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:54:06.293719 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289500    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:54:06.293719 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289503    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:54:06.293719 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289506    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:54:06.293719 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289509    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:54:06.293719 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289512    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:54:06.293719 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289515    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:54:06.293719 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289518    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:54:06.294192 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289521    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:54:06.294192 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289524    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:54:06.294192 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289527    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:54:06.294192 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289529    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:54:06.294192 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289532    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:54:06.294192 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289534    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:54:06.294192 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289537    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:54:06.294192 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289540    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:54:06.294192 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289542    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:54:06.294192 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289545    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:54:06.294192 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289548    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:54:06.294192 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289550    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:54:06.294192 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289553    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:54:06.294192 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289555    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:54:06.294192 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289558    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:54:06.294192 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289560    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:54:06.294192 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289568    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:54:06.294192 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289571    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:54:06.294192 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289574    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:54:06.294192 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289576    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:54:06.294695 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289579    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:54:06.294695 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289582    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:54:06.294695 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289584    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:54:06.294695 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289587    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:54:06.294695 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289589    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:54:06.294695 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289592    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:54:06.294695 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289594    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:54:06.294695 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289597    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:54:06.294695 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289599    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:54:06.294695 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289603    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:54:06.294695 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289606    2585 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:54:06.294695 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289608    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:54:06.294695 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289611    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:54:06.294695 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289615    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:54:06.294695 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289618    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:54:06.295059 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.289623    2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 15:54:06.295059 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289760    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:54:06.295059 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289765    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:54:06.295059 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289768    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:54:06.295059 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289771    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:54:06.295059 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289774    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:54:06.295059 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289776    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:54:06.295059 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289779    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:54:06.295059 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289782    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:54:06.295059 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289785    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:54:06.295059 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289787    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:54:06.295059 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289790    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:54:06.295059 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289794    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:54:06.295059 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289797    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:54:06.295059 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289805    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:54:06.295454 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289808    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:54:06.295454 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289810    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:54:06.295454 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289813    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:54:06.295454 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289815    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:54:06.295454 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289818    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:54:06.295454 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289820    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:54:06.295454 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289823    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:54:06.295454 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289826    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:54:06.295454 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289828    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:54:06.295454 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289838    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:54:06.295454 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289841    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:54:06.295454 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289845    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:54:06.295454 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289847    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:54:06.295454 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289850    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:54:06.295454 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289853    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:54:06.295454 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289856    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:54:06.295454 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289858    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:54:06.295454 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289861    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:54:06.295454 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289863    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:54:06.295454 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289866    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:54:06.295941 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289868    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:54:06.295941 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289871    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:54:06.295941 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289874    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:54:06.295941 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289876    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:54:06.295941 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289879    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:54:06.295941 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289881    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:54:06.295941 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289884    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:54:06.295941 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289887    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:54:06.295941 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289889    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:54:06.295941 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289892    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:54:06.295941 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289894    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:54:06.295941 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289896    2585 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:54:06.295941 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289905    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:54:06.295941 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289908    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:54:06.295941 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289910    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:54:06.295941 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289913    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:54:06.295941 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289915    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:54:06.295941 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289918    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:54:06.295941 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289920    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:54:06.295941 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289923    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:54:06.296563 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289926    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:54:06.296563 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289929    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:54:06.296563 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289931    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:54:06.296563 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289934    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:54:06.296563 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289937    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:54:06.296563 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289939    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:54:06.296563 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289942    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:54:06.296563 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289945    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:54:06.296563 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289949    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:54:06.296563 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289952    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:54:06.296563 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289955    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:54:06.296563 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289958    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:54:06.296563 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289961    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:54:06.296563 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289964    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:54:06.296563 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289967    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:54:06.296563 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289970    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:54:06.296563 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289973    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:54:06.296563 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289975    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:54:06.296563 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289978    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:54:06.297333 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289981    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:54:06.297333 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289984    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:54:06.297333 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289988    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:54:06.297333 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289991    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:54:06.297333 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289994    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:54:06.297333 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.289997    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:54:06.297333 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.290005    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:54:06.297333 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.290008    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:54:06.297333 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.290011    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:54:06.297333 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.290014    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:54:06.297333 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.290017    2585 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:54:06.297333 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.290020    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:54:06.297333 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.290023    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:54:06.297333 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.290025    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:54:06.297333 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.290028    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:54:06.297333 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.290030    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:54:06.297333 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.290033    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:54:06.297333 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.290036    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:54:06.297333 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.290039    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:54:06.298029 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.290041    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:54:06.298029 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.290044    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:54:06.298029 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.290046    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:54:06.298029 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.290049    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:54:06.298029 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.290051    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:54:06.298029 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.290054    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:54:06.298029 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:06.290057    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:54:06.298029 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.290062    2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 15:54:06.298029 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.290188    2585 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 15:54:06.298029 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.292966    2585 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 15:54:06.298029 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.293691    2585 server.go:1019] "Starting client certificate rotation"
Sep 17 15:54:06.298029 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.293799    2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 15:54:06.298029 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.293848    2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 15:54:06.298949 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.298932    2585 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 15:54:06.300172 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.300155    2585 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 15:54:06.305568 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.305548    2585 log.go:25] "Validated CRI v1 runtime API"
Sep 17 15:54:06.307364 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.307342    2585 log.go:25] "Validated CRI v1 image API"
Sep 17 15:54:06.308196 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.308177    2585 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 15:54:06.310243 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.310222    2585 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 976f157d-80be-418a-b00b-36b7bf9e6c1d:/dev/nvme0n1p3 c02c8b7b-05a3-4944-a29d-259a32ce2ac0:/dev/nvme0n1p4]
Sep 17 15:54:06.310313 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.310241    2585 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 15:54:06.314511 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.314387    2585 manager.go:216] Machine: {Timestamp:2026-09-17 15:54:06.313922911 +0000 UTC m=+2.084254700 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3109448 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b3aa28c3f4cce201fd63b22e05500 SystemUUID:ec2b3aa2-8c3f-4cce-201f-d63b22e05500 BootID:9b1a4327-9649-4954-aeed-e4daaf40025f Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ef:06:ce:19 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ef:06:ce:19 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:66:99:a4:a5:30:46 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 15:54:06.314511 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.314506    2585 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 15:54:06.314632 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.314621    2585 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 15:54:06.315920 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.315895    2585 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 15:54:06.316083 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.315923    2585 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-43.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 15:54:06.316612 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.316602    2585 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 15:54:06.316643 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.316615    2585 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 15:54:06.316643 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.316623    2585 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 15:54:06.316643 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.316634    2585 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 15:54:06.316721 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.316703    2585 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 15:54:06.320133 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.320121    2585 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 15:54:06.320238 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.320228    2585 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 15:54:06.320270 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.320260    2585 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 15:54:06.320297 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.320270    2585 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 15:54:06.320297 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.320284    2585 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 15:54:06.320346 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.320312    2585 kubelet.go:398] "Adding apiserver pod source"
Sep 17 15:54:06.320346 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.320320    2585 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 15:54:06.320725 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.320712    2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 15:54:06.320796 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.320740    2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 15:54:06.322344 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.322329    2585 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 15:54:06.324227 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.324210    2585 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 15:54:06.324313 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.324250    2585 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 15:54:06.325182 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.325169    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 15:54:06.325245 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.325190    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 15:54:06.325245 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.325199    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 15:54:06.325245 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.325208    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 15:54:06.325245 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.325221    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 15:54:06.325245 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.325230    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 15:54:06.325468 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.325250    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 15:54:06.325468 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.325260    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 15:54:06.325468 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.325270    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 15:54:06.325468 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.325279    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 15:54:06.325468 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.325291    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 15:54:06.325468 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.325300    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 15:54:06.325468 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.325337    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 15:54:06.325468 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.325346    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 15:54:06.336792 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.336773    2585 server.go:1268] "Started kubelet"
Sep 17 15:54:06.336903 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.336847    2585 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 15:54:06.336981 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.336930    2585 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 15:54:06.337046 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.337004    2585 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 15:54:06.337754 ip-10-0-135-43 systemd[1]: Started Kubernetes Kubelet.
Sep 17 15:54:06.338101 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.338013    2585 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 15:54:06.338101 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.338034    2585 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 15:54:06.341754 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:06.341727    2585 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 15:54:06.343461 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.343445    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-43.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 15:54:06.343533 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:06.343509    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 15:54:06.343593 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:06.343576    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-43.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 15:54:06.343933 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.343910    2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 15:54:06.343933 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.343922    2585 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 15:54:06.344079 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:06.344009    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-43.ec2.internal\" not found"
Sep 17 15:54:06.344470 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.344449    2585 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 15:54:06.344571 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.344450    2585 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 15:54:06.344651 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.344630    2585 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 15:54:06.344749 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.344737    2585 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 15:54:06.344813 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.344765    2585 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 15:54:06.345082 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.345066    2585 factory.go:55] Registering systemd factory
Sep 17 15:54:06.345166 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.345086    2585 factory.go:223] Registration of the systemd container factory successfully
Sep 17 15:54:06.345286 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.345264    2585 factory.go:153] Registering CRI-O factory
Sep 17 15:54:06.345286 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.345282    2585 factory.go:223] Registration of the crio container factory successfully
Sep 17 15:54:06.345391 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.345347    2585 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 15:54:06.345391 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.345370    2585 factory.go:103] Registering Raw factory
Sep 17 15:54:06.345506 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.345402    2585 manager.go:1196] Started watching for new ooms in manager
Sep 17 15:54:06.345828 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.345814    2585 manager.go:349] Starting recovery of all containers
Sep 17 15:54:06.384823 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.384796    2585 manager.go:354] Recovery completed
Sep 17 15:54:06.389307 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.389289    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:54:06.389654 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.389637    2585 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 15:54:06.391249 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:06.391206    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 15:54:06.391331 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.391274    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-43.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:54:06.391331 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.391303    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-43.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:54:06.391331 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.391318    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-43.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:54:06.391514 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:06.391350    2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-43.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 15:54:06.391993 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:06.391156    2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-43.ec2.internal.18d626b1a477f773  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-43.ec2.internal,UID:ip-10-0-135-43.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-43.ec2.internal,},FirstTimestamp:2026-09-17 15:54:06.336743283 +0000 UTC m=+2.107075071,LastTimestamp:2026-09-17 15:54:06.336743283 +0000 UTC m=+2.107075071,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-43.ec2.internal,}"
Sep 17 15:54:06.391993 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.391985    2585 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 15:54:06.392117 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.391999    2585 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 15:54:06.392117 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.392015    2585 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 15:54:06.392117 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.392072    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:54:06.393214 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.393195    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-43.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:54:06.393287 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.393222    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-43.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:54:06.393287 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.393245    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-43.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:54:06.394116 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.394104    2585 policy_none.go:49] "None policy: Start"
Sep 17 15:54:06.394160 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.394120    2585 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 15:54:06.394160 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.394130    2585 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 15:54:06.395978 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.395968    2585 policy_none.go:47] "Start"
Sep 17 15:54:06.397114 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:06.397033    2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-43.ec2.internal.18d626b1a7b85105  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-43.ec2.internal,UID:ip-10-0-135-43.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-43.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-43.ec2.internal,},FirstTimestamp:2026-09-17 15:54:06.391292165 +0000 UTC m=+2.161623960,LastTimestamp:2026-09-17 15:54:06.391292165 +0000 UTC m=+2.161623960,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-43.ec2.internal,}"
Sep 17 15:54:06.402859 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:06.402792    2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-43.ec2.internal.18d626b1a7b8900e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-43.ec2.internal,UID:ip-10-0-135-43.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-43.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-43.ec2.internal,},FirstTimestamp:2026-09-17 15:54:06.391308302 +0000 UTC m=+2.161640092,LastTimestamp:2026-09-17 15:54:06.391308302 +0000 UTC m=+2.161640092,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-43.ec2.internal,}"
Sep 17 15:54:06.419055 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:06.418988    2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-43.ec2.internal.18d626b1a7b8cc1a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-43.ec2.internal,UID:ip-10-0-135-43.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-43.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-43.ec2.internal,},FirstTimestamp:2026-09-17 15:54:06.391323674 +0000 UTC m=+2.161655465,LastTimestamp:2026-09-17 15:54:06.391323674 +0000 UTC m=+2.161655465,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-43.ec2.internal,}"
Sep 17 15:54:06.432643 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:06.432572    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-43.ec2.internal.18d626b1a7b85105\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-43.ec2.internal.18d626b1a7b85105  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-43.ec2.internal,UID:ip-10-0-135-43.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-43.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-43.ec2.internal,},FirstTimestamp:2026-09-17 15:54:06.391292165 +0000 UTC m=+2.161623960,LastTimestamp:2026-09-17 15:54:06.39321136 +0000 UTC m=+2.163543148,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-43.ec2.internal,}"
Sep 17 15:54:06.448316 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.436178    2585 manager.go:337] "Starting Device Plugin manager"
Sep 17 15:54:06.448316 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:06.436213    2585 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 15:54:06.448316 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.436224    2585 server.go:87] "Starting device plugin registration server"
Sep 17 15:54:06.448316 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.436515    2585 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 15:54:06.448316 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.436533    2585 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 15:54:06.448316 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.436625    2585 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 15:54:06.448316 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.436728    2585 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 15:54:06.448316 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.436740    2585 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 15:54:06.448316 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:06.437370    2585 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 15:54:06.448316 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:06.437409    2585 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-43.ec2.internal\" not found"
Sep 17 15:54:06.449741 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:06.449683    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-43.ec2.internal.18d626b1a7b8900e\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-43.ec2.internal.18d626b1a7b8900e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-43.ec2.internal,UID:ip-10-0-135-43.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-43.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-43.ec2.internal,},FirstTimestamp:2026-09-17 15:54:06.391308302 +0000 UTC m=+2.161640092,LastTimestamp:2026-09-17 15:54:06.393227214 +0000 UTC m=+2.163559003,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-43.ec2.internal,}"
Sep 17 15:54:06.467657 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:06.467582    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-43.ec2.internal.18d626b1a7b8cc1a\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-43.ec2.internal.18d626b1a7b8cc1a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-43.ec2.internal,UID:ip-10-0-135-43.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-43.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-43.ec2.internal,},FirstTimestamp:2026-09-17 15:54:06.391323674 +0000 UTC m=+2.161655465,LastTimestamp:2026-09-17 15:54:06.393250344 +0000 UTC m=+2.163582133,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-43.ec2.internal,}"
Sep 17 15:54:06.479227 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:06.479162    2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-43.ec2.internal.18d626b1ab076cc7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-43.ec2.internal,UID:ip-10-0-135-43.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-135-43.ec2.internal,},FirstTimestamp:2026-09-17 15:54:06.446808263 +0000 UTC m=+2.217140039,LastTimestamp:2026-09-17 15:54:06.446808263 +0000 UTC m=+2.217140039,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-43.ec2.internal,}"
Sep 17 15:54:06.504857 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.504818    2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 15:54:06.506242 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.506226    2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 15:54:06.506304 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.506280    2585 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 15:54:06.506304 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.506302    2585 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 15:54:06.506396 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:06.506351    2585 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 15:54:06.517007 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:06.516978    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 17 15:54:06.537110 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.537032    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:54:06.537989 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.537974    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-43.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:54:06.538077 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.538006    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-43.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:54:06.538077 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.538016    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-43.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:54:06.538077 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.538039    2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-43.ec2.internal"
Sep 17 15:54:06.558914 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:06.558818    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-43.ec2.internal.18d626b1a7b85105\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-43.ec2.internal.18d626b1a7b85105  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-43.ec2.internal,UID:ip-10-0-135-43.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-43.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-43.ec2.internal,},FirstTimestamp:2026-09-17 15:54:06.391292165 +0000 UTC m=+2.161623960,LastTimestamp:2026-09-17 15:54:06.537988536 +0000 UTC m=+2.308320324,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-43.ec2.internal,}"
Sep 17 15:54:06.572114 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:06.572089    2585 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-135-43.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-135-43.ec2.internal"
Sep 17 15:54:06.572218 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:06.572066    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-43.ec2.internal.18d626b1a7b8900e\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-43.ec2.internal.18d626b1a7b8900e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-43.ec2.internal,UID:ip-10-0-135-43.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-43.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-43.ec2.internal,},FirstTimestamp:2026-09-17 15:54:06.391308302 +0000 UTC m=+2.161640092,LastTimestamp:2026-09-17 15:54:06.538010817 +0000 UTC m=+2.308342607,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-43.ec2.internal,}"
Sep 17 15:54:06.593551 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:06.593529    2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-43.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="400ms"
Sep 17 15:54:06.593634 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:06.593539    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-43.ec2.internal.18d626b1a7b8cc1a\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-43.ec2.internal.18d626b1a7b8cc1a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-43.ec2.internal,UID:ip-10-0-135-43.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-43.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-43.ec2.internal,},FirstTimestamp:2026-09-17 15:54:06.391323674 +0000 UTC m=+2.161655465,LastTimestamp:2026-09-17 15:54:06.538019643 +0000 UTC m=+2.308351431,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-43.ec2.internal,}"
Sep 17 15:54:06.606653 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.606626    2585 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-43.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-43.ec2.internal"]
Sep 17 15:54:06.606720 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.606710    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:54:06.608076 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.608058    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-43.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:54:06.608148 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.608093    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-43.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:54:06.608148 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.608107    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-43.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:54:06.610408 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.610394    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:54:06.610531 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.610517    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-43.ec2.internal"
Sep 17 15:54:06.610581 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.610547    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:54:06.611134 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.611119    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-43.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:54:06.611200 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.611134    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-43.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:54:06.611200 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.611148    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-43.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:54:06.611200 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.611158    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-43.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:54:06.611200 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.611169    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-43.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:54:06.611381 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.611159    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-43.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:54:06.613654 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.613638    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-43.ec2.internal"
Sep 17 15:54:06.613738 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.613669    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:54:06.614391 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.614366    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-43.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:54:06.614491 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.614392    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-43.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:54:06.614491 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.614407    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-43.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:54:06.615497 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.615483    2585 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-b5dqp"
Sep 17 15:54:06.623501 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:06.623409    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-43.ec2.internal.18d626b1a7b85105\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-43.ec2.internal.18d626b1a7b85105  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-43.ec2.internal,UID:ip-10-0-135-43.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-43.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-43.ec2.internal,},FirstTimestamp:2026-09-17 15:54:06.391292165 +0000 UTC m=+2.161623960,LastTimestamp:2026-09-17 15:54:06.608074961 +0000 UTC m=+2.378406756,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-43.ec2.internal,}"
Sep 17 15:54:06.629577 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.629559    2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-b5dqp"
Sep 17 15:54:06.632593 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:06.632531    2585 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-43.ec2.internal.18d626b1a7b8900e\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-43.ec2.internal.18d626b1a7b8900e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-43.ec2.internal,UID:ip-10-0-135-43.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-43.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-43.ec2.internal,},FirstTimestamp:2026-09-17 15:54:06.391308302 +0000 UTC m=+2.161640092,LastTimestamp:2026-09-17 15:54:06.608100333 +0000 UTC m=+2.378432126,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-43.ec2.internal,}"
Sep 17 15:54:06.639096 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:06.639078    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-43.ec2.internal\" not found" node="ip-10-0-135-43.ec2.internal"
Sep 17 15:54:06.644676 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:06.644658    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-43.ec2.internal\" not found" node="ip-10-0-135-43.ec2.internal"
Sep 17 15:54:06.646774 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.646754    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7156e3815525514fe59f7c5337670664-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-43.ec2.internal\" (UID: \"7156e3815525514fe59f7c5337670664\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-43.ec2.internal"
Sep 17 15:54:06.646865 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.646782    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7156e3815525514fe59f7c5337670664-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-43.ec2.internal\" (UID: \"7156e3815525514fe59f7c5337670664\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-43.ec2.internal"
Sep 17 15:54:06.646865 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.646797    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5692cd509d685463b01f1d043c0bcb98-config\") pod \"kube-apiserver-proxy-ip-10-0-135-43.ec2.internal\" (UID: \"5692cd509d685463b01f1d043c0bcb98\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-43.ec2.internal"
Sep 17 15:54:06.747095 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.747064    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7156e3815525514fe59f7c5337670664-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-43.ec2.internal\" (UID: \"7156e3815525514fe59f7c5337670664\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-43.ec2.internal"
Sep 17 15:54:06.747095 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.747095    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5692cd509d685463b01f1d043c0bcb98-config\") pod \"kube-apiserver-proxy-ip-10-0-135-43.ec2.internal\" (UID: \"5692cd509d685463b01f1d043c0bcb98\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-43.ec2.internal"
Sep 17 15:54:06.747273 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.747111    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7156e3815525514fe59f7c5337670664-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-43.ec2.internal\" (UID: \"7156e3815525514fe59f7c5337670664\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-43.ec2.internal"
Sep 17 15:54:06.747273 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.747151    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7156e3815525514fe59f7c5337670664-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-43.ec2.internal\" (UID: \"7156e3815525514fe59f7c5337670664\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-43.ec2.internal"
Sep 17 15:54:06.747273 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.747166    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7156e3815525514fe59f7c5337670664-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-43.ec2.internal\" (UID: \"7156e3815525514fe59f7c5337670664\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-43.ec2.internal"
Sep 17 15:54:06.747273 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.747183    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5692cd509d685463b01f1d043c0bcb98-config\") pod \"kube-apiserver-proxy-ip-10-0-135-43.ec2.internal\" (UID: \"5692cd509d685463b01f1d043c0bcb98\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-43.ec2.internal"
Sep 17 15:54:06.772458 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.772434    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:54:06.773352 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.773332    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-43.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:54:06.773450 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.773361    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-43.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:54:06.773450 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.773371    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-43.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:54:06.773450 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.773397    2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-43.ec2.internal"
Sep 17 15:54:06.783205 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.783187    2585 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-43.ec2.internal"
Sep 17 15:54:06.783251 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:06.783210    2585 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-43.ec2.internal\": node \"ip-10-0-135-43.ec2.internal\" not found"
Sep 17 15:54:06.836732 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:06.836671    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-43.ec2.internal\" not found"
Sep 17 15:54:06.937081 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:06.937054    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-43.ec2.internal\" not found"
Sep 17 15:54:06.942218 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.942200    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-43.ec2.internal"
Sep 17 15:54:06.948527 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:06.948512    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-43.ec2.internal"
Sep 17 15:54:07.038087 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:07.038051    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-43.ec2.internal\" not found"
Sep 17 15:54:07.138623 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:07.138537    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-43.ec2.internal\" not found"
Sep 17 15:54:07.239069 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:07.239023    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-43.ec2.internal\" not found"
Sep 17 15:54:07.293634 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:07.293595    2585 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 15:54:07.324320 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:07.324290    2585 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:54:07.339859 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:07.339833    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-43.ec2.internal\" not found"
Sep 17 15:54:07.420199 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:07.420142    2585 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:54:07.440786 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:07.440761    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-43.ec2.internal\" not found"
Sep 17 15:54:07.530583 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:07.530403    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:54:07.531588 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:07.531573    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-43.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:54:07.531645 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:07.531603    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-43.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:54:07.531645 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:07.531615    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-43.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:54:07.541752 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:07.541734    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-43.ec2.internal\" not found"
Sep 17 15:54:07.601558 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:07.601520    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7156e3815525514fe59f7c5337670664.slice/crio-b8271257bc4b7ee8f91a89ed3aa1fbc4694dfd84d3f50b90d23ba0a3547f7ab1 WatchSource:0}: Error finding container b8271257bc4b7ee8f91a89ed3aa1fbc4694dfd84d3f50b90d23ba0a3547f7ab1: Status 404 returned error can't find the container with id b8271257bc4b7ee8f91a89ed3aa1fbc4694dfd84d3f50b90d23ba0a3547f7ab1
Sep 17 15:54:07.601823 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:07.601803    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5692cd509d685463b01f1d043c0bcb98.slice/crio-fa4b1441f8bed097f53ec5ac1849d8af2b7dd3e5345cc8c6570d4f2bb6406095 WatchSource:0}: Error finding container fa4b1441f8bed097f53ec5ac1849d8af2b7dd3e5345cc8c6570d4f2bb6406095: Status 404 returned error can't find the container with id fa4b1441f8bed097f53ec5ac1849d8af2b7dd3e5345cc8c6570d4f2bb6406095
Sep 17 15:54:07.606695 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:07.606672    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 15:54:07.634982 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:07.634929    2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 15:49:06 +0000 UTC" deadline="2028-03-10 05:25:19.428673853 +0000 UTC"
Sep 17 15:54:07.634982 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:07.634974    2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12949h31m11.793702811s"
Sep 17 15:54:07.642091 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:07.642069    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-43.ec2.internal\" not found"
Sep 17 15:54:07.694973 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:07.694911    2585 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:54:07.742520 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:07.742492    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-43.ec2.internal\" not found"
Sep 17 15:54:07.780915 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:07.780888    2585 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:54:07.844193 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:07.844168    2585 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-43.ec2.internal"
Sep 17 15:54:07.855205 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:07.855179    2585 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 15:54:07.856052 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:07.856036    2585 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-43.ec2.internal"
Sep 17 15:54:07.876812 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:07.876784    2585 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 15:54:08.321501 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.321471    2585 apiserver.go:52] "Watching apiserver"
Sep 17 15:54:08.330012 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.329987    2585 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 15:54:08.330577 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.330552    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-cm82k","openshift-multus/network-metrics-daemon-m6w9z","openshift-network-operator/iptables-alerter-mvssh","kube-system/konnectivity-agent-ntl9t","openshift-cluster-node-tuning-operator/tuned-cdgrj","openshift-dns/node-resolver-xhtm4","openshift-multus/multus-xhrb8","openshift-network-diagnostics/network-check-target-9mgwn","openshift-ovn-kubernetes/ovnkube-node-2b47p","kube-system/global-pull-secret-syncer-2f4bn","kube-system/kube-apiserver-proxy-ip-10-0-135-43.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65v88","openshift-image-registry/node-ca-dtbtf","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-43.ec2.internal"]
Sep 17 15:54:08.333067 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.333045    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mgwn"
Sep 17 15:54:08.333158 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:08.333124    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mgwn" podUID="3114c3ad-7b7e-41ad-b175-ead7d92d5ee3"
Sep 17 15:54:08.335177 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.335157    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dtbtf"
Sep 17 15:54:08.337340 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.337321    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65v88"
Sep 17 15:54:08.339391 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.339371    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 15:54:08.339683 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.339662    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 15:54:08.340055 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.340034    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ntl9t"
Sep 17 15:54:08.340129 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.340078    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-g62kp\""
Sep 17 15:54:08.341088 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.340442    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 15:54:08.341088 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.340578    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 15:54:08.341088 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.340684    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 15:54:08.341088 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.340736    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 15:54:08.341088 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.340945    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wn8rd\""
Sep 17 15:54:08.342395 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.342374    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cdgrj"
Sep 17 15:54:08.344451 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.344414    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 15:54:08.345000 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.344979    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 15:54:08.345264 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.345224    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-bq7m8\""
Sep 17 15:54:08.345962 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.345941    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7k759\""
Sep 17 15:54:08.346052 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.346007    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 15:54:08.346200 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.346182    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 15:54:08.346503 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.346486    2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 15:54:08.348960 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.348941    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xhtm4"
Sep 17 15:54:08.351936 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.351918    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cm82k"
Sep 17 15:54:08.352984 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.352965    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mdp2n\""
Sep 17 15:54:08.353176 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.352965    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 15:54:08.353405 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.353384    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 15:54:08.354025 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.353992    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xhrb8"
Sep 17 15:54:08.355081 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.355062    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 15:54:08.355286 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.355269    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-f22p2\""
Sep 17 15:54:08.357223 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.357204    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.358603 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.358583    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 15:54:08.359459 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.359441    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f4bn"
Sep 17 15:54:08.359551 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:08.359526    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f4bn" podUID="94d1451b-3eae-4481-a504-c29b4315e62e"
Sep 17 15:54:08.359827 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.359808    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 15:54:08.360017 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.360001    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-hxn67\""
Sep 17 15:54:08.362255 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.362237    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8910d933-4fe9-45a0-be48-f225cded154a-etc-selinux\") pod \"aws-ebs-csi-driver-node-65v88\" (UID: \"8910d933-4fe9-45a0-be48-f225cded154a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65v88"
Sep 17 15:54:08.362363 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.362346    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mvssh"
Sep 17 15:54:08.362477 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.362459    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8249ef17-a5c4-414e-a2ca-a22cc46ef3eb-konnectivity-ca\") pod \"konnectivity-agent-ntl9t\" (UID: \"8249ef17-a5c4-414e-a2ca-a22cc46ef3eb\") " pod="kube-system/konnectivity-agent-ntl9t"
Sep 17 15:54:08.362585 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.362570    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2ecfa8d8-297d-4e71-b29c-726646be98bf-etc-modprobe-d\") pod \"tuned-cdgrj\" (UID: \"2ecfa8d8-297d-4e71-b29c-726646be98bf\") " pod="openshift-cluster-node-tuning-operator/tuned-cdgrj"
Sep 17 15:54:08.362784 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.362763    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2ecfa8d8-297d-4e71-b29c-726646be98bf-sys\") pod \"tuned-cdgrj\" (UID: \"2ecfa8d8-297d-4e71-b29c-726646be98bf\") " pod="openshift-cluster-node-tuning-operator/tuned-cdgrj"
Sep 17 15:54:08.362883 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.362871    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2ecfa8d8-297d-4e71-b29c-726646be98bf-host\") pod \"tuned-cdgrj\" (UID: \"2ecfa8d8-297d-4e71-b29c-726646be98bf\") " pod="openshift-cluster-node-tuning-operator/tuned-cdgrj"
Sep 17 15:54:08.363301 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.362948    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 15:54:08.363301 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.362969    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj46v\" (UniqueName: \"kubernetes.io/projected/2ecfa8d8-297d-4e71-b29c-726646be98bf-kube-api-access-sj46v\") pod \"tuned-cdgrj\" (UID: \"2ecfa8d8-297d-4e71-b29c-726646be98bf\") " pod="openshift-cluster-node-tuning-operator/tuned-cdgrj"
Sep 17 15:54:08.363301 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.362995    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8249ef17-a5c4-414e-a2ca-a22cc46ef3eb-agent-certs\") pod \"konnectivity-agent-ntl9t\" (UID: \"8249ef17-a5c4-414e-a2ca-a22cc46ef3eb\") " pod="kube-system/konnectivity-agent-ntl9t"
Sep 17 15:54:08.363301 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.363020    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2ecfa8d8-297d-4e71-b29c-726646be98bf-lib-modules\") pod \"tuned-cdgrj\" (UID: \"2ecfa8d8-297d-4e71-b29c-726646be98bf\") " pod="openshift-cluster-node-tuning-operator/tuned-cdgrj"
Sep 17 15:54:08.363301 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.363043    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c4438a49-8ea1-40aa-bdd9-2707192617af-hosts-file\") pod \"node-resolver-xhtm4\" (UID: \"c4438a49-8ea1-40aa-bdd9-2707192617af\") " pod="openshift-dns/node-resolver-xhtm4"
Sep 17 15:54:08.363301 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.363071    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8910d933-4fe9-45a0-be48-f225cded154a-registration-dir\") pod \"aws-ebs-csi-driver-node-65v88\" (UID: \"8910d933-4fe9-45a0-be48-f225cded154a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65v88"
Sep 17 15:54:08.363301 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.363097    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8910d933-4fe9-45a0-be48-f225cded154a-device-dir\") pod \"aws-ebs-csi-driver-node-65v88\" (UID: \"8910d933-4fe9-45a0-be48-f225cded154a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65v88"
Sep 17 15:54:08.363301 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.363120    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8910d933-4fe9-45a0-be48-f225cded154a-sys-fs\") pod \"aws-ebs-csi-driver-node-65v88\" (UID: \"8910d933-4fe9-45a0-be48-f225cded154a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65v88"
Sep 17 15:54:08.363721 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.363203    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2ecfa8d8-297d-4e71-b29c-726646be98bf-etc-sysctl-d\") pod \"tuned-cdgrj\" (UID: \"2ecfa8d8-297d-4e71-b29c-726646be98bf\") " pod="openshift-cluster-node-tuning-operator/tuned-cdgrj"
Sep 17 15:54:08.363721 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.363548    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2ecfa8d8-297d-4e71-b29c-726646be98bf-run\") pod \"tuned-cdgrj\" (UID: \"2ecfa8d8-297d-4e71-b29c-726646be98bf\") " pod="openshift-cluster-node-tuning-operator/tuned-cdgrj"
Sep 17 15:54:08.363721 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.363577    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gghb\" (UniqueName: \"kubernetes.io/projected/c4438a49-8ea1-40aa-bdd9-2707192617af-kube-api-access-7gghb\") pod \"node-resolver-xhtm4\" (UID: \"c4438a49-8ea1-40aa-bdd9-2707192617af\") " pod="openshift-dns/node-resolver-xhtm4"
Sep 17 15:54:08.363721 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.363600    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2ecfa8d8-297d-4e71-b29c-726646be98bf-etc-systemd\") pod \"tuned-cdgrj\" (UID: \"2ecfa8d8-297d-4e71-b29c-726646be98bf\") " pod="openshift-cluster-node-tuning-operator/tuned-cdgrj"
Sep 17 15:54:08.363721 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.363627    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs87t\" (UniqueName: \"kubernetes.io/projected/3114c3ad-7b7e-41ad-b175-ead7d92d5ee3-kube-api-access-rs87t\") pod \"network-check-target-9mgwn\" (UID: \"3114c3ad-7b7e-41ad-b175-ead7d92d5ee3\") " pod="openshift-network-diagnostics/network-check-target-9mgwn"
Sep 17 15:54:08.363721 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.363651    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86nh5\" (UniqueName: \"kubernetes.io/projected/28231271-38ce-4985-9020-e3494fd366c7-kube-api-access-86nh5\") pod \"node-ca-dtbtf\" (UID: \"28231271-38ce-4985-9020-e3494fd366c7\") " pod="openshift-image-registry/node-ca-dtbtf"
Sep 17 15:54:08.363721 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.363677    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8910d933-4fe9-45a0-be48-f225cded154a-socket-dir\") pod \"aws-ebs-csi-driver-node-65v88\" (UID: \"8910d933-4fe9-45a0-be48-f225cded154a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65v88"
Sep 17 15:54:08.363721 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.363721    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgk99\" (UniqueName: \"kubernetes.io/projected/8910d933-4fe9-45a0-be48-f225cded154a-kube-api-access-qgk99\") pod \"aws-ebs-csi-driver-node-65v88\" (UID: \"8910d933-4fe9-45a0-be48-f225cded154a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65v88"
Sep 17 15:54:08.364087 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.363772    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ecfa8d8-297d-4e71-b29c-726646be98bf-tmp\") pod \"tuned-cdgrj\" (UID: \"2ecfa8d8-297d-4e71-b29c-726646be98bf\") " pod="openshift-cluster-node-tuning-operator/tuned-cdgrj"
Sep 17 15:54:08.364087 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.363796    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8910d933-4fe9-45a0-be48-f225cded154a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-65v88\" (UID: \"8910d933-4fe9-45a0-be48-f225cded154a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65v88"
Sep 17 15:54:08.364087 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.363828    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2ecfa8d8-297d-4e71-b29c-726646be98bf-etc-tuned\") pod \"tuned-cdgrj\" (UID: \"2ecfa8d8-297d-4e71-b29c-726646be98bf\") " pod="openshift-cluster-node-tuning-operator/tuned-cdgrj"
Sep 17 15:54:08.364087 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.363854    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4438a49-8ea1-40aa-bdd9-2707192617af-tmp-dir\") pod \"node-resolver-xhtm4\" (UID: \"c4438a49-8ea1-40aa-bdd9-2707192617af\") " pod="openshift-dns/node-resolver-xhtm4"
Sep 17 15:54:08.364087 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.363890    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/28231271-38ce-4985-9020-e3494fd366c7-serviceca\") pod \"node-ca-dtbtf\" (UID: \"28231271-38ce-4985-9020-e3494fd366c7\") " pod="openshift-image-registry/node-ca-dtbtf"
Sep 17 15:54:08.364087 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.364030    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2ecfa8d8-297d-4e71-b29c-726646be98bf-etc-sysconfig\") pod \"tuned-cdgrj\" (UID: \"2ecfa8d8-297d-4e71-b29c-726646be98bf\") " pod="openshift-cluster-node-tuning-operator/tuned-cdgrj"
Sep 17 15:54:08.364087 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.364066    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ecfa8d8-297d-4e71-b29c-726646be98bf-etc-kubernetes\") pod \"tuned-cdgrj\" (UID: \"2ecfa8d8-297d-4e71-b29c-726646be98bf\") " pod="openshift-cluster-node-tuning-operator/tuned-cdgrj"
Sep 17 15:54:08.364087 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.364088    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2ecfa8d8-297d-4e71-b29c-726646be98bf-etc-sysctl-conf\") pod \"tuned-cdgrj\" (UID: \"2ecfa8d8-297d-4e71-b29c-726646be98bf\") " pod="openshift-cluster-node-tuning-operator/tuned-cdgrj"
Sep 17 15:54:08.364482 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.364111    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ecfa8d8-297d-4e71-b29c-726646be98bf-var-lib-kubelet\") pod \"tuned-cdgrj\" (UID: \"2ecfa8d8-297d-4e71-b29c-726646be98bf\") " pod="openshift-cluster-node-tuning-operator/tuned-cdgrj"
Sep 17 15:54:08.364482 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.364141    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/28231271-38ce-4985-9020-e3494fd366c7-host\") pod \"node-ca-dtbtf\" (UID: \"28231271-38ce-4985-9020-e3494fd366c7\") " pod="openshift-image-registry/node-ca-dtbtf"
Sep 17 15:54:08.364756 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.364737    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6w9z"
Sep 17 15:54:08.364840 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:08.364820    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m6w9z" podUID="7adc5269-e778-473a-a6c7-04754f5e585c"
Sep 17 15:54:08.365581 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.365562    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 15:54:08.366870 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.366450    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 15:54:08.366870 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.366647    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 15:54:08.367007 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.366931    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lvsrt\""
Sep 17 15:54:08.367007 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.366986    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 15:54:08.367271 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.367209    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 15:54:08.367442 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.367409    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 15:54:08.367596 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.367578    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 15:54:08.367671 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.367410    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mhnzn\""
Sep 17 15:54:08.367671 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.367469    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 15:54:08.367772 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.367512    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 15:54:08.367772 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.367522    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 15:54:08.367869 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.367532    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 15:54:08.377691 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.377670    2585 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 15:54:08.413991 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.413968    2585 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-mph7f"
Sep 17 15:54:08.423272 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.423251    2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-mph7f"
Sep 17 15:54:08.445578 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.445551    2585 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 15:54:08.464402 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.464359    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4afdccbc-d33e-41ac-98fd-63585ff4d59b-hostroot\") pod \"multus-xhrb8\" (UID: \"4afdccbc-d33e-41ac-98fd-63585ff4d59b\") " pod="openshift-multus/multus-xhrb8"
Sep 17 15:54:08.464574 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.464448    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a1850a4d-4d01-4a71-8636-b5918f1426c2-ovnkube-script-lib\") pod \"ovnkube-node-2b47p\" (UID: \"a1850a4d-4d01-4a71-8636-b5918f1426c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.464574 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.464497    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7e89f60d-872c-4594-8555-427b0be13b7b-cni-binary-copy\") pod \"multus-additional-cni-plugins-cm82k\" (UID: \"7e89f60d-872c-4594-8555-427b0be13b7b\") " pod="openshift-multus/multus-additional-cni-plugins-cm82k"
Sep 17 15:54:08.464574 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.464528    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2ecfa8d8-297d-4e71-b29c-726646be98bf-etc-modprobe-d\") pod \"tuned-cdgrj\" (UID: \"2ecfa8d8-297d-4e71-b29c-726646be98bf\") " pod="openshift-cluster-node-tuning-operator/tuned-cdgrj"
Sep 17 15:54:08.464574 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.464552    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2ecfa8d8-297d-4e71-b29c-726646be98bf-sys\") pod \"tuned-cdgrj\" (UID: \"2ecfa8d8-297d-4e71-b29c-726646be98bf\") " pod="openshift-cluster-node-tuning-operator/tuned-cdgrj"
Sep 17 15:54:08.464754 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.464581    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1850a4d-4d01-4a71-8636-b5918f1426c2-etc-openvswitch\") pod \"ovnkube-node-2b47p\" (UID: \"a1850a4d-4d01-4a71-8636-b5918f1426c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.464754 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.464661    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1850a4d-4d01-4a71-8636-b5918f1426c2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2b47p\" (UID: \"a1850a4d-4d01-4a71-8636-b5918f1426c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.464754 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.464682    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2ecfa8d8-297d-4e71-b29c-726646be98bf-etc-modprobe-d\") pod \"tuned-cdgrj\" (UID: \"2ecfa8d8-297d-4e71-b29c-726646be98bf\") " pod="openshift-cluster-node-tuning-operator/tuned-cdgrj"
Sep 17 15:54:08.464754 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.464685    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2ecfa8d8-297d-4e71-b29c-726646be98bf-sys\") pod \"tuned-cdgrj\" (UID: \"2ecfa8d8-297d-4e71-b29c-726646be98bf\") " pod="openshift-cluster-node-tuning-operator/tuned-cdgrj"
Sep 17 15:54:08.464754 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.464687    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxxh9\" (UniqueName: \"kubernetes.io/projected/a1850a4d-4d01-4a71-8636-b5918f1426c2-kube-api-access-dxxh9\") pod \"ovnkube-node-2b47p\" (UID: \"a1850a4d-4d01-4a71-8636-b5918f1426c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.464754 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.464739    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/94d1451b-3eae-4481-a504-c29b4315e62e-dbus\") pod \"global-pull-secret-syncer-2f4bn\" (UID: \"94d1451b-3eae-4481-a504-c29b4315e62e\") " pod="kube-system/global-pull-secret-syncer-2f4bn"
Sep 17 15:54:08.465117 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.464768    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zz86\" (UniqueName: \"kubernetes.io/projected/4afdccbc-d33e-41ac-98fd-63585ff4d59b-kube-api-access-4zz86\") pod \"multus-xhrb8\" (UID: \"4afdccbc-d33e-41ac-98fd-63585ff4d59b\") " pod="openshift-multus/multus-xhrb8"
Sep 17 15:54:08.465117 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.464814    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a1850a4d-4d01-4a71-8636-b5918f1426c2-ovnkube-config\") pod \"ovnkube-node-2b47p\" (UID: \"a1850a4d-4d01-4a71-8636-b5918f1426c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.465117 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.464845    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/66be23a4-8549-4ae3-9d3e-33aca9dca8cb-host-slash\") pod \"iptables-alerter-mvssh\" (UID: \"66be23a4-8549-4ae3-9d3e-33aca9dca8cb\") " pod="openshift-network-operator/iptables-alerter-mvssh"
Sep 17 15:54:08.465117 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.464875    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m449m\" (UniqueName: \"kubernetes.io/projected/7adc5269-e778-473a-a6c7-04754f5e585c-kube-api-access-m449m\") pod \"network-metrics-daemon-m6w9z\" (UID: \"7adc5269-e778-473a-a6c7-04754f5e585c\") " pod="openshift-multus/network-metrics-daemon-m6w9z"
Sep 17 15:54:08.465117 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.464911    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c4438a49-8ea1-40aa-bdd9-2707192617af-hosts-file\") pod \"node-resolver-xhtm4\" (UID: \"c4438a49-8ea1-40aa-bdd9-2707192617af\") " pod="openshift-dns/node-resolver-xhtm4"
Sep 17 15:54:08.465117 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.464975    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c4438a49-8ea1-40aa-bdd9-2707192617af-hosts-file\") pod \"node-resolver-xhtm4\" (UID: \"c4438a49-8ea1-40aa-bdd9-2707192617af\") " pod="openshift-dns/node-resolver-xhtm4"
Sep 17 15:54:08.465117 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.464976    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8910d933-4fe9-45a0-be48-f225cded154a-sys-fs\") pod \"aws-ebs-csi-driver-node-65v88\" (UID: \"8910d933-4fe9-45a0-be48-f225cded154a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65v88"
Sep 17 15:54:08.465117 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.465018    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a1850a4d-4d01-4a71-8636-b5918f1426c2-host-run-netns\") pod \"ovnkube-node-2b47p\" (UID: \"a1850a4d-4d01-4a71-8636-b5918f1426c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.465117 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.465033    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8910d933-4fe9-45a0-be48-f225cded154a-sys-fs\") pod \"aws-ebs-csi-driver-node-65v88\" (UID: \"8910d933-4fe9-45a0-be48-f225cded154a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65v88"
Sep 17 15:54:08.465117 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.465041    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a1850a4d-4d01-4a71-8636-b5918f1426c2-run-ovn\") pod \"ovnkube-node-2b47p\" (UID: \"a1850a4d-4d01-4a71-8636-b5918f1426c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.465570 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.465446    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7e89f60d-872c-4594-8555-427b0be13b7b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cm82k\" (UID: \"7e89f60d-872c-4594-8555-427b0be13b7b\") " pod="openshift-multus/multus-additional-cni-plugins-cm82k"
Sep 17 15:54:08.465570 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.465494    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2ecfa8d8-297d-4e71-b29c-726646be98bf-etc-sysctl-d\") pod \"tuned-cdgrj\" (UID: \"2ecfa8d8-297d-4e71-b29c-726646be98bf\") " pod="openshift-cluster-node-tuning-operator/tuned-cdgrj"
Sep 17 15:54:08.465664 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.465576    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4afdccbc-d33e-41ac-98fd-63585ff4d59b-multus-cni-dir\") pod \"multus-xhrb8\" (UID: \"4afdccbc-d33e-41ac-98fd-63585ff4d59b\") " pod="openshift-multus/multus-xhrb8"
Sep 17 15:54:08.465664 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.465611    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4afdccbc-d33e-41ac-98fd-63585ff4d59b-cni-binary-copy\") pod \"multus-xhrb8\" (UID: \"4afdccbc-d33e-41ac-98fd-63585ff4d59b\") " pod="openshift-multus/multus-xhrb8"
Sep 17 15:54:08.465664 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.465641    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4afdccbc-d33e-41ac-98fd-63585ff4d59b-host-var-lib-cni-bin\") pod \"multus-xhrb8\" (UID: \"4afdccbc-d33e-41ac-98fd-63585ff4d59b\") " pod="openshift-multus/multus-xhrb8"
Sep 17 15:54:08.465791 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.465670    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4afdccbc-d33e-41ac-98fd-63585ff4d59b-host-var-lib-kubelet\") pod \"multus-xhrb8\" (UID: \"4afdccbc-d33e-41ac-98fd-63585ff4d59b\") " pod="openshift-multus/multus-xhrb8"
Sep 17 15:54:08.465791 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.465694    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1850a4d-4d01-4a71-8636-b5918f1426c2-host-run-ovn-kubernetes\") pod \"ovnkube-node-2b47p\" (UID: \"a1850a4d-4d01-4a71-8636-b5918f1426c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.465791 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.465726    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs87t\" (UniqueName: \"kubernetes.io/projected/3114c3ad-7b7e-41ad-b175-ead7d92d5ee3-kube-api-access-rs87t\") pod \"network-check-target-9mgwn\" (UID: \"3114c3ad-7b7e-41ad-b175-ead7d92d5ee3\") " pod="openshift-network-diagnostics/network-check-target-9mgwn"
Sep 17 15:54:08.465791 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.465784    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8910d933-4fe9-45a0-be48-f225cded154a-socket-dir\") pod \"aws-ebs-csi-driver-node-65v88\" (UID: \"8910d933-4fe9-45a0-be48-f225cded154a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65v88"
Sep 17 15:54:08.465975 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.465856    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2ecfa8d8-297d-4e71-b29c-726646be98bf-etc-sysctl-d\") pod \"tuned-cdgrj\" (UID: \"2ecfa8d8-297d-4e71-b29c-726646be98bf\") " pod="openshift-cluster-node-tuning-operator/tuned-cdgrj"
Sep 17 15:54:08.465975 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.465812    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgk99\" (UniqueName: \"kubernetes.io/projected/8910d933-4fe9-45a0-be48-f225cded154a-kube-api-access-qgk99\") pod \"aws-ebs-csi-driver-node-65v88\" (UID: \"8910d933-4fe9-45a0-be48-f225cded154a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65v88"
Sep 17 15:54:08.466055 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.465991    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4afdccbc-d33e-41ac-98fd-63585ff4d59b-host-run-netns\") pod \"multus-xhrb8\" (UID: \"4afdccbc-d33e-41ac-98fd-63585ff4d59b\") " pod="openshift-multus/multus-xhrb8"
Sep 17 15:54:08.466055 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.465986    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8910d933-4fe9-45a0-be48-f225cded154a-socket-dir\") pod \"aws-ebs-csi-driver-node-65v88\" (UID: \"8910d933-4fe9-45a0-be48-f225cded154a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65v88"
Sep 17 15:54:08.466055 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.466028    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4afdccbc-d33e-41ac-98fd-63585ff4d59b-host-run-multus-certs\") pod \"multus-xhrb8\" (UID: \"4afdccbc-d33e-41ac-98fd-63585ff4d59b\") " pod="openshift-multus/multus-xhrb8"
Sep 17 15:54:08.466190 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.466087    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a1850a4d-4d01-4a71-8636-b5918f1426c2-run-systemd\") pod \"ovnkube-node-2b47p\" (UID: \"a1850a4d-4d01-4a71-8636-b5918f1426c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.466190 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.466143    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1850a4d-4d01-4a71-8636-b5918f1426c2-run-openvswitch\") pod \"ovnkube-node-2b47p\" (UID: \"a1850a4d-4d01-4a71-8636-b5918f1426c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.466190 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.466171    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7e89f60d-872c-4594-8555-427b0be13b7b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cm82k\" (UID: \"7e89f60d-872c-4594-8555-427b0be13b7b\") " pod="openshift-multus/multus-additional-cni-plugins-cm82k"
Sep 17 15:54:08.466327 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.466204    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2ecfa8d8-297d-4e71-b29c-726646be98bf-etc-tuned\") pod \"tuned-cdgrj\" (UID: \"2ecfa8d8-297d-4e71-b29c-726646be98bf\") " pod="openshift-cluster-node-tuning-operator/tuned-cdgrj"
Sep 17 15:54:08.466327 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.466236    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4438a49-8ea1-40aa-bdd9-2707192617af-tmp-dir\") pod \"node-resolver-xhtm4\" (UID: \"c4438a49-8ea1-40aa-bdd9-2707192617af\") " pod="openshift-dns/node-resolver-xhtm4"
Sep 17 15:54:08.466445 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.466332    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4afdccbc-d33e-41ac-98fd-63585ff4d59b-system-cni-dir\") pod \"multus-xhrb8\" (UID: \"4afdccbc-d33e-41ac-98fd-63585ff4d59b\") " pod="openshift-multus/multus-xhrb8"
Sep 17 15:54:08.466445 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.466372    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4afdccbc-d33e-41ac-98fd-63585ff4d59b-cnibin\") pod \"multus-xhrb8\" (UID: \"4afdccbc-d33e-41ac-98fd-63585ff4d59b\") " pod="openshift-multus/multus-xhrb8"
Sep 17 15:54:08.466445 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.466439    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a1850a4d-4d01-4a71-8636-b5918f1426c2-host-slash\") pod \"ovnkube-node-2b47p\" (UID: \"a1850a4d-4d01-4a71-8636-b5918f1426c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.466617 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.466474    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a1850a4d-4d01-4a71-8636-b5918f1426c2-log-socket\") pod \"ovnkube-node-2b47p\" (UID: \"a1850a4d-4d01-4a71-8636-b5918f1426c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.466617 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.466502    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7e89f60d-872c-4594-8555-427b0be13b7b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cm82k\" (UID: \"7e89f60d-872c-4594-8555-427b0be13b7b\") " pod="openshift-multus/multus-additional-cni-plugins-cm82k"
Sep 17 15:54:08.466617 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.466557    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2ecfa8d8-297d-4e71-b29c-726646be98bf-etc-sysconfig\") pod \"tuned-cdgrj\" (UID: \"2ecfa8d8-297d-4e71-b29c-726646be98bf\") " pod="openshift-cluster-node-tuning-operator/tuned-cdgrj"
Sep 17 15:54:08.466617 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.466584    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ecfa8d8-297d-4e71-b29c-726646be98bf-var-lib-kubelet\") pod \"tuned-cdgrj\" (UID: \"2ecfa8d8-297d-4e71-b29c-726646be98bf\") " pod="openshift-cluster-node-tuning-operator/tuned-cdgrj"
Sep 17 15:54:08.466816 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.466682    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4afdccbc-d33e-41ac-98fd-63585ff4d59b-etc-kubernetes\") pod \"multus-xhrb8\" (UID: \"4afdccbc-d33e-41ac-98fd-63585ff4d59b\") " pod="openshift-multus/multus-xhrb8"
Sep 17 15:54:08.466816 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.466716    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/94d1451b-3eae-4481-a504-c29b4315e62e-kubelet-config\") pod \"global-pull-secret-syncer-2f4bn\" (UID: \"94d1451b-3eae-4481-a504-c29b4315e62e\") " pod="kube-system/global-pull-secret-syncer-2f4bn"
Sep 17 15:54:08.466816 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.466748    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2ecfa8d8-297d-4e71-b29c-726646be98bf-host\") pod \"tuned-cdgrj\" (UID: \"2ecfa8d8-297d-4e71-b29c-726646be98bf\") " pod="openshift-cluster-node-tuning-operator/tuned-cdgrj"
Sep 17 15:54:08.466816 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.466761    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2ecfa8d8-297d-4e71-b29c-726646be98bf-etc-sysconfig\") pod \"tuned-cdgrj\" (UID: \"2ecfa8d8-297d-4e71-b29c-726646be98bf\") " pod="openshift-cluster-node-tuning-operator/tuned-cdgrj"
Sep 17 15:54:08.466816 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.466793    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj46v\" (UniqueName: \"kubernetes.io/projected/2ecfa8d8-297d-4e71-b29c-726646be98bf-kube-api-access-sj46v\") pod \"tuned-cdgrj\" (UID: \"2ecfa8d8-297d-4e71-b29c-726646be98bf\") " pod="openshift-cluster-node-tuning-operator/tuned-cdgrj"
Sep 17 15:54:08.467038 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.466848    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2ecfa8d8-297d-4e71-b29c-726646be98bf-host\") pod \"tuned-cdgrj\" (UID: \"2ecfa8d8-297d-4e71-b29c-726646be98bf\") " pod="openshift-cluster-node-tuning-operator/tuned-cdgrj"
Sep 17 15:54:08.467038 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.466821    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8249ef17-a5c4-414e-a2ca-a22cc46ef3eb-agent-certs\") pod \"konnectivity-agent-ntl9t\" (UID: \"8249ef17-a5c4-414e-a2ca-a22cc46ef3eb\") " pod="kube-system/konnectivity-agent-ntl9t"
Sep 17 15:54:08.467151 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.467130    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4438a49-8ea1-40aa-bdd9-2707192617af-tmp-dir\") pod \"node-resolver-xhtm4\" (UID: \"c4438a49-8ea1-40aa-bdd9-2707192617af\") " pod="openshift-dns/node-resolver-xhtm4"
Sep 17 15:54:08.467253 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.467232    2585 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 15:54:08.467498 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.467474    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a1850a4d-4d01-4a71-8636-b5918f1426c2-host-cni-bin\") pod \"ovnkube-node-2b47p\" (UID: \"a1850a4d-4d01-4a71-8636-b5918f1426c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.467560 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.467515    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/94d1451b-3eae-4481-a504-c29b4315e62e-original-pull-secret\") pod \"global-pull-secret-syncer-2f4bn\" (UID: \"94d1451b-3eae-4481-a504-c29b4315e62e\") " pod="kube-system/global-pull-secret-syncer-2f4bn"
Sep 17 15:54:08.467560 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.467549    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/66be23a4-8549-4ae3-9d3e-33aca9dca8cb-iptables-alerter-script\") pod \"iptables-alerter-mvssh\" (UID: \"66be23a4-8549-4ae3-9d3e-33aca9dca8cb\") " pod="openshift-network-operator/iptables-alerter-mvssh"
Sep 17 15:54:08.467648 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.467583    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4afdccbc-d33e-41ac-98fd-63585ff4d59b-host-var-lib-cni-multus\") pod \"multus-xhrb8\" (UID: \"4afdccbc-d33e-41ac-98fd-63585ff4d59b\") " pod="openshift-multus/multus-xhrb8"
Sep 17 15:54:08.467648 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.467613    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4afdccbc-d33e-41ac-98fd-63585ff4d59b-multus-conf-dir\") pod \"multus-xhrb8\" (UID: \"4afdccbc-d33e-41ac-98fd-63585ff4d59b\") " pod="openshift-multus/multus-xhrb8"
Sep 17 15:54:08.467648 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.467643    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a1850a4d-4d01-4a71-8636-b5918f1426c2-systemd-units\") pod \"ovnkube-node-2b47p\" (UID: \"a1850a4d-4d01-4a71-8636-b5918f1426c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.467778 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.467735    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2ecfa8d8-297d-4e71-b29c-726646be98bf-lib-modules\") pod \"tuned-cdgrj\" (UID: \"2ecfa8d8-297d-4e71-b29c-726646be98bf\") " pod="openshift-cluster-node-tuning-operator/tuned-cdgrj"
Sep 17 15:54:08.467778 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.467767    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8910d933-4fe9-45a0-be48-f225cded154a-registration-dir\") pod \"aws-ebs-csi-driver-node-65v88\" (UID: \"8910d933-4fe9-45a0-be48-f225cded154a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65v88"
Sep 17 15:54:08.467872 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.467792    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8910d933-4fe9-45a0-be48-f225cded154a-device-dir\") pod \"aws-ebs-csi-driver-node-65v88\" (UID: \"8910d933-4fe9-45a0-be48-f225cded154a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65v88"
Sep 17 15:54:08.467872 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.467826    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4afdccbc-d33e-41ac-98fd-63585ff4d59b-multus-socket-dir-parent\") pod \"multus-xhrb8\" (UID: \"4afdccbc-d33e-41ac-98fd-63585ff4d59b\") " pod="openshift-multus/multus-xhrb8"
Sep 17 15:54:08.467872 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.467858    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4afdccbc-d33e-41ac-98fd-63585ff4d59b-multus-daemon-config\") pod \"multus-xhrb8\" (UID: \"4afdccbc-d33e-41ac-98fd-63585ff4d59b\") " pod="openshift-multus/multus-xhrb8"
Sep 17 15:54:08.468008 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.467887    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a1850a4d-4d01-4a71-8636-b5918f1426c2-node-log\") pod \"ovnkube-node-2b47p\" (UID: \"a1850a4d-4d01-4a71-8636-b5918f1426c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.468008 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.467918    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a1850a4d-4d01-4a71-8636-b5918f1426c2-env-overrides\") pod \"ovnkube-node-2b47p\" (UID: \"a1850a4d-4d01-4a71-8636-b5918f1426c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.468008 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.467946    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a1850a4d-4d01-4a71-8636-b5918f1426c2-ovn-node-metrics-cert\") pod \"ovnkube-node-2b47p\" (UID: \"a1850a4d-4d01-4a71-8636-b5918f1426c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.468008 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.467977    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2ecfa8d8-297d-4e71-b29c-726646be98bf-run\") pod \"tuned-cdgrj\" (UID: \"2ecfa8d8-297d-4e71-b29c-726646be98bf\") " pod="openshift-cluster-node-tuning-operator/tuned-cdgrj"
Sep 17 15:54:08.468214 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.468009    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gghb\" (UniqueName: \"kubernetes.io/projected/c4438a49-8ea1-40aa-bdd9-2707192617af-kube-api-access-7gghb\") pod \"node-resolver-xhtm4\" (UID: \"c4438a49-8ea1-40aa-bdd9-2707192617af\") " pod="openshift-dns/node-resolver-xhtm4"
Sep 17 15:54:08.468214 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.468040    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4afdccbc-d33e-41ac-98fd-63585ff4d59b-host-run-k8s-cni-cncf-io\") pod \"multus-xhrb8\" (UID: \"4afdccbc-d33e-41ac-98fd-63585ff4d59b\") " pod="openshift-multus/multus-xhrb8"
Sep 17 15:54:08.468214 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.468074    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbw4w\" (UniqueName: \"kubernetes.io/projected/66be23a4-8549-4ae3-9d3e-33aca9dca8cb-kube-api-access-pbw4w\") pod \"iptables-alerter-mvssh\" (UID: \"66be23a4-8549-4ae3-9d3e-33aca9dca8cb\") " pod="openshift-network-operator/iptables-alerter-mvssh"
Sep 17 15:54:08.468214 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.468194    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc6t2\" (UniqueName: \"kubernetes.io/projected/7e89f60d-872c-4594-8555-427b0be13b7b-kube-api-access-qc6t2\") pod \"multus-additional-cni-plugins-cm82k\" (UID: \"7e89f60d-872c-4594-8555-427b0be13b7b\") " pod="openshift-multus/multus-additional-cni-plugins-cm82k"
Sep 17 15:54:08.468386 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.468223    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2ecfa8d8-297d-4e71-b29c-726646be98bf-etc-systemd\") pod \"tuned-cdgrj\" (UID: \"2ecfa8d8-297d-4e71-b29c-726646be98bf\") " pod="openshift-cluster-node-tuning-operator/tuned-cdgrj"
Sep 17 15:54:08.468386 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.468256    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86nh5\" (UniqueName: \"kubernetes.io/projected/28231271-38ce-4985-9020-e3494fd366c7-kube-api-access-86nh5\") pod \"node-ca-dtbtf\" (UID: \"28231271-38ce-4985-9020-e3494fd366c7\") " pod="openshift-image-registry/node-ca-dtbtf"
Sep 17 15:54:08.468386 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.468290    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a1850a4d-4d01-4a71-8636-b5918f1426c2-host-cni-netd\") pod \"ovnkube-node-2b47p\" (UID: \"a1850a4d-4d01-4a71-8636-b5918f1426c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.468386 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.468319    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7e89f60d-872c-4594-8555-427b0be13b7b-cnibin\") pod \"multus-additional-cni-plugins-cm82k\" (UID: \"7e89f60d-872c-4594-8555-427b0be13b7b\") " pod="openshift-multus/multus-additional-cni-plugins-cm82k"
Sep 17 15:54:08.468386 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.468349    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ecfa8d8-297d-4e71-b29c-726646be98bf-tmp\") pod \"tuned-cdgrj\" (UID: \"2ecfa8d8-297d-4e71-b29c-726646be98bf\") " pod="openshift-cluster-node-tuning-operator/tuned-cdgrj"
Sep 17 15:54:08.468386 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.468381    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8910d933-4fe9-45a0-be48-f225cded154a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-65v88\" (UID: \"8910d933-4fe9-45a0-be48-f225cded154a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65v88"
Sep 17 15:54:08.468677 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.468408    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1850a4d-4d01-4a71-8636-b5918f1426c2-host-kubelet\") pod \"ovnkube-node-2b47p\" (UID: \"a1850a4d-4d01-4a71-8636-b5918f1426c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.468677 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.468451    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1850a4d-4d01-4a71-8636-b5918f1426c2-var-lib-openvswitch\") pod \"ovnkube-node-2b47p\" (UID: \"a1850a4d-4d01-4a71-8636-b5918f1426c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.468677 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.468482    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e89f60d-872c-4594-8555-427b0be13b7b-system-cni-dir\") pod \"multus-additional-cni-plugins-cm82k\" (UID: \"7e89f60d-872c-4594-8555-427b0be13b7b\") " pod="openshift-multus/multus-additional-cni-plugins-cm82k"
Sep 17 15:54:08.468677 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.468515    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/28231271-38ce-4985-9020-e3494fd366c7-serviceca\") pod \"node-ca-dtbtf\" (UID: \"28231271-38ce-4985-9020-e3494fd366c7\") " pod="openshift-image-registry/node-ca-dtbtf"
Sep 17 15:54:08.468677 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.468545    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7adc5269-e778-473a-a6c7-04754f5e585c-metrics-certs\") pod \"network-metrics-daemon-m6w9z\" (UID: \"7adc5269-e778-473a-a6c7-04754f5e585c\") " pod="openshift-multus/network-metrics-daemon-m6w9z"
Sep 17 15:54:08.468677 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.468576    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ecfa8d8-297d-4e71-b29c-726646be98bf-etc-kubernetes\") pod \"tuned-cdgrj\" (UID: \"2ecfa8d8-297d-4e71-b29c-726646be98bf\") " pod="openshift-cluster-node-tuning-operator/tuned-cdgrj"
Sep 17 15:54:08.468677 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.468608    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2ecfa8d8-297d-4e71-b29c-726646be98bf-etc-sysctl-conf\") pod \"tuned-cdgrj\" (UID: \"2ecfa8d8-297d-4e71-b29c-726646be98bf\") " pod="openshift-cluster-node-tuning-operator/tuned-cdgrj"
Sep 17 15:54:08.468677 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.468638    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/28231271-38ce-4985-9020-e3494fd366c7-host\") pod \"node-ca-dtbtf\" (UID: \"28231271-38ce-4985-9020-e3494fd366c7\") " pod="openshift-image-registry/node-ca-dtbtf"
Sep 17 15:54:08.468677 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.468670    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8910d933-4fe9-45a0-be48-f225cded154a-etc-selinux\") pod \"aws-ebs-csi-driver-node-65v88\" (UID: \"8910d933-4fe9-45a0-be48-f225cded154a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65v88"
Sep 17 15:54:08.469056 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.468701    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8249ef17-a5c4-414e-a2ca-a22cc46ef3eb-konnectivity-ca\") pod \"konnectivity-agent-ntl9t\" (UID: \"8249ef17-a5c4-414e-a2ca-a22cc46ef3eb\") " pod="kube-system/konnectivity-agent-ntl9t"
Sep 17 15:54:08.469325 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.469304    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8249ef17-a5c4-414e-a2ca-a22cc46ef3eb-konnectivity-ca\") pod \"konnectivity-agent-ntl9t\" (UID: \"8249ef17-a5c4-414e-a2ca-a22cc46ef3eb\") " pod="kube-system/konnectivity-agent-ntl9t"
Sep 17 15:54:08.469456 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.469438    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ecfa8d8-297d-4e71-b29c-726646be98bf-var-lib-kubelet\") pod \"tuned-cdgrj\" (UID: \"2ecfa8d8-297d-4e71-b29c-726646be98bf\") " pod="openshift-cluster-node-tuning-operator/tuned-cdgrj"
Sep 17 15:54:08.469524 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.469462    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2ecfa8d8-297d-4e71-b29c-726646be98bf-run\") pod \"tuned-cdgrj\" (UID: \"2ecfa8d8-297d-4e71-b29c-726646be98bf\") " pod="openshift-cluster-node-tuning-operator/tuned-cdgrj"
Sep 17 15:54:08.469583 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.469512    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2ecfa8d8-297d-4e71-b29c-726646be98bf-lib-modules\") pod \"tuned-cdgrj\" (UID: \"2ecfa8d8-297d-4e71-b29c-726646be98bf\") " pod="openshift-cluster-node-tuning-operator/tuned-cdgrj"
Sep 17 15:54:08.469583 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.469569    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ecfa8d8-297d-4e71-b29c-726646be98bf-etc-kubernetes\") pod \"tuned-cdgrj\" (UID: \"2ecfa8d8-297d-4e71-b29c-726646be98bf\") " pod="openshift-cluster-node-tuning-operator/tuned-cdgrj"
Sep 17 15:54:08.469675 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.469567    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2ecfa8d8-297d-4e71-b29c-726646be98bf-etc-systemd\") pod \"tuned-cdgrj\" (UID: \"2ecfa8d8-297d-4e71-b29c-726646be98bf\") " pod="openshift-cluster-node-tuning-operator/tuned-cdgrj"
Sep 17 15:54:08.469675 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.469586    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8910d933-4fe9-45a0-be48-f225cded154a-registration-dir\") pod \"aws-ebs-csi-driver-node-65v88\" (UID: \"8910d933-4fe9-45a0-be48-f225cded154a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65v88"
Sep 17 15:54:08.469675 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.469636    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/28231271-38ce-4985-9020-e3494fd366c7-host\") pod \"node-ca-dtbtf\" (UID: \"28231271-38ce-4985-9020-e3494fd366c7\") " pod="openshift-image-registry/node-ca-dtbtf"
Sep 17 15:54:08.469814 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.469674    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8910d933-4fe9-45a0-be48-f225cded154a-etc-selinux\") pod \"aws-ebs-csi-driver-node-65v88\" (UID: \"8910d933-4fe9-45a0-be48-f225cded154a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65v88"
Sep 17 15:54:08.469872 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.469810    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8910d933-4fe9-45a0-be48-f225cded154a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-65v88\" (UID: \"8910d933-4fe9-45a0-be48-f225cded154a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65v88"
Sep 17 15:54:08.470636 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.470342    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8910d933-4fe9-45a0-be48-f225cded154a-device-dir\") pod \"aws-ebs-csi-driver-node-65v88\" (UID: \"8910d933-4fe9-45a0-be48-f225cded154a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65v88"
Sep 17 15:54:08.470636 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.470561    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2ecfa8d8-297d-4e71-b29c-726646be98bf-etc-sysctl-conf\") pod \"tuned-cdgrj\" (UID: \"2ecfa8d8-297d-4e71-b29c-726646be98bf\") " pod="openshift-cluster-node-tuning-operator/tuned-cdgrj"
Sep 17 15:54:08.471657 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.470987    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/28231271-38ce-4985-9020-e3494fd366c7-serviceca\") pod \"node-ca-dtbtf\" (UID: \"28231271-38ce-4985-9020-e3494fd366c7\") " pod="openshift-image-registry/node-ca-dtbtf"
Sep 17 15:54:08.472882 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.472837    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2ecfa8d8-297d-4e71-b29c-726646be98bf-etc-tuned\") pod \"tuned-cdgrj\" (UID: \"2ecfa8d8-297d-4e71-b29c-726646be98bf\") " pod="openshift-cluster-node-tuning-operator/tuned-cdgrj"
Sep 17 15:54:08.473112 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.473082    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8249ef17-a5c4-414e-a2ca-a22cc46ef3eb-agent-certs\") pod \"konnectivity-agent-ntl9t\" (UID: \"8249ef17-a5c4-414e-a2ca-a22cc46ef3eb\") " pod="kube-system/konnectivity-agent-ntl9t"
Sep 17 15:54:08.473195 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.473116    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ecfa8d8-297d-4e71-b29c-726646be98bf-tmp\") pod \"tuned-cdgrj\" (UID: \"2ecfa8d8-297d-4e71-b29c-726646be98bf\") " pod="openshift-cluster-node-tuning-operator/tuned-cdgrj"
Sep 17 15:54:08.487747 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.487723    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgk99\" (UniqueName: \"kubernetes.io/projected/8910d933-4fe9-45a0-be48-f225cded154a-kube-api-access-qgk99\") pod \"aws-ebs-csi-driver-node-65v88\" (UID: \"8910d933-4fe9-45a0-be48-f225cded154a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65v88"
Sep 17 15:54:08.490300 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:08.490276    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:54:08.490438 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:08.490305    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:54:08.490438 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:08.490320    2585 projected.go:196] Error preparing data for projected volume kube-api-access-rs87t for pod openshift-network-diagnostics/network-check-target-9mgwn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:54:08.490438 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:08.490388    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3114c3ad-7b7e-41ad-b175-ead7d92d5ee3-kube-api-access-rs87t podName:3114c3ad-7b7e-41ad-b175-ead7d92d5ee3 nodeName:}" failed. No retries permitted until 2026-09-17 15:54:08.990366848 +0000 UTC m=+4.760698644 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rs87t" (UniqueName: "kubernetes.io/projected/3114c3ad-7b7e-41ad-b175-ead7d92d5ee3-kube-api-access-rs87t") pod "network-check-target-9mgwn" (UID: "3114c3ad-7b7e-41ad-b175-ead7d92d5ee3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:54:08.492066 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.492039    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj46v\" (UniqueName: \"kubernetes.io/projected/2ecfa8d8-297d-4e71-b29c-726646be98bf-kube-api-access-sj46v\") pod \"tuned-cdgrj\" (UID: \"2ecfa8d8-297d-4e71-b29c-726646be98bf\") " pod="openshift-cluster-node-tuning-operator/tuned-cdgrj"
Sep 17 15:54:08.492776 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.492751    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86nh5\" (UniqueName: \"kubernetes.io/projected/28231271-38ce-4985-9020-e3494fd366c7-kube-api-access-86nh5\") pod \"node-ca-dtbtf\" (UID: \"28231271-38ce-4985-9020-e3494fd366c7\") " pod="openshift-image-registry/node-ca-dtbtf"
Sep 17 15:54:08.493014 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.492992    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gghb\" (UniqueName: \"kubernetes.io/projected/c4438a49-8ea1-40aa-bdd9-2707192617af-kube-api-access-7gghb\") pod \"node-resolver-xhtm4\" (UID: \"c4438a49-8ea1-40aa-bdd9-2707192617af\") " pod="openshift-dns/node-resolver-xhtm4"
Sep 17 15:54:08.512571 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.512515    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-43.ec2.internal" event={"ID":"7156e3815525514fe59f7c5337670664","Type":"ContainerStarted","Data":"b8271257bc4b7ee8f91a89ed3aa1fbc4694dfd84d3f50b90d23ba0a3547f7ab1"}
Sep 17 15:54:08.513973 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.513946    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-43.ec2.internal" event={"ID":"5692cd509d685463b01f1d043c0bcb98","Type":"ContainerStarted","Data":"fa4b1441f8bed097f53ec5ac1849d8af2b7dd3e5345cc8c6570d4f2bb6406095"}
Sep 17 15:54:08.569162 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.569132    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4afdccbc-d33e-41ac-98fd-63585ff4d59b-multus-socket-dir-parent\") pod \"multus-xhrb8\" (UID: \"4afdccbc-d33e-41ac-98fd-63585ff4d59b\") " pod="openshift-multus/multus-xhrb8"
Sep 17 15:54:08.569162 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.569165    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4afdccbc-d33e-41ac-98fd-63585ff4d59b-multus-daemon-config\") pod \"multus-xhrb8\" (UID: \"4afdccbc-d33e-41ac-98fd-63585ff4d59b\") " pod="openshift-multus/multus-xhrb8"
Sep 17 15:54:08.569387 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.569186    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a1850a4d-4d01-4a71-8636-b5918f1426c2-node-log\") pod \"ovnkube-node-2b47p\" (UID: \"a1850a4d-4d01-4a71-8636-b5918f1426c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.569387 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.569252    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4afdccbc-d33e-41ac-98fd-63585ff4d59b-multus-socket-dir-parent\") pod \"multus-xhrb8\" (UID: \"4afdccbc-d33e-41ac-98fd-63585ff4d59b\") " pod="openshift-multus/multus-xhrb8"
Sep 17 15:54:08.569387 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.569288    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a1850a4d-4d01-4a71-8636-b5918f1426c2-env-overrides\") pod \"ovnkube-node-2b47p\" (UID: \"a1850a4d-4d01-4a71-8636-b5918f1426c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.569387 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.569309    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a1850a4d-4d01-4a71-8636-b5918f1426c2-node-log\") pod \"ovnkube-node-2b47p\" (UID: \"a1850a4d-4d01-4a71-8636-b5918f1426c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.569387 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.569316    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a1850a4d-4d01-4a71-8636-b5918f1426c2-ovn-node-metrics-cert\") pod \"ovnkube-node-2b47p\" (UID: \"a1850a4d-4d01-4a71-8636-b5918f1426c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.569387 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.569351    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4afdccbc-d33e-41ac-98fd-63585ff4d59b-host-run-k8s-cni-cncf-io\") pod \"multus-xhrb8\" (UID: \"4afdccbc-d33e-41ac-98fd-63585ff4d59b\") " pod="openshift-multus/multus-xhrb8"
Sep 17 15:54:08.569387 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.569374    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbw4w\" (UniqueName: \"kubernetes.io/projected/66be23a4-8549-4ae3-9d3e-33aca9dca8cb-kube-api-access-pbw4w\") pod \"iptables-alerter-mvssh\" (UID: \"66be23a4-8549-4ae3-9d3e-33aca9dca8cb\") " pod="openshift-network-operator/iptables-alerter-mvssh"
Sep 17 15:54:08.569727 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.569399    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc6t2\" (UniqueName: \"kubernetes.io/projected/7e89f60d-872c-4594-8555-427b0be13b7b-kube-api-access-qc6t2\") pod \"multus-additional-cni-plugins-cm82k\" (UID: \"7e89f60d-872c-4594-8555-427b0be13b7b\") " pod="openshift-multus/multus-additional-cni-plugins-cm82k"
Sep 17 15:54:08.569727 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.569442    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a1850a4d-4d01-4a71-8636-b5918f1426c2-host-cni-netd\") pod \"ovnkube-node-2b47p\" (UID: \"a1850a4d-4d01-4a71-8636-b5918f1426c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.569727 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.569476    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7e89f60d-872c-4594-8555-427b0be13b7b-cnibin\") pod \"multus-additional-cni-plugins-cm82k\" (UID: \"7e89f60d-872c-4594-8555-427b0be13b7b\") " pod="openshift-multus/multus-additional-cni-plugins-cm82k"
Sep 17 15:54:08.569727 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.569501    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1850a4d-4d01-4a71-8636-b5918f1426c2-host-kubelet\") pod \"ovnkube-node-2b47p\" (UID: \"a1850a4d-4d01-4a71-8636-b5918f1426c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.569727 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.569524    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1850a4d-4d01-4a71-8636-b5918f1426c2-var-lib-openvswitch\") pod \"ovnkube-node-2b47p\" (UID: \"a1850a4d-4d01-4a71-8636-b5918f1426c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.569727 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.569548    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e89f60d-872c-4594-8555-427b0be13b7b-system-cni-dir\") pod \"multus-additional-cni-plugins-cm82k\" (UID: \"7e89f60d-872c-4594-8555-427b0be13b7b\") " pod="openshift-multus/multus-additional-cni-plugins-cm82k"
Sep 17 15:54:08.569727 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.569575    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7adc5269-e778-473a-a6c7-04754f5e585c-metrics-certs\") pod \"network-metrics-daemon-m6w9z\" (UID: \"7adc5269-e778-473a-a6c7-04754f5e585c\") " pod="openshift-multus/network-metrics-daemon-m6w9z"
Sep 17 15:54:08.569727 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.569604    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4afdccbc-d33e-41ac-98fd-63585ff4d59b-hostroot\") pod \"multus-xhrb8\" (UID: \"4afdccbc-d33e-41ac-98fd-63585ff4d59b\") " pod="openshift-multus/multus-xhrb8"
Sep 17 15:54:08.569727 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.569627    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a1850a4d-4d01-4a71-8636-b5918f1426c2-ovnkube-script-lib\") pod \"ovnkube-node-2b47p\" (UID: \"a1850a4d-4d01-4a71-8636-b5918f1426c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.569727 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.569651    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7e89f60d-872c-4594-8555-427b0be13b7b-cni-binary-copy\") pod \"multus-additional-cni-plugins-cm82k\" (UID: \"7e89f60d-872c-4594-8555-427b0be13b7b\") " pod="openshift-multus/multus-additional-cni-plugins-cm82k"
Sep 17 15:54:08.569727 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.569677    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1850a4d-4d01-4a71-8636-b5918f1426c2-etc-openvswitch\") pod \"ovnkube-node-2b47p\" (UID: \"a1850a4d-4d01-4a71-8636-b5918f1426c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.569727 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.569685    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a1850a4d-4d01-4a71-8636-b5918f1426c2-host-cni-netd\") pod \"ovnkube-node-2b47p\" (UID: \"a1850a4d-4d01-4a71-8636-b5918f1426c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.569727 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.569710    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7e89f60d-872c-4594-8555-427b0be13b7b-cnibin\") pod \"multus-additional-cni-plugins-cm82k\" (UID: \"7e89f60d-872c-4594-8555-427b0be13b7b\") " pod="openshift-multus/multus-additional-cni-plugins-cm82k"
Sep 17 15:54:08.569727 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.569727    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4afdccbc-d33e-41ac-98fd-63585ff4d59b-multus-daemon-config\") pod \"multus-xhrb8\" (UID: \"4afdccbc-d33e-41ac-98fd-63585ff4d59b\") " pod="openshift-multus/multus-xhrb8"
Sep 17 15:54:08.570380 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.569738    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4afdccbc-d33e-41ac-98fd-63585ff4d59b-hostroot\") pod \"multus-xhrb8\" (UID: \"4afdccbc-d33e-41ac-98fd-63585ff4d59b\") " pod="openshift-multus/multus-xhrb8"
Sep 17 15:54:08.570380 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.569749    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a1850a4d-4d01-4a71-8636-b5918f1426c2-host-kubelet\") pod \"ovnkube-node-2b47p\" (UID: \"a1850a4d-4d01-4a71-8636-b5918f1426c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.570380 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.569772    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a1850a4d-4d01-4a71-8636-b5918f1426c2-env-overrides\") pod \"ovnkube-node-2b47p\" (UID: \"a1850a4d-4d01-4a71-8636-b5918f1426c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.570380 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.569777    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1850a4d-4d01-4a71-8636-b5918f1426c2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2b47p\" (UID: \"a1850a4d-4d01-4a71-8636-b5918f1426c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.570380 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.569693    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1850a4d-4d01-4a71-8636-b5918f1426c2-var-lib-openvswitch\") pod \"ovnkube-node-2b47p\" (UID: \"a1850a4d-4d01-4a71-8636-b5918f1426c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.570380 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.569810    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1850a4d-4d01-4a71-8636-b5918f1426c2-etc-openvswitch\") pod \"ovnkube-node-2b47p\" (UID: \"a1850a4d-4d01-4a71-8636-b5918f1426c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.570380 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.569823    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxxh9\" (UniqueName: \"kubernetes.io/projected/a1850a4d-4d01-4a71-8636-b5918f1426c2-kube-api-access-dxxh9\") pod \"ovnkube-node-2b47p\" (UID: \"a1850a4d-4d01-4a71-8636-b5918f1426c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.570380 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.569841    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4afdccbc-d33e-41ac-98fd-63585ff4d59b-host-run-k8s-cni-cncf-io\") pod \"multus-xhrb8\" (UID: \"4afdccbc-d33e-41ac-98fd-63585ff4d59b\") " pod="openshift-multus/multus-xhrb8"
Sep 17 15:54:08.570380 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:08.569850    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:54:08.570380 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.569880    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1850a4d-4d01-4a71-8636-b5918f1426c2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2b47p\" (UID: \"a1850a4d-4d01-4a71-8636-b5918f1426c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.570380 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.569882    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/94d1451b-3eae-4481-a504-c29b4315e62e-dbus\") pod \"global-pull-secret-syncer-2f4bn\" (UID: \"94d1451b-3eae-4481-a504-c29b4315e62e\") " pod="kube-system/global-pull-secret-syncer-2f4bn"
Sep 17 15:54:08.570380 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.569853    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e89f60d-872c-4594-8555-427b0be13b7b-system-cni-dir\") pod \"multus-additional-cni-plugins-cm82k\" (UID: \"7e89f60d-872c-4594-8555-427b0be13b7b\") " pod="openshift-multus/multus-additional-cni-plugins-cm82k"
Sep 17 15:54:08.570380 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:08.569928    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7adc5269-e778-473a-a6c7-04754f5e585c-metrics-certs podName:7adc5269-e778-473a-a6c7-04754f5e585c nodeName:}" failed. No retries permitted until 2026-09-17 15:54:09.069909729 +0000 UTC m=+4.840241524 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7adc5269-e778-473a-a6c7-04754f5e585c-metrics-certs") pod "network-metrics-daemon-m6w9z" (UID: "7adc5269-e778-473a-a6c7-04754f5e585c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:54:08.570380 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.570010    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/94d1451b-3eae-4481-a504-c29b4315e62e-dbus\") pod \"global-pull-secret-syncer-2f4bn\" (UID: \"94d1451b-3eae-4481-a504-c29b4315e62e\") " pod="kube-system/global-pull-secret-syncer-2f4bn"
Sep 17 15:54:08.570380 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.570046    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zz86\" (UniqueName: \"kubernetes.io/projected/4afdccbc-d33e-41ac-98fd-63585ff4d59b-kube-api-access-4zz86\") pod \"multus-xhrb8\" (UID: \"4afdccbc-d33e-41ac-98fd-63585ff4d59b\") " pod="openshift-multus/multus-xhrb8"
Sep 17 15:54:08.570380 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.570167    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a1850a4d-4d01-4a71-8636-b5918f1426c2-ovnkube-config\") pod \"ovnkube-node-2b47p\" (UID: \"a1850a4d-4d01-4a71-8636-b5918f1426c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.570380 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.570226    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7e89f60d-872c-4594-8555-427b0be13b7b-cni-binary-copy\") pod \"multus-additional-cni-plugins-cm82k\" (UID: \"7e89f60d-872c-4594-8555-427b0be13b7b\") " pod="openshift-multus/multus-additional-cni-plugins-cm82k"
Sep 17 15:54:08.571132 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.570260    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a1850a4d-4d01-4a71-8636-b5918f1426c2-ovnkube-script-lib\") pod \"ovnkube-node-2b47p\" (UID: \"a1850a4d-4d01-4a71-8636-b5918f1426c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.571132 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.570338    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/66be23a4-8549-4ae3-9d3e-33aca9dca8cb-host-slash\") pod \"iptables-alerter-mvssh\" (UID: \"66be23a4-8549-4ae3-9d3e-33aca9dca8cb\") " pod="openshift-network-operator/iptables-alerter-mvssh"
Sep 17 15:54:08.571132 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.570373    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m449m\" (UniqueName: \"kubernetes.io/projected/7adc5269-e778-473a-a6c7-04754f5e585c-kube-api-access-m449m\") pod \"network-metrics-daemon-m6w9z\" (UID: \"7adc5269-e778-473a-a6c7-04754f5e585c\") " pod="openshift-multus/network-metrics-daemon-m6w9z"
Sep 17 15:54:08.571132 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.570384    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/66be23a4-8549-4ae3-9d3e-33aca9dca8cb-host-slash\") pod \"iptables-alerter-mvssh\" (UID: \"66be23a4-8549-4ae3-9d3e-33aca9dca8cb\") " pod="openshift-network-operator/iptables-alerter-mvssh"
Sep 17 15:54:08.571132 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.570461    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a1850a4d-4d01-4a71-8636-b5918f1426c2-host-run-netns\") pod \"ovnkube-node-2b47p\" (UID: \"a1850a4d-4d01-4a71-8636-b5918f1426c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.571132 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.570524    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a1850a4d-4d01-4a71-8636-b5918f1426c2-host-run-netns\") pod \"ovnkube-node-2b47p\" (UID: \"a1850a4d-4d01-4a71-8636-b5918f1426c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.571132 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.570566    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a1850a4d-4d01-4a71-8636-b5918f1426c2-run-ovn\") pod \"ovnkube-node-2b47p\" (UID: \"a1850a4d-4d01-4a71-8636-b5918f1426c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.571132 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.570593    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7e89f60d-872c-4594-8555-427b0be13b7b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cm82k\" (UID: \"7e89f60d-872c-4594-8555-427b0be13b7b\") " pod="openshift-multus/multus-additional-cni-plugins-cm82k"
Sep 17 15:54:08.571132 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.570622    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4afdccbc-d33e-41ac-98fd-63585ff4d59b-multus-cni-dir\") pod \"multus-xhrb8\" (UID: \"4afdccbc-d33e-41ac-98fd-63585ff4d59b\") " pod="openshift-multus/multus-xhrb8"
Sep 17 15:54:08.571132 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.570623    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a1850a4d-4d01-4a71-8636-b5918f1426c2-run-ovn\") pod \"ovnkube-node-2b47p\" (UID: \"a1850a4d-4d01-4a71-8636-b5918f1426c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.571132 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.570646    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4afdccbc-d33e-41ac-98fd-63585ff4d59b-cni-binary-copy\") pod \"multus-xhrb8\" (UID: \"4afdccbc-d33e-41ac-98fd-63585ff4d59b\") " pod="openshift-multus/multus-xhrb8"
Sep 17 15:54:08.571132 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.570669    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4afdccbc-d33e-41ac-98fd-63585ff4d59b-host-var-lib-cni-bin\") pod \"multus-xhrb8\" (UID: \"4afdccbc-d33e-41ac-98fd-63585ff4d59b\") " pod="openshift-multus/multus-xhrb8"
Sep 17 15:54:08.571132 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.570694    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4afdccbc-d33e-41ac-98fd-63585ff4d59b-host-var-lib-kubelet\") pod \"multus-xhrb8\" (UID: \"4afdccbc-d33e-41ac-98fd-63585ff4d59b\") " pod="openshift-multus/multus-xhrb8"
Sep 17 15:54:08.571132 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.570709    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a1850a4d-4d01-4a71-8636-b5918f1426c2-ovnkube-config\") pod \"ovnkube-node-2b47p\" (UID: \"a1850a4d-4d01-4a71-8636-b5918f1426c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.571132 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.570718    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1850a4d-4d01-4a71-8636-b5918f1426c2-host-run-ovn-kubernetes\") pod \"ovnkube-node-2b47p\" (UID: \"a1850a4d-4d01-4a71-8636-b5918f1426c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.571132 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.570772    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4afdccbc-d33e-41ac-98fd-63585ff4d59b-host-var-lib-kubelet\") pod \"multus-xhrb8\" (UID: \"4afdccbc-d33e-41ac-98fd-63585ff4d59b\") " pod="openshift-multus/multus-xhrb8"
Sep 17 15:54:08.571132 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.570780    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a1850a4d-4d01-4a71-8636-b5918f1426c2-host-run-ovn-kubernetes\") pod \"ovnkube-node-2b47p\" (UID: \"a1850a4d-4d01-4a71-8636-b5918f1426c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.571894 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.570776    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7e89f60d-872c-4594-8555-427b0be13b7b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cm82k\" (UID: \"7e89f60d-872c-4594-8555-427b0be13b7b\") " pod="openshift-multus/multus-additional-cni-plugins-cm82k"
Sep 17 15:54:08.571894 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.570695    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4afdccbc-d33e-41ac-98fd-63585ff4d59b-multus-cni-dir\") pod \"multus-xhrb8\" (UID: \"4afdccbc-d33e-41ac-98fd-63585ff4d59b\") " pod="openshift-multus/multus-xhrb8"
Sep 17 15:54:08.571894 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.570802    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4afdccbc-d33e-41ac-98fd-63585ff4d59b-host-run-netns\") pod \"multus-xhrb8\" (UID: \"4afdccbc-d33e-41ac-98fd-63585ff4d59b\") " pod="openshift-multus/multus-xhrb8"
Sep 17 15:54:08.571894 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.570819    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4afdccbc-d33e-41ac-98fd-63585ff4d59b-host-var-lib-cni-bin\") pod \"multus-xhrb8\" (UID: \"4afdccbc-d33e-41ac-98fd-63585ff4d59b\") " pod="openshift-multus/multus-xhrb8"
Sep 17 15:54:08.571894 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.570833    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4afdccbc-d33e-41ac-98fd-63585ff4d59b-host-run-multus-certs\") pod \"multus-xhrb8\" (UID: \"4afdccbc-d33e-41ac-98fd-63585ff4d59b\") " pod="openshift-multus/multus-xhrb8"
Sep 17 15:54:08.571894 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.570857    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a1850a4d-4d01-4a71-8636-b5918f1426c2-run-systemd\") pod \"ovnkube-node-2b47p\" (UID: \"a1850a4d-4d01-4a71-8636-b5918f1426c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.571894 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.570835    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4afdccbc-d33e-41ac-98fd-63585ff4d59b-host-run-netns\") pod \"multus-xhrb8\" (UID: \"4afdccbc-d33e-41ac-98fd-63585ff4d59b\") " pod="openshift-multus/multus-xhrb8"
Sep 17 15:54:08.571894 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.570868    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4afdccbc-d33e-41ac-98fd-63585ff4d59b-host-run-multus-certs\") pod \"multus-xhrb8\" (UID: \"4afdccbc-d33e-41ac-98fd-63585ff4d59b\") " pod="openshift-multus/multus-xhrb8"
Sep 17 15:54:08.571894 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.570891    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1850a4d-4d01-4a71-8636-b5918f1426c2-run-openvswitch\") pod \"ovnkube-node-2b47p\" (UID: \"a1850a4d-4d01-4a71-8636-b5918f1426c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.571894 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.570903    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a1850a4d-4d01-4a71-8636-b5918f1426c2-run-systemd\") pod \"ovnkube-node-2b47p\" (UID: \"a1850a4d-4d01-4a71-8636-b5918f1426c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.571894 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.570935    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7e89f60d-872c-4594-8555-427b0be13b7b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cm82k\" (UID: \"7e89f60d-872c-4594-8555-427b0be13b7b\") " pod="openshift-multus/multus-additional-cni-plugins-cm82k"
Sep 17 15:54:08.571894 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.570930    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a1850a4d-4d01-4a71-8636-b5918f1426c2-run-openvswitch\") pod \"ovnkube-node-2b47p\" (UID: \"a1850a4d-4d01-4a71-8636-b5918f1426c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.571894 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.570971    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4afdccbc-d33e-41ac-98fd-63585ff4d59b-system-cni-dir\") pod \"multus-xhrb8\" (UID: \"4afdccbc-d33e-41ac-98fd-63585ff4d59b\") " pod="openshift-multus/multus-xhrb8"
Sep 17 15:54:08.571894 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.570994    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4afdccbc-d33e-41ac-98fd-63585ff4d59b-cnibin\") pod \"multus-xhrb8\" (UID: \"4afdccbc-d33e-41ac-98fd-63585ff4d59b\") " pod="openshift-multus/multus-xhrb8"
Sep 17 15:54:08.571894 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.571020    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a1850a4d-4d01-4a71-8636-b5918f1426c2-host-slash\") pod \"ovnkube-node-2b47p\" (UID: \"a1850a4d-4d01-4a71-8636-b5918f1426c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.571894 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.571041    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4afdccbc-d33e-41ac-98fd-63585ff4d59b-system-cni-dir\") pod \"multus-xhrb8\" (UID: \"4afdccbc-d33e-41ac-98fd-63585ff4d59b\") " pod="openshift-multus/multus-xhrb8"
Sep 17 15:54:08.571894 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.571082    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a1850a4d-4d01-4a71-8636-b5918f1426c2-host-slash\") pod \"ovnkube-node-2b47p\" (UID: \"a1850a4d-4d01-4a71-8636-b5918f1426c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.571894 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.571094    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4afdccbc-d33e-41ac-98fd-63585ff4d59b-cnibin\") pod \"multus-xhrb8\" (UID: \"4afdccbc-d33e-41ac-98fd-63585ff4d59b\") " pod="openshift-multus/multus-xhrb8"
Sep 17 15:54:08.572743 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.571106    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a1850a4d-4d01-4a71-8636-b5918f1426c2-log-socket\") pod \"ovnkube-node-2b47p\" (UID: \"a1850a4d-4d01-4a71-8636-b5918f1426c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.572743 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.571134    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4afdccbc-d33e-41ac-98fd-63585ff4d59b-cni-binary-copy\") pod \"multus-xhrb8\" (UID: \"4afdccbc-d33e-41ac-98fd-63585ff4d59b\") " pod="openshift-multus/multus-xhrb8"
Sep 17 15:54:08.572743 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.571132    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7e89f60d-872c-4594-8555-427b0be13b7b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cm82k\" (UID: \"7e89f60d-872c-4594-8555-427b0be13b7b\") " pod="openshift-multus/multus-additional-cni-plugins-cm82k"
Sep 17 15:54:08.572743 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.571183    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4afdccbc-d33e-41ac-98fd-63585ff4d59b-etc-kubernetes\") pod \"multus-xhrb8\" (UID: \"4afdccbc-d33e-41ac-98fd-63585ff4d59b\") " pod="openshift-multus/multus-xhrb8"
Sep 17 15:54:08.572743 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.571207    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/94d1451b-3eae-4481-a504-c29b4315e62e-kubelet-config\") pod \"global-pull-secret-syncer-2f4bn\" (UID: \"94d1451b-3eae-4481-a504-c29b4315e62e\") " pod="kube-system/global-pull-secret-syncer-2f4bn"
Sep 17 15:54:08.572743 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.571217    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a1850a4d-4d01-4a71-8636-b5918f1426c2-log-socket\") pod \"ovnkube-node-2b47p\" (UID: \"a1850a4d-4d01-4a71-8636-b5918f1426c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.572743 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.571233    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a1850a4d-4d01-4a71-8636-b5918f1426c2-host-cni-bin\") pod \"ovnkube-node-2b47p\" (UID: \"a1850a4d-4d01-4a71-8636-b5918f1426c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.572743 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.571259    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/94d1451b-3eae-4481-a504-c29b4315e62e-original-pull-secret\") pod \"global-pull-secret-syncer-2f4bn\" (UID: \"94d1451b-3eae-4481-a504-c29b4315e62e\") " pod="kube-system/global-pull-secret-syncer-2f4bn"
Sep 17 15:54:08.572743 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.571266    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4afdccbc-d33e-41ac-98fd-63585ff4d59b-etc-kubernetes\") pod \"multus-xhrb8\" (UID: \"4afdccbc-d33e-41ac-98fd-63585ff4d59b\") " pod="openshift-multus/multus-xhrb8"
Sep 17 15:54:08.572743 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.571283    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/66be23a4-8549-4ae3-9d3e-33aca9dca8cb-iptables-alerter-script\") pod \"iptables-alerter-mvssh\" (UID: \"66be23a4-8549-4ae3-9d3e-33aca9dca8cb\") " pod="openshift-network-operator/iptables-alerter-mvssh"
Sep 17 15:54:08.572743 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.571303    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a1850a4d-4d01-4a71-8636-b5918f1426c2-host-cni-bin\") pod \"ovnkube-node-2b47p\" (UID: \"a1850a4d-4d01-4a71-8636-b5918f1426c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.572743 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.571323    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4afdccbc-d33e-41ac-98fd-63585ff4d59b-host-var-lib-cni-multus\") pod \"multus-xhrb8\" (UID: \"4afdccbc-d33e-41ac-98fd-63585ff4d59b\") " pod="openshift-multus/multus-xhrb8"
Sep 17 15:54:08.572743 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.571346    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4afdccbc-d33e-41ac-98fd-63585ff4d59b-multus-conf-dir\") pod \"multus-xhrb8\" (UID: \"4afdccbc-d33e-41ac-98fd-63585ff4d59b\") " pod="openshift-multus/multus-xhrb8"
Sep 17 15:54:08.572743 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.571351    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/94d1451b-3eae-4481-a504-c29b4315e62e-kubelet-config\") pod \"global-pull-secret-syncer-2f4bn\" (UID: \"94d1451b-3eae-4481-a504-c29b4315e62e\") " pod="kube-system/global-pull-secret-syncer-2f4bn"
Sep 17 15:54:08.572743 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.571368    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a1850a4d-4d01-4a71-8636-b5918f1426c2-systemd-units\") pod \"ovnkube-node-2b47p\" (UID: \"a1850a4d-4d01-4a71-8636-b5918f1426c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.572743 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.571410    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7e89f60d-872c-4594-8555-427b0be13b7b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cm82k\" (UID: \"7e89f60d-872c-4594-8555-427b0be13b7b\") " pod="openshift-multus/multus-additional-cni-plugins-cm82k"
Sep 17 15:54:08.572743 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.571449    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a1850a4d-4d01-4a71-8636-b5918f1426c2-systemd-units\") pod \"ovnkube-node-2b47p\" (UID: \"a1850a4d-4d01-4a71-8636-b5918f1426c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.573509 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.571487    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4afdccbc-d33e-41ac-98fd-63585ff4d59b-host-var-lib-cni-multus\") pod \"multus-xhrb8\" (UID: \"4afdccbc-d33e-41ac-98fd-63585ff4d59b\") " pod="openshift-multus/multus-xhrb8"
Sep 17 15:54:08.573509 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:08.571520    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:54:08.573509 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.571520    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4afdccbc-d33e-41ac-98fd-63585ff4d59b-multus-conf-dir\") pod \"multus-xhrb8\" (UID: \"4afdccbc-d33e-41ac-98fd-63585ff4d59b\") " pod="openshift-multus/multus-xhrb8"
Sep 17 15:54:08.573509 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:08.571571    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94d1451b-3eae-4481-a504-c29b4315e62e-original-pull-secret podName:94d1451b-3eae-4481-a504-c29b4315e62e nodeName:}" failed. No retries permitted until 2026-09-17 15:54:09.071555452 +0000 UTC m=+4.841887232 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/94d1451b-3eae-4481-a504-c29b4315e62e-original-pull-secret") pod "global-pull-secret-syncer-2f4bn" (UID: "94d1451b-3eae-4481-a504-c29b4315e62e") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:54:08.573509 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.571576    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7e89f60d-872c-4594-8555-427b0be13b7b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cm82k\" (UID: \"7e89f60d-872c-4594-8555-427b0be13b7b\") " pod="openshift-multus/multus-additional-cni-plugins-cm82k"
Sep 17 15:54:08.573509 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.571883    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/66be23a4-8549-4ae3-9d3e-33aca9dca8cb-iptables-alerter-script\") pod \"iptables-alerter-mvssh\" (UID: \"66be23a4-8549-4ae3-9d3e-33aca9dca8cb\") " pod="openshift-network-operator/iptables-alerter-mvssh"
Sep 17 15:54:08.573509 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.573260    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a1850a4d-4d01-4a71-8636-b5918f1426c2-ovn-node-metrics-cert\") pod \"ovnkube-node-2b47p\" (UID: \"a1850a4d-4d01-4a71-8636-b5918f1426c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.585058 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.585021    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbw4w\" (UniqueName: \"kubernetes.io/projected/66be23a4-8549-4ae3-9d3e-33aca9dca8cb-kube-api-access-pbw4w\") pod \"iptables-alerter-mvssh\" (UID: \"66be23a4-8549-4ae3-9d3e-33aca9dca8cb\") " pod="openshift-network-operator/iptables-alerter-mvssh"
Sep 17 15:54:08.585203 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.585180    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m449m\" (UniqueName: \"kubernetes.io/projected/7adc5269-e778-473a-a6c7-04754f5e585c-kube-api-access-m449m\") pod \"network-metrics-daemon-m6w9z\" (UID: \"7adc5269-e778-473a-a6c7-04754f5e585c\") " pod="openshift-multus/network-metrics-daemon-m6w9z"
Sep 17 15:54:08.586052 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.586023    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc6t2\" (UniqueName: \"kubernetes.io/projected/7e89f60d-872c-4594-8555-427b0be13b7b-kube-api-access-qc6t2\") pod \"multus-additional-cni-plugins-cm82k\" (UID: \"7e89f60d-872c-4594-8555-427b0be13b7b\") " pod="openshift-multus/multus-additional-cni-plugins-cm82k"
Sep 17 15:54:08.591716 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.591690    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxxh9\" (UniqueName: \"kubernetes.io/projected/a1850a4d-4d01-4a71-8636-b5918f1426c2-kube-api-access-dxxh9\") pod \"ovnkube-node-2b47p\" (UID: \"a1850a4d-4d01-4a71-8636-b5918f1426c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.591935 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.591916    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zz86\" (UniqueName: \"kubernetes.io/projected/4afdccbc-d33e-41ac-98fd-63585ff4d59b-kube-api-access-4zz86\") pod \"multus-xhrb8\" (UID: \"4afdccbc-d33e-41ac-98fd-63585ff4d59b\") " pod="openshift-multus/multus-xhrb8"
Sep 17 15:54:08.654135 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.654105    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dtbtf"
Sep 17 15:54:08.664093 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.664070    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65v88"
Sep 17 15:54:08.676723 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.676700    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ntl9t"
Sep 17 15:54:08.682474 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.682454    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cdgrj"
Sep 17 15:54:08.690352 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.690336    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xhtm4"
Sep 17 15:54:08.703920 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.703885    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cm82k"
Sep 17 15:54:08.712824 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.712797    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xhrb8"
Sep 17 15:54:08.719659 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.719631    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:08.726482 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:08.726463    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mvssh"
Sep 17 15:54:09.074516 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:09.074471    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs87t\" (UniqueName: \"kubernetes.io/projected/3114c3ad-7b7e-41ad-b175-ead7d92d5ee3-kube-api-access-rs87t\") pod \"network-check-target-9mgwn\" (UID: \"3114c3ad-7b7e-41ad-b175-ead7d92d5ee3\") " pod="openshift-network-diagnostics/network-check-target-9mgwn"
Sep 17 15:54:09.074708 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:09.074543    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/94d1451b-3eae-4481-a504-c29b4315e62e-original-pull-secret\") pod \"global-pull-secret-syncer-2f4bn\" (UID: \"94d1451b-3eae-4481-a504-c29b4315e62e\") " pod="kube-system/global-pull-secret-syncer-2f4bn"
Sep 17 15:54:09.074708 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:09.074583    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7adc5269-e778-473a-a6c7-04754f5e585c-metrics-certs\") pod \"network-metrics-daemon-m6w9z\" (UID: \"7adc5269-e778-473a-a6c7-04754f5e585c\") " pod="openshift-multus/network-metrics-daemon-m6w9z"
Sep 17 15:54:09.074708 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:09.074658    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:54:09.074708 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:09.074677    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:54:09.074708 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:09.074684    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:54:09.074708 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:09.074694    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:54:09.074964 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:09.074697    2585 projected.go:196] Error preparing data for projected volume kube-api-access-rs87t for pod openshift-network-diagnostics/network-check-target-9mgwn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:54:09.074964 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:09.074730    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94d1451b-3eae-4481-a504-c29b4315e62e-original-pull-secret podName:94d1451b-3eae-4481-a504-c29b4315e62e nodeName:}" failed. No retries permitted until 2026-09-17 15:54:10.074716773 +0000 UTC m=+5.845048565 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/94d1451b-3eae-4481-a504-c29b4315e62e-original-pull-secret") pod "global-pull-secret-syncer-2f4bn" (UID: "94d1451b-3eae-4481-a504-c29b4315e62e") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:54:09.074964 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:09.074781    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7adc5269-e778-473a-a6c7-04754f5e585c-metrics-certs podName:7adc5269-e778-473a-a6c7-04754f5e585c nodeName:}" failed. No retries permitted until 2026-09-17 15:54:10.074763675 +0000 UTC m=+5.845095455 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7adc5269-e778-473a-a6c7-04754f5e585c-metrics-certs") pod "network-metrics-daemon-m6w9z" (UID: "7adc5269-e778-473a-a6c7-04754f5e585c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:54:09.074964 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:09.074797    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3114c3ad-7b7e-41ad-b175-ead7d92d5ee3-kube-api-access-rs87t podName:3114c3ad-7b7e-41ad-b175-ead7d92d5ee3 nodeName:}" failed. No retries permitted until 2026-09-17 15:54:10.074787429 +0000 UTC m=+5.845119213 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rs87t" (UniqueName: "kubernetes.io/projected/3114c3ad-7b7e-41ad-b175-ead7d92d5ee3-kube-api-access-rs87t") pod "network-check-target-9mgwn" (UID: "3114c3ad-7b7e-41ad-b175-ead7d92d5ee3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:54:09.428592 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:09.428501    2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 15:49:08 +0000 UTC" deadline="2028-04-03 17:48:43.814636219 +0000 UTC"
Sep 17 15:54:09.428592 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:09.428539    2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13537h54m34.386104211s"
Sep 17 15:54:09.507287 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:09.507261    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mgwn"
Sep 17 15:54:09.507481 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:09.507360    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mgwn" podUID="3114c3ad-7b7e-41ad-b175-ead7d92d5ee3"
Sep 17 15:54:09.533033 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:09.533007    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1850a4d_4d01_4a71_8636_b5918f1426c2.slice/crio-54445529f0d2ab818cf695e69c8343c2012f492487414ad918b1ec1e02b68c20 WatchSource:0}: Error finding container 54445529f0d2ab818cf695e69c8343c2012f492487414ad918b1ec1e02b68c20: Status 404 returned error can't find the container with id 54445529f0d2ab818cf695e69c8343c2012f492487414ad918b1ec1e02b68c20
Sep 17 15:54:09.534542 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:09.534480    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8910d933_4fe9_45a0_be48_f225cded154a.slice/crio-7283cc6bccffa6c21c267e85e0849b68926fd22b02427527412b7b2b325f6cf0 WatchSource:0}: Error finding container 7283cc6bccffa6c21c267e85e0849b68926fd22b02427527412b7b2b325f6cf0: Status 404 returned error can't find the container with id 7283cc6bccffa6c21c267e85e0849b68926fd22b02427527412b7b2b325f6cf0
Sep 17 15:54:09.543152 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:09.543131    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66be23a4_8549_4ae3_9d3e_33aca9dca8cb.slice/crio-6a42a62c8fba0b5894a9f0c51c7d2b2282415bacf2d8f3b706f6639aa8aa2111 WatchSource:0}: Error finding container 6a42a62c8fba0b5894a9f0c51c7d2b2282415bacf2d8f3b706f6639aa8aa2111: Status 404 returned error can't find the container with id 6a42a62c8fba0b5894a9f0c51c7d2b2282415bacf2d8f3b706f6639aa8aa2111
Sep 17 15:54:09.544218 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:09.544178    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ecfa8d8_297d_4e71_b29c_726646be98bf.slice/crio-cc745180001cc424fbf18f798765b24d5136e07b326133566870bb82b72dd95a WatchSource:0}: Error finding container cc745180001cc424fbf18f798765b24d5136e07b326133566870bb82b72dd95a: Status 404 returned error can't find the container with id cc745180001cc424fbf18f798765b24d5136e07b326133566870bb82b72dd95a
Sep 17 15:54:09.564532 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:09.564498    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4438a49_8ea1_40aa_bdd9_2707192617af.slice/crio-7de904bfc4ac0d8238d383cab7a8b20e4be26354bc797cdb70671055a61a22b8 WatchSource:0}: Error finding container 7de904bfc4ac0d8238d383cab7a8b20e4be26354bc797cdb70671055a61a22b8: Status 404 returned error can't find the container with id 7de904bfc4ac0d8238d383cab7a8b20e4be26354bc797cdb70671055a61a22b8
Sep 17 15:54:09.565286 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:09.565261    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e89f60d_872c_4594_8555_427b0be13b7b.slice/crio-f0541e6479a047a691d548cb8a0dc77732992323e326c0a4118e346daf353a29 WatchSource:0}: Error finding container f0541e6479a047a691d548cb8a0dc77732992323e326c0a4118e346daf353a29: Status 404 returned error can't find the container with id f0541e6479a047a691d548cb8a0dc77732992323e326c0a4118e346daf353a29
Sep 17 15:54:09.566239 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:09.566218    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4afdccbc_d33e_41ac_98fd_63585ff4d59b.slice/crio-e3c53ae9366992379f4f8d6af8a0af704c267c7f332a626974bbc2e7b1e2c308 WatchSource:0}: Error finding container e3c53ae9366992379f4f8d6af8a0af704c267c7f332a626974bbc2e7b1e2c308: Status 404 returned error can't find the container with id e3c53ae9366992379f4f8d6af8a0af704c267c7f332a626974bbc2e7b1e2c308
Sep 17 15:54:09.567072 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:09.567044    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28231271_38ce_4985_9020_e3494fd366c7.slice/crio-5362c62623e46e362b18e57e588c1b83d9aeca4e818e5634748daed4e5f4d80d WatchSource:0}: Error finding container 5362c62623e46e362b18e57e588c1b83d9aeca4e818e5634748daed4e5f4d80d: Status 404 returned error can't find the container with id 5362c62623e46e362b18e57e588c1b83d9aeca4e818e5634748daed4e5f4d80d
Sep 17 15:54:09.568439 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:09.568398    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8249ef17_a5c4_414e_a2ca_a22cc46ef3eb.slice/crio-4480560ed63788f25573fe8836f91591c2ea26c421b6554989d9089c410bf809 WatchSource:0}: Error finding container 4480560ed63788f25573fe8836f91591c2ea26c421b6554989d9089c410bf809: Status 404 returned error can't find the container with id 4480560ed63788f25573fe8836f91591c2ea26c421b6554989d9089c410bf809
Sep 17 15:54:10.080533 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:10.080446    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7adc5269-e778-473a-a6c7-04754f5e585c-metrics-certs\") pod \"network-metrics-daemon-m6w9z\" (UID: \"7adc5269-e778-473a-a6c7-04754f5e585c\") " pod="openshift-multus/network-metrics-daemon-m6w9z"
Sep 17 15:54:10.080533 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:10.080501    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs87t\" (UniqueName: \"kubernetes.io/projected/3114c3ad-7b7e-41ad-b175-ead7d92d5ee3-kube-api-access-rs87t\") pod \"network-check-target-9mgwn\" (UID: \"3114c3ad-7b7e-41ad-b175-ead7d92d5ee3\") " pod="openshift-network-diagnostics/network-check-target-9mgwn"
Sep 17 15:54:10.080804 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:10.080538    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/94d1451b-3eae-4481-a504-c29b4315e62e-original-pull-secret\") pod \"global-pull-secret-syncer-2f4bn\" (UID: \"94d1451b-3eae-4481-a504-c29b4315e62e\") " pod="kube-system/global-pull-secret-syncer-2f4bn"
Sep 17 15:54:10.080804 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:10.080653    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:54:10.080804 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:10.080706    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94d1451b-3eae-4481-a504-c29b4315e62e-original-pull-secret podName:94d1451b-3eae-4481-a504-c29b4315e62e nodeName:}" failed. No retries permitted until 2026-09-17 15:54:12.080689852 +0000 UTC m=+7.851021628 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/94d1451b-3eae-4481-a504-c29b4315e62e-original-pull-secret") pod "global-pull-secret-syncer-2f4bn" (UID: "94d1451b-3eae-4481-a504-c29b4315e62e") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:54:10.081107 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:10.081086    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:54:10.081197 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:10.081143    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7adc5269-e778-473a-a6c7-04754f5e585c-metrics-certs podName:7adc5269-e778-473a-a6c7-04754f5e585c nodeName:}" failed. No retries permitted until 2026-09-17 15:54:12.081127526 +0000 UTC m=+7.851459326 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7adc5269-e778-473a-a6c7-04754f5e585c-metrics-certs") pod "network-metrics-daemon-m6w9z" (UID: "7adc5269-e778-473a-a6c7-04754f5e585c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:54:10.081371 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:10.081221    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:54:10.081371 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:10.081238    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:54:10.081371 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:10.081250    2585 projected.go:196] Error preparing data for projected volume kube-api-access-rs87t for pod openshift-network-diagnostics/network-check-target-9mgwn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:54:10.081371 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:10.081285    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3114c3ad-7b7e-41ad-b175-ead7d92d5ee3-kube-api-access-rs87t podName:3114c3ad-7b7e-41ad-b175-ead7d92d5ee3 nodeName:}" failed. No retries permitted until 2026-09-17 15:54:12.081273512 +0000 UTC m=+7.851605306 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rs87t" (UniqueName: "kubernetes.io/projected/3114c3ad-7b7e-41ad-b175-ead7d92d5ee3-kube-api-access-rs87t") pod "network-check-target-9mgwn" (UID: "3114c3ad-7b7e-41ad-b175-ead7d92d5ee3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:54:10.429154 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:10.429030    2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 15:49:08 +0000 UTC" deadline="2028-05-10 06:16:58.39540697 +0000 UTC"
Sep 17 15:54:10.429154 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:10.429097    2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14414h22m47.966315223s"
Sep 17 15:54:10.509061 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:10.509031    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f4bn"
Sep 17 15:54:10.509243 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:10.509164    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f4bn" podUID="94d1451b-3eae-4481-a504-c29b4315e62e"
Sep 17 15:54:10.509305 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:10.509251    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6w9z"
Sep 17 15:54:10.509356 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:10.509340    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m6w9z" podUID="7adc5269-e778-473a-a6c7-04754f5e585c"
Sep 17 15:54:10.522272 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:10.522176    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ntl9t" event={"ID":"8249ef17-a5c4-414e-a2ca-a22cc46ef3eb","Type":"ContainerStarted","Data":"4480560ed63788f25573fe8836f91591c2ea26c421b6554989d9089c410bf809"}
Sep 17 15:54:10.527014 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:10.526938    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xhtm4" event={"ID":"c4438a49-8ea1-40aa-bdd9-2707192617af","Type":"ContainerStarted","Data":"7de904bfc4ac0d8238d383cab7a8b20e4be26354bc797cdb70671055a61a22b8"}
Sep 17 15:54:10.553283 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:10.553247    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xhrb8" event={"ID":"4afdccbc-d33e-41ac-98fd-63585ff4d59b","Type":"ContainerStarted","Data":"e3c53ae9366992379f4f8d6af8a0af704c267c7f332a626974bbc2e7b1e2c308"}
Sep 17 15:54:10.558386 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:10.558345    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65v88" event={"ID":"8910d933-4fe9-45a0-be48-f225cded154a","Type":"ContainerStarted","Data":"7283cc6bccffa6c21c267e85e0849b68926fd22b02427527412b7b2b325f6cf0"}
Sep 17 15:54:10.572904 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:10.572868    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2b47p" event={"ID":"a1850a4d-4d01-4a71-8636-b5918f1426c2","Type":"ContainerStarted","Data":"54445529f0d2ab818cf695e69c8343c2012f492487414ad918b1ec1e02b68c20"}
Sep 17 15:54:10.582843 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:10.582813    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-43.ec2.internal" event={"ID":"5692cd509d685463b01f1d043c0bcb98","Type":"ContainerStarted","Data":"2da47c2955c7abb765cb1131995eaf7bad1501178124a4ea2fd599e4282617ab"}
Sep 17 15:54:10.600458 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:10.600410    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dtbtf" event={"ID":"28231271-38ce-4985-9020-e3494fd366c7","Type":"ContainerStarted","Data":"5362c62623e46e362b18e57e588c1b83d9aeca4e818e5634748daed4e5f4d80d"}
Sep 17 15:54:10.607562 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:10.607509    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-43.ec2.internal" podStartSLOduration=3.607490374 podStartE2EDuration="3.607490374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:54:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:54:10.606904621 +0000 UTC m=+6.377236422" watchObservedRunningTime="2026-09-17 15:54:10.607490374 +0000 UTC m=+6.377822170"
Sep 17 15:54:10.610836 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:10.610742    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cm82k" event={"ID":"7e89f60d-872c-4594-8555-427b0be13b7b","Type":"ContainerStarted","Data":"f0541e6479a047a691d548cb8a0dc77732992323e326c0a4118e346daf353a29"}
Sep 17 15:54:10.621366 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:10.621336    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cdgrj" event={"ID":"2ecfa8d8-297d-4e71-b29c-726646be98bf","Type":"ContainerStarted","Data":"cc745180001cc424fbf18f798765b24d5136e07b326133566870bb82b72dd95a"}
Sep 17 15:54:10.639863 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:10.639813    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mvssh" event={"ID":"66be23a4-8549-4ae3-9d3e-33aca9dca8cb","Type":"ContainerStarted","Data":"6a42a62c8fba0b5894a9f0c51c7d2b2282415bacf2d8f3b706f6639aa8aa2111"}
Sep 17 15:54:10.652887 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:10.652855    2585 generic.go:356] "Generic (PLEG): container finished" podID="7156e3815525514fe59f7c5337670664" containerID="d6c04ef44e7d430a4a09eb79624c21525046d01679f23cb6edf186ece2234960" exitCode=0
Sep 17 15:54:10.653026 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:10.652899    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-43.ec2.internal" event={"ID":"7156e3815525514fe59f7c5337670664","Type":"ContainerDied","Data":"d6c04ef44e7d430a4a09eb79624c21525046d01679f23cb6edf186ece2234960"}
Sep 17 15:54:11.507536 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:11.507502    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mgwn"
Sep 17 15:54:11.508022 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:11.507634    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mgwn" podUID="3114c3ad-7b7e-41ad-b175-ead7d92d5ee3"
Sep 17 15:54:11.685901 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:11.685482    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-43.ec2.internal" event={"ID":"7156e3815525514fe59f7c5337670664","Type":"ContainerStarted","Data":"4e7d81a11473e7b203f45636020d50b094744be2c31ce7aa2e9cdd9b3eac60a8"}
Sep 17 15:54:11.704074 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:11.703114    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-43.ec2.internal" podStartSLOduration=4.703093318 podStartE2EDuration="4.703093318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:54:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:54:11.703032525 +0000 UTC m=+7.473364324" watchObservedRunningTime="2026-09-17 15:54:11.703093318 +0000 UTC m=+7.473425119"
Sep 17 15:54:12.102246 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:12.101295    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/94d1451b-3eae-4481-a504-c29b4315e62e-original-pull-secret\") pod \"global-pull-secret-syncer-2f4bn\" (UID: \"94d1451b-3eae-4481-a504-c29b4315e62e\") " pod="kube-system/global-pull-secret-syncer-2f4bn"
Sep 17 15:54:12.102246 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:12.101367    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7adc5269-e778-473a-a6c7-04754f5e585c-metrics-certs\") pod \"network-metrics-daemon-m6w9z\" (UID: \"7adc5269-e778-473a-a6c7-04754f5e585c\") " pod="openshift-multus/network-metrics-daemon-m6w9z"
Sep 17 15:54:12.102246 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:12.101412    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs87t\" (UniqueName: \"kubernetes.io/projected/3114c3ad-7b7e-41ad-b175-ead7d92d5ee3-kube-api-access-rs87t\") pod \"network-check-target-9mgwn\" (UID: \"3114c3ad-7b7e-41ad-b175-ead7d92d5ee3\") " pod="openshift-network-diagnostics/network-check-target-9mgwn"
Sep 17 15:54:12.102246 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:12.101566    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:54:12.102246 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:12.101589    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:54:12.102246 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:12.101602    2585 projected.go:196] Error preparing data for projected volume kube-api-access-rs87t for pod openshift-network-diagnostics/network-check-target-9mgwn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:54:12.102246 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:12.101661    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3114c3ad-7b7e-41ad-b175-ead7d92d5ee3-kube-api-access-rs87t podName:3114c3ad-7b7e-41ad-b175-ead7d92d5ee3 nodeName:}" failed. No retries permitted until 2026-09-17 15:54:16.101643894 +0000 UTC m=+11.871975673 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-rs87t" (UniqueName: "kubernetes.io/projected/3114c3ad-7b7e-41ad-b175-ead7d92d5ee3-kube-api-access-rs87t") pod "network-check-target-9mgwn" (UID: "3114c3ad-7b7e-41ad-b175-ead7d92d5ee3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:54:12.102246 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:12.102078    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:54:12.102246 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:12.102125    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94d1451b-3eae-4481-a504-c29b4315e62e-original-pull-secret podName:94d1451b-3eae-4481-a504-c29b4315e62e nodeName:}" failed. No retries permitted until 2026-09-17 15:54:16.102111492 +0000 UTC m=+11.872443293 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/94d1451b-3eae-4481-a504-c29b4315e62e-original-pull-secret") pod "global-pull-secret-syncer-2f4bn" (UID: "94d1451b-3eae-4481-a504-c29b4315e62e") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:54:12.102246 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:12.102181    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:54:12.102246 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:12.102211    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7adc5269-e778-473a-a6c7-04754f5e585c-metrics-certs podName:7adc5269-e778-473a-a6c7-04754f5e585c nodeName:}" failed. No retries permitted until 2026-09-17 15:54:16.102200268 +0000 UTC m=+11.872532047 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7adc5269-e778-473a-a6c7-04754f5e585c-metrics-certs") pod "network-metrics-daemon-m6w9z" (UID: "7adc5269-e778-473a-a6c7-04754f5e585c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:54:12.507207 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:12.507122    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f4bn"
Sep 17 15:54:12.507359 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:12.507254    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f4bn" podUID="94d1451b-3eae-4481-a504-c29b4315e62e"
Sep 17 15:54:12.507755 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:12.507732    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6w9z"
Sep 17 15:54:12.508126 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:12.507850    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m6w9z" podUID="7adc5269-e778-473a-a6c7-04754f5e585c"
Sep 17 15:54:13.507005 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:13.506970    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mgwn"
Sep 17 15:54:13.507174 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:13.507095    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mgwn" podUID="3114c3ad-7b7e-41ad-b175-ead7d92d5ee3"
Sep 17 15:54:14.506983 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:14.506947    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f4bn"
Sep 17 15:54:14.507461 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:14.506988    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6w9z"
Sep 17 15:54:14.507461 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:14.507089    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f4bn" podUID="94d1451b-3eae-4481-a504-c29b4315e62e"
Sep 17 15:54:14.507461 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:14.507234    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m6w9z" podUID="7adc5269-e778-473a-a6c7-04754f5e585c"
Sep 17 15:54:15.507280 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:15.507237    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mgwn"
Sep 17 15:54:15.507734 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:15.507368    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mgwn" podUID="3114c3ad-7b7e-41ad-b175-ead7d92d5ee3"
Sep 17 15:54:16.135768 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:16.135729    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7adc5269-e778-473a-a6c7-04754f5e585c-metrics-certs\") pod \"network-metrics-daemon-m6w9z\" (UID: \"7adc5269-e778-473a-a6c7-04754f5e585c\") " pod="openshift-multus/network-metrics-daemon-m6w9z"
Sep 17 15:54:16.135966 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:16.135822    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs87t\" (UniqueName: \"kubernetes.io/projected/3114c3ad-7b7e-41ad-b175-ead7d92d5ee3-kube-api-access-rs87t\") pod \"network-check-target-9mgwn\" (UID: \"3114c3ad-7b7e-41ad-b175-ead7d92d5ee3\") " pod="openshift-network-diagnostics/network-check-target-9mgwn"
Sep 17 15:54:16.135966 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:16.135875    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/94d1451b-3eae-4481-a504-c29b4315e62e-original-pull-secret\") pod \"global-pull-secret-syncer-2f4bn\" (UID: \"94d1451b-3eae-4481-a504-c29b4315e62e\") " pod="kube-system/global-pull-secret-syncer-2f4bn"
Sep 17 15:54:16.135966 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:16.135887    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:54:16.135966 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:16.135959    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:54:16.136172 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:16.135979    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:54:16.136172 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:16.135991    2585 projected.go:196] Error preparing data for projected volume kube-api-access-rs87t for pod openshift-network-diagnostics/network-check-target-9mgwn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:54:16.136172 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:16.135991    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:54:16.136172 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:16.135961    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7adc5269-e778-473a-a6c7-04754f5e585c-metrics-certs podName:7adc5269-e778-473a-a6c7-04754f5e585c nodeName:}" failed. No retries permitted until 2026-09-17 15:54:24.135942963 +0000 UTC m=+19.906274758 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7adc5269-e778-473a-a6c7-04754f5e585c-metrics-certs") pod "network-metrics-daemon-m6w9z" (UID: "7adc5269-e778-473a-a6c7-04754f5e585c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:54:16.136172 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:16.136046    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3114c3ad-7b7e-41ad-b175-ead7d92d5ee3-kube-api-access-rs87t podName:3114c3ad-7b7e-41ad-b175-ead7d92d5ee3 nodeName:}" failed. No retries permitted until 2026-09-17 15:54:24.13602818 +0000 UTC m=+19.906359960 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-rs87t" (UniqueName: "kubernetes.io/projected/3114c3ad-7b7e-41ad-b175-ead7d92d5ee3-kube-api-access-rs87t") pod "network-check-target-9mgwn" (UID: "3114c3ad-7b7e-41ad-b175-ead7d92d5ee3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:54:16.136172 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:16.136060    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94d1451b-3eae-4481-a504-c29b4315e62e-original-pull-secret podName:94d1451b-3eae-4481-a504-c29b4315e62e nodeName:}" failed. No retries permitted until 2026-09-17 15:54:24.136053133 +0000 UTC m=+19.906384913 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/94d1451b-3eae-4481-a504-c29b4315e62e-original-pull-secret") pod "global-pull-secret-syncer-2f4bn" (UID: "94d1451b-3eae-4481-a504-c29b4315e62e") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:54:16.508173 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:16.508073    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6w9z"
Sep 17 15:54:16.508666 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:16.508198    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f4bn"
Sep 17 15:54:16.508666 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:16.508277    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f4bn" podUID="94d1451b-3eae-4481-a504-c29b4315e62e"
Sep 17 15:54:16.508666 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:16.508192    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m6w9z" podUID="7adc5269-e778-473a-a6c7-04754f5e585c"
Sep 17 15:54:17.507188 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:17.507159    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mgwn"
Sep 17 15:54:17.507308 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:17.507285    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mgwn" podUID="3114c3ad-7b7e-41ad-b175-ead7d92d5ee3"
Sep 17 15:54:18.507401 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:18.507369    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6w9z"
Sep 17 15:54:18.507852 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:18.507515    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m6w9z" podUID="7adc5269-e778-473a-a6c7-04754f5e585c"
Sep 17 15:54:18.507905 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:18.507880    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f4bn"
Sep 17 15:54:18.507979 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:18.507960    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f4bn" podUID="94d1451b-3eae-4481-a504-c29b4315e62e"
Sep 17 15:54:19.507235 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:19.507200    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mgwn"
Sep 17 15:54:19.507546 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:19.507319    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mgwn" podUID="3114c3ad-7b7e-41ad-b175-ead7d92d5ee3"
Sep 17 15:54:20.507254 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:20.507218    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f4bn"
Sep 17 15:54:20.507444 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:20.507218    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6w9z"
Sep 17 15:54:20.507444 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:20.507353    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f4bn" podUID="94d1451b-3eae-4481-a504-c29b4315e62e"
Sep 17 15:54:20.507561 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:20.507479    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m6w9z" podUID="7adc5269-e778-473a-a6c7-04754f5e585c"
Sep 17 15:54:21.506628 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:21.506587    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mgwn"
Sep 17 15:54:21.506851 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:21.506727    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mgwn" podUID="3114c3ad-7b7e-41ad-b175-ead7d92d5ee3"
Sep 17 15:54:22.507303 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:22.507264    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6w9z"
Sep 17 15:54:22.507739 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:22.507282    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f4bn"
Sep 17 15:54:22.507739 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:22.507414    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m6w9z" podUID="7adc5269-e778-473a-a6c7-04754f5e585c"
Sep 17 15:54:22.507739 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:22.507500    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f4bn" podUID="94d1451b-3eae-4481-a504-c29b4315e62e"
Sep 17 15:54:23.507558 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:23.507469    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mgwn"
Sep 17 15:54:23.508040 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:23.507602    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mgwn" podUID="3114c3ad-7b7e-41ad-b175-ead7d92d5ee3"
Sep 17 15:54:24.195583 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:24.195546    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs87t\" (UniqueName: \"kubernetes.io/projected/3114c3ad-7b7e-41ad-b175-ead7d92d5ee3-kube-api-access-rs87t\") pod \"network-check-target-9mgwn\" (UID: \"3114c3ad-7b7e-41ad-b175-ead7d92d5ee3\") " pod="openshift-network-diagnostics/network-check-target-9mgwn"
Sep 17 15:54:24.195842 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:24.195603    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/94d1451b-3eae-4481-a504-c29b4315e62e-original-pull-secret\") pod \"global-pull-secret-syncer-2f4bn\" (UID: \"94d1451b-3eae-4481-a504-c29b4315e62e\") " pod="kube-system/global-pull-secret-syncer-2f4bn"
Sep 17 15:54:24.195842 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:24.195644    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7adc5269-e778-473a-a6c7-04754f5e585c-metrics-certs\") pod \"network-metrics-daemon-m6w9z\" (UID: \"7adc5269-e778-473a-a6c7-04754f5e585c\") " pod="openshift-multus/network-metrics-daemon-m6w9z"
Sep 17 15:54:24.195842 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:24.195775    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:54:24.195842 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:24.195793    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:54:24.195842 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:24.195821    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:54:24.195842 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:24.195835    2585 projected.go:196] Error preparing data for projected volume kube-api-access-rs87t for pod openshift-network-diagnostics/network-check-target-9mgwn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:54:24.195842 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:24.195836    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:54:24.196161 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:24.195847    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94d1451b-3eae-4481-a504-c29b4315e62e-original-pull-secret podName:94d1451b-3eae-4481-a504-c29b4315e62e nodeName:}" failed. No retries permitted until 2026-09-17 15:54:40.195827769 +0000 UTC m=+35.966159565 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/94d1451b-3eae-4481-a504-c29b4315e62e-original-pull-secret") pod "global-pull-secret-syncer-2f4bn" (UID: "94d1451b-3eae-4481-a504-c29b4315e62e") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:54:24.196161 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:24.195897    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3114c3ad-7b7e-41ad-b175-ead7d92d5ee3-kube-api-access-rs87t podName:3114c3ad-7b7e-41ad-b175-ead7d92d5ee3 nodeName:}" failed. No retries permitted until 2026-09-17 15:54:40.195880069 +0000 UTC m=+35.966211845 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-rs87t" (UniqueName: "kubernetes.io/projected/3114c3ad-7b7e-41ad-b175-ead7d92d5ee3-kube-api-access-rs87t") pod "network-check-target-9mgwn" (UID: "3114c3ad-7b7e-41ad-b175-ead7d92d5ee3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:54:24.196161 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:24.195915    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7adc5269-e778-473a-a6c7-04754f5e585c-metrics-certs podName:7adc5269-e778-473a-a6c7-04754f5e585c nodeName:}" failed. No retries permitted until 2026-09-17 15:54:40.195904563 +0000 UTC m=+35.966236345 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7adc5269-e778-473a-a6c7-04754f5e585c-metrics-certs") pod "network-metrics-daemon-m6w9z" (UID: "7adc5269-e778-473a-a6c7-04754f5e585c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:54:24.507038 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:24.506999    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f4bn"
Sep 17 15:54:24.507232 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:24.506999    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6w9z"
Sep 17 15:54:24.507232 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:24.507127    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f4bn" podUID="94d1451b-3eae-4481-a504-c29b4315e62e"
Sep 17 15:54:24.507232 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:24.507223    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m6w9z" podUID="7adc5269-e778-473a-a6c7-04754f5e585c"
Sep 17 15:54:25.507294 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:25.507257    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mgwn"
Sep 17 15:54:25.507717 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:25.507368    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mgwn" podUID="3114c3ad-7b7e-41ad-b175-ead7d92d5ee3"
Sep 17 15:54:26.507473 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:26.507436    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6w9z"
Sep 17 15:54:26.507916 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:26.507551    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m6w9z" podUID="7adc5269-e778-473a-a6c7-04754f5e585c"
Sep 17 15:54:26.507916 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:26.507597    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f4bn"
Sep 17 15:54:26.507916 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:26.507695    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f4bn" podUID="94d1451b-3eae-4481-a504-c29b4315e62e"
Sep 17 15:54:27.506512 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:27.506490    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mgwn"
Sep 17 15:54:27.506678 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:27.506624    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mgwn" podUID="3114c3ad-7b7e-41ad-b175-ead7d92d5ee3"
Sep 17 15:54:27.719338 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:27.719134    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ntl9t" event={"ID":"8249ef17-a5c4-414e-a2ca-a22cc46ef3eb","Type":"ContainerStarted","Data":"763a33c5bbeff6cebb6ca6c418b1346c224b942f0df6224bd1ede935478a81ad"}
Sep 17 15:54:27.720638 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:27.720615    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xhtm4" event={"ID":"c4438a49-8ea1-40aa-bdd9-2707192617af","Type":"ContainerStarted","Data":"f0d2a277bdca894d65f49ca1317e5b76acb3c000e7b7920768bc58b34bae62f9"}
Sep 17 15:54:27.721954 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:27.721930    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xhrb8" event={"ID":"4afdccbc-d33e-41ac-98fd-63585ff4d59b","Type":"ContainerStarted","Data":"13a2c77369732aa12edf540c892b766ea96a0d9fceb56139c7320037ed7b1235"}
Sep 17 15:54:27.723280 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:27.723257    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65v88" event={"ID":"8910d933-4fe9-45a0-be48-f225cded154a","Type":"ContainerStarted","Data":"6e760ef8bce06de2cb75ddc69ad6a1eed92d96a1c17a9e3d0fb130ac2e665445"}
Sep 17 15:54:27.725278 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:27.725260    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2b47p_a1850a4d-4d01-4a71-8636-b5918f1426c2/ovn-acl-logging/0.log"
Sep 17 15:54:27.725606 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:27.725589    2585 generic.go:356] "Generic (PLEG): container finished" podID="a1850a4d-4d01-4a71-8636-b5918f1426c2" containerID="447a023d5f63f583990c2de83cf482a71962b929122899a3e7e731df7f992aae" exitCode=1
Sep 17 15:54:27.725672 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:27.725653    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2b47p" event={"ID":"a1850a4d-4d01-4a71-8636-b5918f1426c2","Type":"ContainerStarted","Data":"7c73959290a4c05bee1c0f9fee9cc5b530b2f9f86b2c35b254e451a5fc768152"}
Sep 17 15:54:27.725723 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:27.725710    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2b47p" event={"ID":"a1850a4d-4d01-4a71-8636-b5918f1426c2","Type":"ContainerStarted","Data":"28d7456ca216e1e5a72ea8b83d5cf470c6c124912aea765a702683978fd6c1f8"}
Sep 17 15:54:27.725777 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:27.725765    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2b47p" event={"ID":"a1850a4d-4d01-4a71-8636-b5918f1426c2","Type":"ContainerDied","Data":"447a023d5f63f583990c2de83cf482a71962b929122899a3e7e731df7f992aae"}
Sep 17 15:54:27.725816 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:27.725800    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2b47p" event={"ID":"a1850a4d-4d01-4a71-8636-b5918f1426c2","Type":"ContainerStarted","Data":"597b7236088a4415840172610cbc6e11e6d273eab0fcac5bcd588e964b53a455"}
Sep 17 15:54:27.727053 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:27.726966    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dtbtf" event={"ID":"28231271-38ce-4985-9020-e3494fd366c7","Type":"ContainerStarted","Data":"7bfc80cbf6b6977b6f681b7b82d09accd5ddcc4ba763eef0c49486324b724d33"}
Sep 17 15:54:27.728523 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:27.728503    2585 generic.go:356] "Generic (PLEG): container finished" podID="7e89f60d-872c-4594-8555-427b0be13b7b" containerID="ab4b220cd29f9548a7cb547aabc7c2f5db49ceb286f150a8180bad0e811a4365" exitCode=0
Sep 17 15:54:27.728614 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:27.728567    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cm82k" event={"ID":"7e89f60d-872c-4594-8555-427b0be13b7b","Type":"ContainerDied","Data":"ab4b220cd29f9548a7cb547aabc7c2f5db49ceb286f150a8180bad0e811a4365"}
Sep 17 15:54:27.730169 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:27.730148    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cdgrj" event={"ID":"2ecfa8d8-297d-4e71-b29c-726646be98bf","Type":"ContainerStarted","Data":"25c5f803fd3bd571f8d58556cac13de3e458886656e100a800e69a501a514a47"}
Sep 17 15:54:27.787562 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:27.787522    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ntl9t" podStartSLOduration=8.133792732 podStartE2EDuration="21.787506011s" totalImagesPullingTime="13.653713279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:54:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:54:27.758030873 +0000 UTC m=+23.528362674" watchObservedRunningTime="2026-09-17 15:54:27.787506011 +0000 UTC m=+23.557837810"
Sep 17 15:54:27.787692 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:27.787675    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-xhrb8" podStartSLOduration=3.217568365 podStartE2EDuration="20.787668613s" totalImagesPullingTime="17.570100248s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:54:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:54:27.787336265 +0000 UTC m=+23.557668063" watchObservedRunningTime="2026-09-17 15:54:27.787668613 +0000 UTC m=+23.558000411"
Sep 17 15:54:27.815868 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:27.815816    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-cdgrj" podStartSLOduration=4.247457859 podStartE2EDuration="21.81579601s" totalImagesPullingTime="17.568338151s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:54:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:54:27.815332914 +0000 UTC m=+23.585664713" watchObservedRunningTime="2026-09-17 15:54:27.81579601 +0000 UTC m=+23.586127809"
Sep 17 15:54:27.867096 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:27.867033    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-dtbtf" podStartSLOduration=4.309396608 podStartE2EDuration="21.867011509s" totalImagesPullingTime="17.557614901s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:54:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:54:27.841587531 +0000 UTC m=+23.611919331" watchObservedRunningTime="2026-09-17 15:54:27.867011509 +0000 UTC m=+23.637343308"
Sep 17 15:54:27.867528 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:27.867486    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-xhtm4" podStartSLOduration=8.20858163 podStartE2EDuration="21.867477537s" totalImagesPullingTime="13.658895907s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:54:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:54:27.866970944 +0000 UTC m=+23.637302741" watchObservedRunningTime="2026-09-17 15:54:27.867477537 +0000 UTC m=+23.637809336"
Sep 17 15:54:28.506980 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:28.506799    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6w9z"
Sep 17 15:54:28.507133 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:28.506868    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f4bn"
Sep 17 15:54:28.507133 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:28.507068    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m6w9z" podUID="7adc5269-e778-473a-a6c7-04754f5e585c"
Sep 17 15:54:28.507201 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:28.507159    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f4bn" podUID="94d1451b-3eae-4481-a504-c29b4315e62e"
Sep 17 15:54:28.703832 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:28.703600    2585 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 15:54:28.734161 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:28.734117    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mvssh" event={"ID":"66be23a4-8549-4ae3-9d3e-33aca9dca8cb","Type":"ContainerStarted","Data":"d18d38db6750b29c536b9a4a930479ed33918a5c02628687a44617390aa4a339"}
Sep 17 15:54:28.736227 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:28.736195    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65v88" event={"ID":"8910d933-4fe9-45a0-be48-f225cded154a","Type":"ContainerStarted","Data":"0cedb25295063262ff8a4435da3e4fbbb4d4635ae0b716e929b8e3c5bc1dfd7b"}
Sep 17 15:54:28.739513 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:28.739487    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2b47p_a1850a4d-4d01-4a71-8636-b5918f1426c2/ovn-acl-logging/0.log"
Sep 17 15:54:28.739881 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:28.739857    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2b47p" event={"ID":"a1850a4d-4d01-4a71-8636-b5918f1426c2","Type":"ContainerStarted","Data":"d41ccc16880065ac36feec0dc295addab7cad7c6bc86fa6dbf5bfd482e1e021e"}
Sep 17 15:54:28.739992 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:28.739931    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2b47p" event={"ID":"a1850a4d-4d01-4a71-8636-b5918f1426c2","Type":"ContainerStarted","Data":"820de365f2bacd83082111e43570da6bc8d48ba9e261c219bd15081f2eb1122b"}
Sep 17 15:54:28.760789 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:28.760678    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ntl9t"
Sep 17 15:54:28.761214 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:28.761192    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ntl9t"
Sep 17 15:54:28.788836 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:28.788785    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-mvssh" podStartSLOduration=4.222336364 podStartE2EDuration="21.788768774s" totalImagesPullingTime="17.56643241s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:54:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:54:28.760224714 +0000 UTC m=+24.530556514" watchObservedRunningTime="2026-09-17 15:54:28.788768774 +0000 UTC m=+24.559100572"
Sep 17 15:54:29.453437 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:29.453318    2585 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T15:54:28.703631649Z","UUID":"e49464ce-3920-4c28-a47e-2bc95c2f17ed","Handler":null,"Name":"","Endpoint":""}
Sep 17 15:54:29.455286 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:29.455261    2585 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 15:54:29.455431 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:29.455298    2585 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 15:54:29.507215 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:29.507182    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mgwn"
Sep 17 15:54:29.507389 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:29.507291    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mgwn" podUID="3114c3ad-7b7e-41ad-b175-ead7d92d5ee3"
Sep 17 15:54:29.742467 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:29.742440    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ntl9t"
Sep 17 15:54:29.742959 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:29.742936    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ntl9t"
Sep 17 15:54:30.506532 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:30.506497    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6w9z"
Sep 17 15:54:30.506717 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:30.506497    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f4bn"
Sep 17 15:54:30.506717 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:30.506638    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m6w9z" podUID="7adc5269-e778-473a-a6c7-04754f5e585c"
Sep 17 15:54:30.506717 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:30.506669    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f4bn" podUID="94d1451b-3eae-4481-a504-c29b4315e62e"
Sep 17 15:54:30.746634 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:30.746595    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65v88" event={"ID":"8910d933-4fe9-45a0-be48-f225cded154a","Type":"ContainerStarted","Data":"f2ab4d5fc8a3e99743086c7c11914fd22c13535d0417b71703f7e41b7926376b"}
Sep 17 15:54:30.750101 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:30.750079    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2b47p_a1850a4d-4d01-4a71-8636-b5918f1426c2/ovn-acl-logging/0.log"
Sep 17 15:54:30.750573 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:30.750541    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2b47p" event={"ID":"a1850a4d-4d01-4a71-8636-b5918f1426c2","Type":"ContainerStarted","Data":"7d98ddf1dc1c2d90a602beeacd3b827dbac6811a68dc7dacb4d1ac14395c12fc"}
Sep 17 15:54:30.798673 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:30.798570    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-65v88" podStartSLOduration=5.069892812 podStartE2EDuration="24.798548349s" totalImagesPullingTime="19.728655537s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:54:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:54:30.797851804 +0000 UTC m=+26.568183602" watchObservedRunningTime="2026-09-17 15:54:30.798548349 +0000 UTC m=+26.568880150"
Sep 17 15:54:31.506528 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:31.506495    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mgwn"
Sep 17 15:54:31.506718 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:31.506616    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mgwn" podUID="3114c3ad-7b7e-41ad-b175-ead7d92d5ee3"
Sep 17 15:54:32.507564 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:32.507366    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6w9z"
Sep 17 15:54:32.508184 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:32.507371    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f4bn"
Sep 17 15:54:32.508184 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:32.507679    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m6w9z" podUID="7adc5269-e778-473a-a6c7-04754f5e585c"
Sep 17 15:54:32.508184 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:32.507700    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f4bn" podUID="94d1451b-3eae-4481-a504-c29b4315e62e"
Sep 17 15:54:32.759220 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:32.759192    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2b47p_a1850a4d-4d01-4a71-8636-b5918f1426c2/ovn-acl-logging/0.log"
Sep 17 15:54:32.759630 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:32.759602    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2b47p" event={"ID":"a1850a4d-4d01-4a71-8636-b5918f1426c2","Type":"ContainerStarted","Data":"251d16bf6283330aaceba41b33e87de4cf4b319460357e230b810aad61d2950b"}
Sep 17 15:54:32.759907 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:32.759868    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:32.760051 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:32.760036    2585 scope.go:117] "RemoveContainer" containerID="447a023d5f63f583990c2de83cf482a71962b929122899a3e7e731df7f992aae"
Sep 17 15:54:32.761746 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:32.761716    2585 generic.go:356] "Generic (PLEG): container finished" podID="7e89f60d-872c-4594-8555-427b0be13b7b" containerID="79653f35dbaf35c5873d66c723944a58e5556c777e4066f3957abfaf44d8b164" exitCode=0
Sep 17 15:54:32.761846 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:32.761751    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cm82k" event={"ID":"7e89f60d-872c-4594-8555-427b0be13b7b","Type":"ContainerDied","Data":"79653f35dbaf35c5873d66c723944a58e5556c777e4066f3957abfaf44d8b164"}
Sep 17 15:54:32.778997 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:32.778972    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:33.506856 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:33.506825    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mgwn"
Sep 17 15:54:33.506991 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:33.506948    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mgwn" podUID="3114c3ad-7b7e-41ad-b175-ead7d92d5ee3"
Sep 17 15:54:33.768858 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:33.768789    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2b47p_a1850a4d-4d01-4a71-8636-b5918f1426c2/ovn-acl-logging/0.log"
Sep 17 15:54:33.769440 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:33.769125    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2b47p" event={"ID":"a1850a4d-4d01-4a71-8636-b5918f1426c2","Type":"ContainerStarted","Data":"b5cd0875f2cae6207c004f553d71fdde4f747178d15a1e64c5fc48ac4475d966"}
Sep 17 15:54:33.769440 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:33.769284    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:33.769440 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:33.769323    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:33.783880 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:33.783855    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:54:33.816392 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:33.816346    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2b47p" podStartSLOduration=9.165868685 podStartE2EDuration="26.816331766s" totalImagesPullingTime="17.650463081s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:54:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:54:33.813467159 +0000 UTC m=+29.583798959" watchObservedRunningTime="2026-09-17 15:54:33.816331766 +0000 UTC m=+29.586663546"
Sep 17 15:54:34.192972 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:34.192705    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2f4bn"]
Sep 17 15:54:34.193107 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:34.192998    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f4bn"
Sep 17 15:54:34.193107 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:34.193086    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f4bn" podUID="94d1451b-3eae-4481-a504-c29b4315e62e"
Sep 17 15:54:34.203994 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:34.203963    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9mgwn"]
Sep 17 15:54:34.204130 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:34.204057    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mgwn"
Sep 17 15:54:34.204200 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:34.204128    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mgwn" podUID="3114c3ad-7b7e-41ad-b175-ead7d92d5ee3"
Sep 17 15:54:34.208918 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:34.208890    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-m6w9z"]
Sep 17 15:54:34.209024 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:34.208988    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6w9z"
Sep 17 15:54:34.209076 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:34.209061    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m6w9z" podUID="7adc5269-e778-473a-a6c7-04754f5e585c"
Sep 17 15:54:34.774274 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:34.774238    2585 generic.go:356] "Generic (PLEG): container finished" podID="7e89f60d-872c-4594-8555-427b0be13b7b" containerID="09c9acf9945a9bbc9a5f51d1789c5a1979328b247aea44c868bd0c75c8ff79b8" exitCode=0
Sep 17 15:54:34.774805 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:34.774318    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cm82k" event={"ID":"7e89f60d-872c-4594-8555-427b0be13b7b","Type":"ContainerDied","Data":"09c9acf9945a9bbc9a5f51d1789c5a1979328b247aea44c868bd0c75c8ff79b8"}
Sep 17 15:54:35.506607 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:35.506573    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6w9z"
Sep 17 15:54:35.506607 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:35.506602    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mgwn"
Sep 17 15:54:35.506825 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:35.506574    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f4bn"
Sep 17 15:54:35.506825 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:35.506703    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m6w9z" podUID="7adc5269-e778-473a-a6c7-04754f5e585c"
Sep 17 15:54:35.506889 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:35.506826    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f4bn" podUID="94d1451b-3eae-4481-a504-c29b4315e62e"
Sep 17 15:54:35.506926 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:35.506888    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mgwn" podUID="3114c3ad-7b7e-41ad-b175-ead7d92d5ee3"
Sep 17 15:54:36.780482 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:36.780447    2585 generic.go:356] "Generic (PLEG): container finished" podID="7e89f60d-872c-4594-8555-427b0be13b7b" containerID="e393e3a9416323c3e3ee8ad889d241b524a6c52a122921c699e45c14efcb7fbe" exitCode=0
Sep 17 15:54:36.780883 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:36.780497    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cm82k" event={"ID":"7e89f60d-872c-4594-8555-427b0be13b7b","Type":"ContainerDied","Data":"e393e3a9416323c3e3ee8ad889d241b524a6c52a122921c699e45c14efcb7fbe"}
Sep 17 15:54:37.507218 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:37.507174    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mgwn"
Sep 17 15:54:37.507415 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:37.507315    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6w9z"
Sep 17 15:54:37.507415 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:37.507314    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mgwn" podUID="3114c3ad-7b7e-41ad-b175-ead7d92d5ee3"
Sep 17 15:54:37.507415 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:37.507400    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f4bn"
Sep 17 15:54:37.507577 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:37.507513    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f4bn" podUID="94d1451b-3eae-4481-a504-c29b4315e62e"
Sep 17 15:54:37.507624 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:37.507606    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m6w9z" podUID="7adc5269-e778-473a-a6c7-04754f5e585c"
Sep 17 15:54:39.507096 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:39.507058    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6w9z"
Sep 17 15:54:39.507522 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:39.507058    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mgwn"
Sep 17 15:54:39.507522 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:39.507198    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m6w9z" podUID="7adc5269-e778-473a-a6c7-04754f5e585c"
Sep 17 15:54:39.507522 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:39.507263    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9mgwn" podUID="3114c3ad-7b7e-41ad-b175-ead7d92d5ee3"
Sep 17 15:54:39.507522 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:39.507058    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f4bn"
Sep 17 15:54:39.507522 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:39.507389    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2f4bn" podUID="94d1451b-3eae-4481-a504-c29b4315e62e"
Sep 17 15:54:40.131628 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:40.131556    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-43.ec2.internal" event="NodeReady"
Sep 17 15:54:40.131783 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:40.131706    2585 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 15:54:40.213683 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:40.213651    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7adc5269-e778-473a-a6c7-04754f5e585c-metrics-certs\") pod \"network-metrics-daemon-m6w9z\" (UID: \"7adc5269-e778-473a-a6c7-04754f5e585c\") " pod="openshift-multus/network-metrics-daemon-m6w9z"
Sep 17 15:54:40.213888 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:40.213715    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs87t\" (UniqueName: \"kubernetes.io/projected/3114c3ad-7b7e-41ad-b175-ead7d92d5ee3-kube-api-access-rs87t\") pod \"network-check-target-9mgwn\" (UID: \"3114c3ad-7b7e-41ad-b175-ead7d92d5ee3\") " pod="openshift-network-diagnostics/network-check-target-9mgwn"
Sep 17 15:54:40.213888 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:40.213775    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/94d1451b-3eae-4481-a504-c29b4315e62e-original-pull-secret\") pod \"global-pull-secret-syncer-2f4bn\" (UID: \"94d1451b-3eae-4481-a504-c29b4315e62e\") " pod="kube-system/global-pull-secret-syncer-2f4bn"
Sep 17 15:54:40.213888 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:40.213828    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:54:40.214046 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:40.213883    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:54:40.214046 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:40.213907    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:54:40.214046 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:40.213912    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7adc5269-e778-473a-a6c7-04754f5e585c-metrics-certs podName:7adc5269-e778-473a-a6c7-04754f5e585c nodeName:}" failed. No retries permitted until 2026-09-17 15:55:12.213891812 +0000 UTC m=+67.984223606 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7adc5269-e778-473a-a6c7-04754f5e585c-metrics-certs") pod "network-metrics-daemon-m6w9z" (UID: "7adc5269-e778-473a-a6c7-04754f5e585c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:54:40.214046 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:40.213921    2585 projected.go:196] Error preparing data for projected volume kube-api-access-rs87t for pod openshift-network-diagnostics/network-check-target-9mgwn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:54:40.214046 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:40.213891    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:54:40.214046 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:40.213975    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3114c3ad-7b7e-41ad-b175-ead7d92d5ee3-kube-api-access-rs87t podName:3114c3ad-7b7e-41ad-b175-ead7d92d5ee3 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:12.213959723 +0000 UTC m=+67.984291516 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-rs87t" (UniqueName: "kubernetes.io/projected/3114c3ad-7b7e-41ad-b175-ead7d92d5ee3-kube-api-access-rs87t") pod "network-check-target-9mgwn" (UID: "3114c3ad-7b7e-41ad-b175-ead7d92d5ee3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:54:40.214046 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:40.214010    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94d1451b-3eae-4481-a504-c29b4315e62e-original-pull-secret podName:94d1451b-3eae-4481-a504-c29b4315e62e nodeName:}" failed. No retries permitted until 2026-09-17 15:55:12.213997992 +0000 UTC m=+67.984329767 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/94d1451b-3eae-4481-a504-c29b4315e62e-original-pull-secret") pod "global-pull-secret-syncer-2f4bn" (UID: "94d1451b-3eae-4481-a504-c29b4315e62e") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:54:40.229458 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:40.229414    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-cvmck"]
Sep 17 15:54:40.258634 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:40.258601    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6qpcq"]
Sep 17 15:54:40.259091 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:40.258739    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cvmck"
Sep 17 15:54:40.264022 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:40.263995    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qcdnd\""
Sep 17 15:54:40.264366 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:40.264345    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 15:54:40.264495 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:40.264381    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 15:54:40.278482 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:40.278461    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cvmck"]
Sep 17 15:54:40.278589 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:40.278576    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6qpcq"
Sep 17 15:54:40.283784 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:40.283764    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 15:54:40.283887 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:40.283837    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 15:54:40.284131 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:40.284113    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 15:54:40.284207 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:40.284185    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-j6zt9\""
Sep 17 15:54:40.290456 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:40.287741    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6qpcq"]
Sep 17 15:54:40.416288 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:40.416208    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bxlb\" (UniqueName: \"kubernetes.io/projected/5cad5942-b49d-4bad-b6c4-d881654b6ef5-kube-api-access-8bxlb\") pod \"dns-default-cvmck\" (UID: \"5cad5942-b49d-4bad-b6c4-d881654b6ef5\") " pod="openshift-dns/dns-default-cvmck"
Sep 17 15:54:40.416288 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:40.416255    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5cad5942-b49d-4bad-b6c4-d881654b6ef5-metrics-tls\") pod \"dns-default-cvmck\" (UID: \"5cad5942-b49d-4bad-b6c4-d881654b6ef5\") " pod="openshift-dns/dns-default-cvmck"
Sep 17 15:54:40.416288 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:40.416284    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8x2s\" (UniqueName: \"kubernetes.io/projected/1f3d6b60-625e-4015-942c-6c8150fa364f-kube-api-access-b8x2s\") pod \"ingress-canary-6qpcq\" (UID: \"1f3d6b60-625e-4015-942c-6c8150fa364f\") " pod="openshift-ingress-canary/ingress-canary-6qpcq"
Sep 17 15:54:40.416585 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:40.416346    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5cad5942-b49d-4bad-b6c4-d881654b6ef5-config-volume\") pod \"dns-default-cvmck\" (UID: \"5cad5942-b49d-4bad-b6c4-d881654b6ef5\") " pod="openshift-dns/dns-default-cvmck"
Sep 17 15:54:40.416585 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:40.416394    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cad5942-b49d-4bad-b6c4-d881654b6ef5-tmp-dir\") pod \"dns-default-cvmck\" (UID: \"5cad5942-b49d-4bad-b6c4-d881654b6ef5\") " pod="openshift-dns/dns-default-cvmck"
Sep 17 15:54:40.416585 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:40.416454    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1f3d6b60-625e-4015-942c-6c8150fa364f-cert\") pod \"ingress-canary-6qpcq\" (UID: \"1f3d6b60-625e-4015-942c-6c8150fa364f\") " pod="openshift-ingress-canary/ingress-canary-6qpcq"
Sep 17 15:54:40.516770 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:40.516741    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bxlb\" (UniqueName: \"kubernetes.io/projected/5cad5942-b49d-4bad-b6c4-d881654b6ef5-kube-api-access-8bxlb\") pod \"dns-default-cvmck\" (UID: \"5cad5942-b49d-4bad-b6c4-d881654b6ef5\") " pod="openshift-dns/dns-default-cvmck"
Sep 17 15:54:40.517209 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:40.516782    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5cad5942-b49d-4bad-b6c4-d881654b6ef5-metrics-tls\") pod \"dns-default-cvmck\" (UID: \"5cad5942-b49d-4bad-b6c4-d881654b6ef5\") " pod="openshift-dns/dns-default-cvmck"
Sep 17 15:54:40.517209 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:40.516802    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8x2s\" (UniqueName: \"kubernetes.io/projected/1f3d6b60-625e-4015-942c-6c8150fa364f-kube-api-access-b8x2s\") pod \"ingress-canary-6qpcq\" (UID: \"1f3d6b60-625e-4015-942c-6c8150fa364f\") " pod="openshift-ingress-canary/ingress-canary-6qpcq"
Sep 17 15:54:40.517209 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:40.516842    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5cad5942-b49d-4bad-b6c4-d881654b6ef5-config-volume\") pod \"dns-default-cvmck\" (UID: \"5cad5942-b49d-4bad-b6c4-d881654b6ef5\") " pod="openshift-dns/dns-default-cvmck"
Sep 17 15:54:40.517209 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:40.516873    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cad5942-b49d-4bad-b6c4-d881654b6ef5-tmp-dir\") pod \"dns-default-cvmck\" (UID: \"5cad5942-b49d-4bad-b6c4-d881654b6ef5\") " pod="openshift-dns/dns-default-cvmck"
Sep 17 15:54:40.517209 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:40.516903    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1f3d6b60-625e-4015-942c-6c8150fa364f-cert\") pod \"ingress-canary-6qpcq\" (UID: \"1f3d6b60-625e-4015-942c-6c8150fa364f\") " pod="openshift-ingress-canary/ingress-canary-6qpcq"
Sep 17 15:54:40.517209 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:40.516917    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:54:40.517209 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:40.516989    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cad5942-b49d-4bad-b6c4-d881654b6ef5-metrics-tls podName:5cad5942-b49d-4bad-b6c4-d881654b6ef5 nodeName:}" failed. No retries permitted until 2026-09-17 15:54:41.016967709 +0000 UTC m=+36.787299488 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5cad5942-b49d-4bad-b6c4-d881654b6ef5-metrics-tls") pod "dns-default-cvmck" (UID: "5cad5942-b49d-4bad-b6c4-d881654b6ef5") : secret "dns-default-metrics-tls" not found
Sep 17 15:54:40.517627 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:40.517271    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cad5942-b49d-4bad-b6c4-d881654b6ef5-tmp-dir\") pod \"dns-default-cvmck\" (UID: \"5cad5942-b49d-4bad-b6c4-d881654b6ef5\") " pod="openshift-dns/dns-default-cvmck"
Sep 17 15:54:40.517627 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:40.517602    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5cad5942-b49d-4bad-b6c4-d881654b6ef5-config-volume\") pod \"dns-default-cvmck\" (UID: \"5cad5942-b49d-4bad-b6c4-d881654b6ef5\") " pod="openshift-dns/dns-default-cvmck"
Sep 17 15:54:40.521386 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:40.521366    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1f3d6b60-625e-4015-942c-6c8150fa364f-cert\") pod \"ingress-canary-6qpcq\" (UID: \"1f3d6b60-625e-4015-942c-6c8150fa364f\") " pod="openshift-ingress-canary/ingress-canary-6qpcq"
Sep 17 15:54:40.537986 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:40.537838    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8x2s\" (UniqueName: \"kubernetes.io/projected/1f3d6b60-625e-4015-942c-6c8150fa364f-kube-api-access-b8x2s\") pod \"ingress-canary-6qpcq\" (UID: \"1f3d6b60-625e-4015-942c-6c8150fa364f\") " pod="openshift-ingress-canary/ingress-canary-6qpcq"
Sep 17 15:54:40.540876 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:40.540852    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bxlb\" (UniqueName: \"kubernetes.io/projected/5cad5942-b49d-4bad-b6c4-d881654b6ef5-kube-api-access-8bxlb\") pod \"dns-default-cvmck\" (UID: \"5cad5942-b49d-4bad-b6c4-d881654b6ef5\") " pod="openshift-dns/dns-default-cvmck"
Sep 17 15:54:40.588848 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:40.588804    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6qpcq"
Sep 17 15:54:41.020879 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:41.020797    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5cad5942-b49d-4bad-b6c4-d881654b6ef5-metrics-tls\") pod \"dns-default-cvmck\" (UID: \"5cad5942-b49d-4bad-b6c4-d881654b6ef5\") " pod="openshift-dns/dns-default-cvmck"
Sep 17 15:54:41.021023 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:41.020935    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:54:41.021023 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:41.021015    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cad5942-b49d-4bad-b6c4-d881654b6ef5-metrics-tls podName:5cad5942-b49d-4bad-b6c4-d881654b6ef5 nodeName:}" failed. No retries permitted until 2026-09-17 15:54:42.020996535 +0000 UTC m=+37.791328329 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5cad5942-b49d-4bad-b6c4-d881654b6ef5-metrics-tls") pod "dns-default-cvmck" (UID: "5cad5942-b49d-4bad-b6c4-d881654b6ef5") : secret "dns-default-metrics-tls" not found
Sep 17 15:54:41.034500 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:41.034472    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6qpcq"]
Sep 17 15:54:41.037835 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:54:41.037811    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f3d6b60_625e_4015_942c_6c8150fa364f.slice/crio-5e195bf53eaa635eb1c62f917af3e22e486ccd3e15a7c081cc70c7f82425d4f7 WatchSource:0}: Error finding container 5e195bf53eaa635eb1c62f917af3e22e486ccd3e15a7c081cc70c7f82425d4f7: Status 404 returned error can't find the container with id 5e195bf53eaa635eb1c62f917af3e22e486ccd3e15a7c081cc70c7f82425d4f7
Sep 17 15:54:41.507436 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:41.507385    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6w9z"
Sep 17 15:54:41.507436 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:41.507412    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f4bn"
Sep 17 15:54:41.507658 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:41.507430    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mgwn"
Sep 17 15:54:41.511381 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:41.511274    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hxr8k\""
Sep 17 15:54:41.512182 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:41.512164    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 15:54:41.515525 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:41.515501    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 15:54:41.515624 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:41.515532    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 15:54:41.515795 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:41.515778    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 15:54:41.516675 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:41.516521    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9smtj\""
Sep 17 15:54:41.795965 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:41.795932    2585 generic.go:356] "Generic (PLEG): container finished" podID="7e89f60d-872c-4594-8555-427b0be13b7b" containerID="2deb3fef61b81c8fae950c722d14e4129c2b43abce8b227f7e200e7078ed4633" exitCode=0
Sep 17 15:54:41.796744 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:41.796013    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cm82k" event={"ID":"7e89f60d-872c-4594-8555-427b0be13b7b","Type":"ContainerDied","Data":"2deb3fef61b81c8fae950c722d14e4129c2b43abce8b227f7e200e7078ed4633"}
Sep 17 15:54:41.797448 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:41.797406    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6qpcq" event={"ID":"1f3d6b60-625e-4015-942c-6c8150fa364f","Type":"ContainerStarted","Data":"5e195bf53eaa635eb1c62f917af3e22e486ccd3e15a7c081cc70c7f82425d4f7"}
Sep 17 15:54:42.029917 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:42.029887    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5cad5942-b49d-4bad-b6c4-d881654b6ef5-metrics-tls\") pod \"dns-default-cvmck\" (UID: \"5cad5942-b49d-4bad-b6c4-d881654b6ef5\") " pod="openshift-dns/dns-default-cvmck"
Sep 17 15:54:42.030061 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:42.030039    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:54:42.030122 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:42.030112    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cad5942-b49d-4bad-b6c4-d881654b6ef5-metrics-tls podName:5cad5942-b49d-4bad-b6c4-d881654b6ef5 nodeName:}" failed. No retries permitted until 2026-09-17 15:54:44.030096364 +0000 UTC m=+39.800428139 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5cad5942-b49d-4bad-b6c4-d881654b6ef5-metrics-tls") pod "dns-default-cvmck" (UID: "5cad5942-b49d-4bad-b6c4-d881654b6ef5") : secret "dns-default-metrics-tls" not found
Sep 17 15:54:42.803842 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:42.803811    2585 generic.go:356] "Generic (PLEG): container finished" podID="7e89f60d-872c-4594-8555-427b0be13b7b" containerID="a4af0ce73686635aa088738bd14bdae7fe19ed3bbf6d195780be618a3ce38a49" exitCode=0
Sep 17 15:54:42.804337 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:42.803867    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cm82k" event={"ID":"7e89f60d-872c-4594-8555-427b0be13b7b","Type":"ContainerDied","Data":"a4af0ce73686635aa088738bd14bdae7fe19ed3bbf6d195780be618a3ce38a49"}
Sep 17 15:54:43.809239 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:43.809201    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cm82k" event={"ID":"7e89f60d-872c-4594-8555-427b0be13b7b","Type":"ContainerStarted","Data":"9ce3d0ead3e512db48dc335c66a67ba6eea528669e841e9e1c5fc56362a8bcd4"}
Sep 17 15:54:43.810658 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:43.810635    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6qpcq" event={"ID":"1f3d6b60-625e-4015-942c-6c8150fa364f","Type":"ContainerStarted","Data":"90c67fd11ea9156182cc94c3ec6b7e7f14955697fc45388de644a75bef168e73"}
Sep 17 15:54:43.839500 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:43.839453    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-cm82k" podStartSLOduration=9.311888206 podStartE2EDuration="37.839439563s" totalImagesPullingTime="28.527551357s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:54:06 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:54:43.83788146 +0000 UTC m=+39.608213258" watchObservedRunningTime="2026-09-17 15:54:43.839439563 +0000 UTC m=+39.609771355"
Sep 17 15:54:43.865951 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:43.865905    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6qpcq" podStartSLOduration=1.775455804 podStartE2EDuration="3.865890399s" totalImagesPullingTime="2.090434595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:54:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:54:43.865412151 +0000 UTC m=+39.635743949" watchObservedRunningTime="2026-09-17 15:54:43.865890399 +0000 UTC m=+39.636222221"
Sep 17 15:54:44.048651 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:44.048564    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5cad5942-b49d-4bad-b6c4-d881654b6ef5-metrics-tls\") pod \"dns-default-cvmck\" (UID: \"5cad5942-b49d-4bad-b6c4-d881654b6ef5\") " pod="openshift-dns/dns-default-cvmck"
Sep 17 15:54:44.048782 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:44.048717    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:54:44.048782 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:44.048773    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cad5942-b49d-4bad-b6c4-d881654b6ef5-metrics-tls podName:5cad5942-b49d-4bad-b6c4-d881654b6ef5 nodeName:}" failed. No retries permitted until 2026-09-17 15:54:48.048760729 +0000 UTC m=+43.819092509 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5cad5942-b49d-4bad-b6c4-d881654b6ef5-metrics-tls") pod "dns-default-cvmck" (UID: "5cad5942-b49d-4bad-b6c4-d881654b6ef5") : secret "dns-default-metrics-tls" not found
Sep 17 15:54:48.074616 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:48.074576    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5cad5942-b49d-4bad-b6c4-d881654b6ef5-metrics-tls\") pod \"dns-default-cvmck\" (UID: \"5cad5942-b49d-4bad-b6c4-d881654b6ef5\") " pod="openshift-dns/dns-default-cvmck"
Sep 17 15:54:48.075018 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:48.074728    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:54:48.075018 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:48.074791    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cad5942-b49d-4bad-b6c4-d881654b6ef5-metrics-tls podName:5cad5942-b49d-4bad-b6c4-d881654b6ef5 nodeName:}" failed. No retries permitted until 2026-09-17 15:54:56.074774106 +0000 UTC m=+51.845105891 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5cad5942-b49d-4bad-b6c4-d881654b6ef5-metrics-tls") pod "dns-default-cvmck" (UID: "5cad5942-b49d-4bad-b6c4-d881654b6ef5") : secret "dns-default-metrics-tls" not found
Sep 17 15:54:56.134651 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:54:56.134612    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5cad5942-b49d-4bad-b6c4-d881654b6ef5-metrics-tls\") pod \"dns-default-cvmck\" (UID: \"5cad5942-b49d-4bad-b6c4-d881654b6ef5\") " pod="openshift-dns/dns-default-cvmck"
Sep 17 15:54:56.135218 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:56.134785    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:54:56.135218 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:54:56.134882    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cad5942-b49d-4bad-b6c4-d881654b6ef5-metrics-tls podName:5cad5942-b49d-4bad-b6c4-d881654b6ef5 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:12.134859534 +0000 UTC m=+67.905191310 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5cad5942-b49d-4bad-b6c4-d881654b6ef5-metrics-tls") pod "dns-default-cvmck" (UID: "5cad5942-b49d-4bad-b6c4-d881654b6ef5") : secret "dns-default-metrics-tls" not found
Sep 17 15:55:05.798955 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:05.798923    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2b47p"
Sep 17 15:55:12.141168 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:12.141130    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5cad5942-b49d-4bad-b6c4-d881654b6ef5-metrics-tls\") pod \"dns-default-cvmck\" (UID: \"5cad5942-b49d-4bad-b6c4-d881654b6ef5\") " pod="openshift-dns/dns-default-cvmck"
Sep 17 15:55:12.141569 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:12.141279    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:55:12.141569 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:12.141357    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cad5942-b49d-4bad-b6c4-d881654b6ef5-metrics-tls podName:5cad5942-b49d-4bad-b6c4-d881654b6ef5 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:44.141339497 +0000 UTC m=+99.911671273 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5cad5942-b49d-4bad-b6c4-d881654b6ef5-metrics-tls") pod "dns-default-cvmck" (UID: "5cad5942-b49d-4bad-b6c4-d881654b6ef5") : secret "dns-default-metrics-tls" not found
Sep 17 15:55:12.241655 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:12.241604    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7adc5269-e778-473a-a6c7-04754f5e585c-metrics-certs\") pod \"network-metrics-daemon-m6w9z\" (UID: \"7adc5269-e778-473a-a6c7-04754f5e585c\") " pod="openshift-multus/network-metrics-daemon-m6w9z"
Sep 17 15:55:12.241771 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:12.241695    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs87t\" (UniqueName: \"kubernetes.io/projected/3114c3ad-7b7e-41ad-b175-ead7d92d5ee3-kube-api-access-rs87t\") pod \"network-check-target-9mgwn\" (UID: \"3114c3ad-7b7e-41ad-b175-ead7d92d5ee3\") " pod="openshift-network-diagnostics/network-check-target-9mgwn"
Sep 17 15:55:12.241771 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:12.241744    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/94d1451b-3eae-4481-a504-c29b4315e62e-original-pull-secret\") pod \"global-pull-secret-syncer-2f4bn\" (UID: \"94d1451b-3eae-4481-a504-c29b4315e62e\") " pod="kube-system/global-pull-secret-syncer-2f4bn"
Sep 17 15:55:12.245731 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:12.245712    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 15:55:12.245888 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:12.245871    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 15:55:12.246289 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:12.246275    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 15:55:12.252680 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:12.252662    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 15:55:12.252741 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:12.252730    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7adc5269-e778-473a-a6c7-04754f5e585c-metrics-certs podName:7adc5269-e778-473a-a6c7-04754f5e585c nodeName:}" failed. No retries permitted until 2026-09-17 15:56:16.252711304 +0000 UTC m=+132.023043083 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7adc5269-e778-473a-a6c7-04754f5e585c-metrics-certs") pod "network-metrics-daemon-m6w9z" (UID: "7adc5269-e778-473a-a6c7-04754f5e585c") : secret "metrics-daemon-secret" not found
Sep 17 15:55:12.255700 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:12.255675    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/94d1451b-3eae-4481-a504-c29b4315e62e-original-pull-secret\") pod \"global-pull-secret-syncer-2f4bn\" (UID: \"94d1451b-3eae-4481-a504-c29b4315e62e\") " pod="kube-system/global-pull-secret-syncer-2f4bn"
Sep 17 15:55:12.255974 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:12.255960    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 15:55:12.265046 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:12.265023    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs87t\" (UniqueName: \"kubernetes.io/projected/3114c3ad-7b7e-41ad-b175-ead7d92d5ee3-kube-api-access-rs87t\") pod \"network-check-target-9mgwn\" (UID: \"3114c3ad-7b7e-41ad-b175-ead7d92d5ee3\") " pod="openshift-network-diagnostics/network-check-target-9mgwn"
Sep 17 15:55:12.418886 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:12.418810    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2f4bn"
Sep 17 15:55:12.434521 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:12.434498    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hxr8k\""
Sep 17 15:55:12.442088 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:12.442067    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9mgwn"
Sep 17 15:55:12.703749 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:12.703720    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2f4bn"]
Sep 17 15:55:12.726689 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:12.726619    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9mgwn"]
Sep 17 15:55:12.730861 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:55:12.730833    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3114c3ad_7b7e_41ad_b175_ead7d92d5ee3.slice/crio-861db4bd902a858b51e96a595481b103509856bbd5e1bfbba4307f54641e7fd3 WatchSource:0}: Error finding container 861db4bd902a858b51e96a595481b103509856bbd5e1bfbba4307f54641e7fd3: Status 404 returned error can't find the container with id 861db4bd902a858b51e96a595481b103509856bbd5e1bfbba4307f54641e7fd3
Sep 17 15:55:12.880207 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:12.880175    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9mgwn" event={"ID":"3114c3ad-7b7e-41ad-b175-ead7d92d5ee3","Type":"ContainerStarted","Data":"861db4bd902a858b51e96a595481b103509856bbd5e1bfbba4307f54641e7fd3"}
Sep 17 15:55:12.881338 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:12.881305    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2f4bn" event={"ID":"94d1451b-3eae-4481-a504-c29b4315e62e","Type":"ContainerStarted","Data":"af753608446b2ffd78852352e8796125f616c011c5fc107c291ff9349db34d09"}
Sep 17 15:55:17.897570 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:17.897525    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9mgwn" event={"ID":"3114c3ad-7b7e-41ad-b175-ead7d92d5ee3","Type":"ContainerStarted","Data":"33d8ad9b46697677b4124272281c5426931886031e7351b70b036309347c520c"}
Sep 17 15:55:17.898029 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:17.897652    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9mgwn"
Sep 17 15:55:17.899162 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:17.899133    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2f4bn" event={"ID":"94d1451b-3eae-4481-a504-c29b4315e62e","Type":"ContainerStarted","Data":"882043e5fa8b62365bf2dc57785db2fde8e1e535e4b067507e4a79241a920457"}
Sep 17 15:55:17.955917 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:17.955867    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9mgwn" podStartSLOduration=67.62109139 podStartE2EDuration="1m11.955850696s" totalImagesPullingTime="4.334759306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:54:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:55:17.954651865 +0000 UTC m=+73.724983674" watchObservedRunningTime="2026-09-17 15:55:17.955850696 +0000 UTC m=+73.726182493"
Sep 17 15:55:17.985031 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:17.984972    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-2f4bn" podStartSLOduration=66.623019809 podStartE2EDuration="1m10.984954435s" totalImagesPullingTime="4.361934626s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:54:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:55:17.984370603 +0000 UTC m=+73.754702401" watchObservedRunningTime="2026-09-17 15:55:17.984954435 +0000 UTC m=+73.755286233"
Sep 17 15:55:22.847945 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:22.847910    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-gsbw2"]
Sep 17 15:55:22.849699 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:22.849678    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-gsbw2"
Sep 17 15:55:22.855112 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:22.855083    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 17 15:55:22.856369 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:22.856344    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 15:55:22.856470 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:22.856397    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 15:55:22.857811 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:22.857792    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-5dkcq\""
Sep 17 15:55:22.857871 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:22.857858    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 17 15:55:22.866521 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:22.866500    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-gsbw2"]
Sep 17 15:55:22.919397 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:22.919364    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0b74e69e-14e3-41ab-a94c-44487eafb173-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-gsbw2\" (UID: \"0b74e69e-14e3-41ab-a94c-44487eafb173\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-gsbw2"
Sep 17 15:55:22.919578 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:22.919453    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0b74e69e-14e3-41ab-a94c-44487eafb173-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-gsbw2\" (UID: \"0b74e69e-14e3-41ab-a94c-44487eafb173\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-gsbw2"
Sep 17 15:55:22.919578 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:22.919480    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnl7j\" (UniqueName: \"kubernetes.io/projected/0b74e69e-14e3-41ab-a94c-44487eafb173-kube-api-access-jnl7j\") pod \"cluster-monitoring-operator-847b8bd8d7-gsbw2\" (UID: \"0b74e69e-14e3-41ab-a94c-44487eafb173\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-gsbw2"
Sep 17 15:55:22.950498 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:22.950464    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-f65c965db-7f4nh"]
Sep 17 15:55:22.952151 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:22.952135    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f65c965db-7f4nh"
Sep 17 15:55:22.960038 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:22.960018    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 15:55:22.960155 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:22.960042    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 15:55:22.960735 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:22.960719    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 15:55:22.960844 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:22.960769    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-qw6lt\""
Sep 17 15:55:22.971581 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:22.971562    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 15:55:22.976683 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:22.976551    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-hzd4v"]
Sep 17 15:55:22.978399 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:22.978381    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-cmn7w"]
Sep 17 15:55:22.978541 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:22.978524    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-hzd4v"
Sep 17 15:55:22.979886 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:22.979870    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-f65c965db-7f4nh"]
Sep 17 15:55:22.979993 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:22.979950    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-cmn7w"
Sep 17 15:55:22.987083 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:22.987065    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-sg5z7\""
Sep 17 15:55:22.989330 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:22.989313    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 17 15:55:22.989900 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:22.989879    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 17 15:55:22.990000 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:22.989899    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 17 15:55:22.990000 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:22.989910    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-xfg8b\""
Sep 17 15:55:22.990000 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:22.989961    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 17 15:55:22.990000 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:22.989907    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 17 15:55:23.011099 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.011064    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-cmn7w"]
Sep 17 15:55:23.020465 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.020436    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e0a58c2f-ea54-489b-86a9-d9c1d887b713-bound-sa-token\") pod \"image-registry-f65c965db-7f4nh\" (UID: \"e0a58c2f-ea54-489b-86a9-d9c1d887b713\") " pod="openshift-image-registry/image-registry-f65c965db-7f4nh"
Sep 17 15:55:23.020611 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.020506    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk58b\" (UniqueName: \"kubernetes.io/projected/8d059980-8f58-483c-973a-1181ef86e82b-kube-api-access-fk58b\") pod \"cluster-samples-operator-685d7d68b5-cmn7w\" (UID: \"8d059980-8f58-483c-973a-1181ef86e82b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-cmn7w"
Sep 17 15:55:23.020611 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.020548    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0b74e69e-14e3-41ab-a94c-44487eafb173-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-gsbw2\" (UID: \"0b74e69e-14e3-41ab-a94c-44487eafb173\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-gsbw2"
Sep 17 15:55:23.020611 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.020576    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnl7j\" (UniqueName: \"kubernetes.io/projected/0b74e69e-14e3-41ab-a94c-44487eafb173-kube-api-access-jnl7j\") pod \"cluster-monitoring-operator-847b8bd8d7-gsbw2\" (UID: \"0b74e69e-14e3-41ab-a94c-44487eafb173\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-gsbw2"
Sep 17 15:55:23.020611 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.020602    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e0a58c2f-ea54-489b-86a9-d9c1d887b713-registry-tls\") pod \"image-registry-f65c965db-7f4nh\" (UID: \"e0a58c2f-ea54-489b-86a9-d9c1d887b713\") " pod="openshift-image-registry/image-registry-f65c965db-7f4nh"
Sep 17 15:55:23.020831 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.020652    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e0a58c2f-ea54-489b-86a9-d9c1d887b713-trusted-ca\") pod \"image-registry-f65c965db-7f4nh\" (UID: \"e0a58c2f-ea54-489b-86a9-d9c1d887b713\") " pod="openshift-image-registry/image-registry-f65c965db-7f4nh"
Sep 17 15:55:23.020831 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.020704    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e0a58c2f-ea54-489b-86a9-d9c1d887b713-image-registry-private-configuration\") pod \"image-registry-f65c965db-7f4nh\" (UID: \"e0a58c2f-ea54-489b-86a9-d9c1d887b713\") " pod="openshift-image-registry/image-registry-f65c965db-7f4nh"
Sep 17 15:55:23.020831 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.020731    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e0a58c2f-ea54-489b-86a9-d9c1d887b713-installation-pull-secrets\") pod \"image-registry-f65c965db-7f4nh\" (UID: \"e0a58c2f-ea54-489b-86a9-d9c1d887b713\") " pod="openshift-image-registry/image-registry-f65c965db-7f4nh"
Sep 17 15:55:23.020831 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.020759    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28jtt\" (UniqueName: \"kubernetes.io/projected/e0a58c2f-ea54-489b-86a9-d9c1d887b713-kube-api-access-28jtt\") pod \"image-registry-f65c965db-7f4nh\" (UID: \"e0a58c2f-ea54-489b-86a9-d9c1d887b713\") " pod="openshift-image-registry/image-registry-f65c965db-7f4nh"
Sep 17 15:55:23.020831 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.020795    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e0a58c2f-ea54-489b-86a9-d9c1d887b713-registry-certificates\") pod \"image-registry-f65c965db-7f4nh\" (UID: \"e0a58c2f-ea54-489b-86a9-d9c1d887b713\") " pod="openshift-image-registry/image-registry-f65c965db-7f4nh"
Sep 17 15:55:23.020831 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.020823    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d059980-8f58-483c-973a-1181ef86e82b-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-cmn7w\" (UID: \"8d059980-8f58-483c-973a-1181ef86e82b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-cmn7w"
Sep 17 15:55:23.021118 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.020862    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0b74e69e-14e3-41ab-a94c-44487eafb173-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-gsbw2\" (UID: \"0b74e69e-14e3-41ab-a94c-44487eafb173\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-gsbw2"
Sep 17 15:55:23.021118 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:23.020953    2585 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 15:55:23.021118 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:23.021020    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b74e69e-14e3-41ab-a94c-44487eafb173-cluster-monitoring-operator-tls podName:0b74e69e-14e3-41ab-a94c-44487eafb173 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:23.521000253 +0000 UTC m=+79.291332045 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0b74e69e-14e3-41ab-a94c-44487eafb173-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-gsbw2" (UID: "0b74e69e-14e3-41ab-a94c-44487eafb173") : secret "cluster-monitoring-operator-tls" not found
Sep 17 15:55:23.021118 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.021039    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4shfx\" (UniqueName: \"kubernetes.io/projected/81d65819-6f7d-4d1e-a851-ef3ae13b5535-kube-api-access-4shfx\") pod \"volume-data-source-validator-5c4db8bdf9-hzd4v\" (UID: \"81d65819-6f7d-4d1e-a851-ef3ae13b5535\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-hzd4v"
Sep 17 15:55:23.021118 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.021067    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e0a58c2f-ea54-489b-86a9-d9c1d887b713-ca-trust-extracted\") pod \"image-registry-f65c965db-7f4nh\" (UID: \"e0a58c2f-ea54-489b-86a9-d9c1d887b713\") " pod="openshift-image-registry/image-registry-f65c965db-7f4nh"
Sep 17 15:55:23.021462 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.021440    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0b74e69e-14e3-41ab-a94c-44487eafb173-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-gsbw2\" (UID: \"0b74e69e-14e3-41ab-a94c-44487eafb173\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-gsbw2"
Sep 17 15:55:23.021628 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.021611    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-hzd4v"]
Sep 17 15:55:23.037898 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.037599    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnl7j\" (UniqueName: \"kubernetes.io/projected/0b74e69e-14e3-41ab-a94c-44487eafb173-kube-api-access-jnl7j\") pod \"cluster-monitoring-operator-847b8bd8d7-gsbw2\" (UID: \"0b74e69e-14e3-41ab-a94c-44487eafb173\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-gsbw2"
Sep 17 15:55:23.122063 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.121974    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e0a58c2f-ea54-489b-86a9-d9c1d887b713-registry-tls\") pod \"image-registry-f65c965db-7f4nh\" (UID: \"e0a58c2f-ea54-489b-86a9-d9c1d887b713\") " pod="openshift-image-registry/image-registry-f65c965db-7f4nh"
Sep 17 15:55:23.122063 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.122011    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e0a58c2f-ea54-489b-86a9-d9c1d887b713-trusted-ca\") pod \"image-registry-f65c965db-7f4nh\" (UID: \"e0a58c2f-ea54-489b-86a9-d9c1d887b713\") " pod="openshift-image-registry/image-registry-f65c965db-7f4nh"
Sep 17 15:55:23.122063 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.122037    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e0a58c2f-ea54-489b-86a9-d9c1d887b713-image-registry-private-configuration\") pod \"image-registry-f65c965db-7f4nh\" (UID: \"e0a58c2f-ea54-489b-86a9-d9c1d887b713\") " pod="openshift-image-registry/image-registry-f65c965db-7f4nh"
Sep 17 15:55:23.122063 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.122063    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e0a58c2f-ea54-489b-86a9-d9c1d887b713-installation-pull-secrets\") pod \"image-registry-f65c965db-7f4nh\" (UID: \"e0a58c2f-ea54-489b-86a9-d9c1d887b713\") " pod="openshift-image-registry/image-registry-f65c965db-7f4nh"
Sep 17 15:55:23.122396 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:23.122120    2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 15:55:23.122396 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:23.122143    2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f65c965db-7f4nh: secret "image-registry-tls" not found
Sep 17 15:55:23.122396 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.122198    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28jtt\" (UniqueName: \"kubernetes.io/projected/e0a58c2f-ea54-489b-86a9-d9c1d887b713-kube-api-access-28jtt\") pod \"image-registry-f65c965db-7f4nh\" (UID: \"e0a58c2f-ea54-489b-86a9-d9c1d887b713\") " pod="openshift-image-registry/image-registry-f65c965db-7f4nh"
Sep 17 15:55:23.122396 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:23.122222    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0a58c2f-ea54-489b-86a9-d9c1d887b713-registry-tls podName:e0a58c2f-ea54-489b-86a9-d9c1d887b713 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:23.622199148 +0000 UTC m=+79.392530924 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e0a58c2f-ea54-489b-86a9-d9c1d887b713-registry-tls") pod "image-registry-f65c965db-7f4nh" (UID: "e0a58c2f-ea54-489b-86a9-d9c1d887b713") : secret "image-registry-tls" not found
Sep 17 15:55:23.122396 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.122267    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e0a58c2f-ea54-489b-86a9-d9c1d887b713-registry-certificates\") pod \"image-registry-f65c965db-7f4nh\" (UID: \"e0a58c2f-ea54-489b-86a9-d9c1d887b713\") " pod="openshift-image-registry/image-registry-f65c965db-7f4nh"
Sep 17 15:55:23.122396 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.122297    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d059980-8f58-483c-973a-1181ef86e82b-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-cmn7w\" (UID: \"8d059980-8f58-483c-973a-1181ef86e82b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-cmn7w"
Sep 17 15:55:23.122396 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.122344    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4shfx\" (UniqueName: \"kubernetes.io/projected/81d65819-6f7d-4d1e-a851-ef3ae13b5535-kube-api-access-4shfx\") pod \"volume-data-source-validator-5c4db8bdf9-hzd4v\" (UID: \"81d65819-6f7d-4d1e-a851-ef3ae13b5535\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-hzd4v"
Sep 17 15:55:23.122396 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.122367    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e0a58c2f-ea54-489b-86a9-d9c1d887b713-ca-trust-extracted\") pod \"image-registry-f65c965db-7f4nh\" (UID: \"e0a58c2f-ea54-489b-86a9-d9c1d887b713\") " pod="openshift-image-registry/image-registry-f65c965db-7f4nh"
Sep 17 15:55:23.122396 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.122399    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e0a58c2f-ea54-489b-86a9-d9c1d887b713-bound-sa-token\") pod \"image-registry-f65c965db-7f4nh\" (UID: \"e0a58c2f-ea54-489b-86a9-d9c1d887b713\") " pod="openshift-image-registry/image-registry-f65c965db-7f4nh"
Sep 17 15:55:23.122856 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:23.122414    2585 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 15:55:23.122856 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.122471    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk58b\" (UniqueName: \"kubernetes.io/projected/8d059980-8f58-483c-973a-1181ef86e82b-kube-api-access-fk58b\") pod \"cluster-samples-operator-685d7d68b5-cmn7w\" (UID: \"8d059980-8f58-483c-973a-1181ef86e82b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-cmn7w"
Sep 17 15:55:23.122856 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:23.122504    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d059980-8f58-483c-973a-1181ef86e82b-samples-operator-tls podName:8d059980-8f58-483c-973a-1181ef86e82b nodeName:}" failed. No retries permitted until 2026-09-17 15:55:23.622485754 +0000 UTC m=+79.392817544 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8d059980-8f58-483c-973a-1181ef86e82b-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-cmn7w" (UID: "8d059980-8f58-483c-973a-1181ef86e82b") : secret "samples-operator-tls" not found
Sep 17 15:55:23.122856 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.122817    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e0a58c2f-ea54-489b-86a9-d9c1d887b713-ca-trust-extracted\") pod \"image-registry-f65c965db-7f4nh\" (UID: \"e0a58c2f-ea54-489b-86a9-d9c1d887b713\") " pod="openshift-image-registry/image-registry-f65c965db-7f4nh"
Sep 17 15:55:23.123162 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.123144    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e0a58c2f-ea54-489b-86a9-d9c1d887b713-registry-certificates\") pod \"image-registry-f65c965db-7f4nh\" (UID: \"e0a58c2f-ea54-489b-86a9-d9c1d887b713\") " pod="openshift-image-registry/image-registry-f65c965db-7f4nh"
Sep 17 15:55:23.123197 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.123141    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e0a58c2f-ea54-489b-86a9-d9c1d887b713-trusted-ca\") pod \"image-registry-f65c965db-7f4nh\" (UID: \"e0a58c2f-ea54-489b-86a9-d9c1d887b713\") " pod="openshift-image-registry/image-registry-f65c965db-7f4nh"
Sep 17 15:55:23.124653 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.124631    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e0a58c2f-ea54-489b-86a9-d9c1d887b713-image-registry-private-configuration\") pod \"image-registry-f65c965db-7f4nh\" (UID: \"e0a58c2f-ea54-489b-86a9-d9c1d887b713\") " pod="openshift-image-registry/image-registry-f65c965db-7f4nh"
Sep 17 15:55:23.124829 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.124808    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e0a58c2f-ea54-489b-86a9-d9c1d887b713-installation-pull-secrets\") pod \"image-registry-f65c965db-7f4nh\" (UID: \"e0a58c2f-ea54-489b-86a9-d9c1d887b713\") " pod="openshift-image-registry/image-registry-f65c965db-7f4nh"
Sep 17 15:55:23.136485 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.136458    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4shfx\" (UniqueName: \"kubernetes.io/projected/81d65819-6f7d-4d1e-a851-ef3ae13b5535-kube-api-access-4shfx\") pod \"volume-data-source-validator-5c4db8bdf9-hzd4v\" (UID: \"81d65819-6f7d-4d1e-a851-ef3ae13b5535\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-hzd4v"
Sep 17 15:55:23.137254 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.137228    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk58b\" (UniqueName: \"kubernetes.io/projected/8d059980-8f58-483c-973a-1181ef86e82b-kube-api-access-fk58b\") pod \"cluster-samples-operator-685d7d68b5-cmn7w\" (UID: \"8d059980-8f58-483c-973a-1181ef86e82b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-cmn7w"
Sep 17 15:55:23.138004 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.137982    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e0a58c2f-ea54-489b-86a9-d9c1d887b713-bound-sa-token\") pod \"image-registry-f65c965db-7f4nh\" (UID: \"e0a58c2f-ea54-489b-86a9-d9c1d887b713\") " pod="openshift-image-registry/image-registry-f65c965db-7f4nh"
Sep 17 15:55:23.138659 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.138638    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28jtt\" (UniqueName: \"kubernetes.io/projected/e0a58c2f-ea54-489b-86a9-d9c1d887b713-kube-api-access-28jtt\") pod \"image-registry-f65c965db-7f4nh\" (UID: \"e0a58c2f-ea54-489b-86a9-d9c1d887b713\") " pod="openshift-image-registry/image-registry-f65c965db-7f4nh"
Sep 17 15:55:23.149827 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.149804    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-mb9mz"]
Sep 17 15:55:23.152136 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.152122    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-mb9mz"
Sep 17 15:55:23.155529 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.155512    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 17 15:55:23.155794 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.155776    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 17 15:55:23.155849 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.155830    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 17 15:55:23.155891 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.155861    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-65z5v\""
Sep 17 15:55:23.156185 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.156171    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 17 15:55:23.168683 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.168657    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-mb9mz"]
Sep 17 15:55:23.223625 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.223593    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwnk9\" (UniqueName: \"kubernetes.io/projected/84d8c186-5cbe-4c2e-9465-cc6729bf2b21-kube-api-access-qwnk9\") pod \"service-ca-operator-57cdc4b768-mb9mz\" (UID: \"84d8c186-5cbe-4c2e-9465-cc6729bf2b21\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-mb9mz"
Sep 17 15:55:23.223809 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.223664    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/84d8c186-5cbe-4c2e-9465-cc6729bf2b21-config\") pod \"service-ca-operator-57cdc4b768-mb9mz\" (UID: \"84d8c186-5cbe-4c2e-9465-cc6729bf2b21\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-mb9mz"
Sep 17 15:55:23.223809 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.223782    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/84d8c186-5cbe-4c2e-9465-cc6729bf2b21-serving-cert\") pod \"service-ca-operator-57cdc4b768-mb9mz\" (UID: \"84d8c186-5cbe-4c2e-9465-cc6729bf2b21\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-mb9mz"
Sep 17 15:55:23.287953 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.287916    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-hzd4v"
Sep 17 15:55:23.324534 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.324496    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/84d8c186-5cbe-4c2e-9465-cc6729bf2b21-config\") pod \"service-ca-operator-57cdc4b768-mb9mz\" (UID: \"84d8c186-5cbe-4c2e-9465-cc6729bf2b21\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-mb9mz"
Sep 17 15:55:23.324713 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.324693    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/84d8c186-5cbe-4c2e-9465-cc6729bf2b21-serving-cert\") pod \"service-ca-operator-57cdc4b768-mb9mz\" (UID: \"84d8c186-5cbe-4c2e-9465-cc6729bf2b21\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-mb9mz"
Sep 17 15:55:23.324775 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.324733    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwnk9\" (UniqueName: \"kubernetes.io/projected/84d8c186-5cbe-4c2e-9465-cc6729bf2b21-kube-api-access-qwnk9\") pod \"service-ca-operator-57cdc4b768-mb9mz\" (UID: \"84d8c186-5cbe-4c2e-9465-cc6729bf2b21\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-mb9mz"
Sep 17 15:55:23.325017 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.324996    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/84d8c186-5cbe-4c2e-9465-cc6729bf2b21-config\") pod \"service-ca-operator-57cdc4b768-mb9mz\" (UID: \"84d8c186-5cbe-4c2e-9465-cc6729bf2b21\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-mb9mz"
Sep 17 15:55:23.326899 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.326878    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/84d8c186-5cbe-4c2e-9465-cc6729bf2b21-serving-cert\") pod \"service-ca-operator-57cdc4b768-mb9mz\" (UID: \"84d8c186-5cbe-4c2e-9465-cc6729bf2b21\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-mb9mz"
Sep 17 15:55:23.337320 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.337288    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwnk9\" (UniqueName: \"kubernetes.io/projected/84d8c186-5cbe-4c2e-9465-cc6729bf2b21-kube-api-access-qwnk9\") pod \"service-ca-operator-57cdc4b768-mb9mz\" (UID: \"84d8c186-5cbe-4c2e-9465-cc6729bf2b21\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-mb9mz"
Sep 17 15:55:23.460212 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.460169    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-mb9mz"
Sep 17 15:55:23.525890 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.525859    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0b74e69e-14e3-41ab-a94c-44487eafb173-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-gsbw2\" (UID: \"0b74e69e-14e3-41ab-a94c-44487eafb173\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-gsbw2"
Sep 17 15:55:23.526047 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:23.526025    2585 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 15:55:23.526104 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:23.526094    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b74e69e-14e3-41ab-a94c-44487eafb173-cluster-monitoring-operator-tls podName:0b74e69e-14e3-41ab-a94c-44487eafb173 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:24.5260792 +0000 UTC m=+80.296410981 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0b74e69e-14e3-41ab-a94c-44487eafb173-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-gsbw2" (UID: "0b74e69e-14e3-41ab-a94c-44487eafb173") : secret "cluster-monitoring-operator-tls" not found
Sep 17 15:55:23.562383 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.562352    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-hzd4v"]
Sep 17 15:55:23.565664 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:55:23.565635    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81d65819_6f7d_4d1e_a851_ef3ae13b5535.slice/crio-e79972d7433585ea7e965a2cb668c9ff8914f1e0a879e255fe1ae4c3fa737076 WatchSource:0}: Error finding container e79972d7433585ea7e965a2cb668c9ff8914f1e0a879e255fe1ae4c3fa737076: Status 404 returned error can't find the container with id e79972d7433585ea7e965a2cb668c9ff8914f1e0a879e255fe1ae4c3fa737076
Sep 17 15:55:23.626983 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.626947    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e0a58c2f-ea54-489b-86a9-d9c1d887b713-registry-tls\") pod \"image-registry-f65c965db-7f4nh\" (UID: \"e0a58c2f-ea54-489b-86a9-d9c1d887b713\") " pod="openshift-image-registry/image-registry-f65c965db-7f4nh"
Sep 17 15:55:23.627147 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.626994    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d059980-8f58-483c-973a-1181ef86e82b-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-cmn7w\" (UID: \"8d059980-8f58-483c-973a-1181ef86e82b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-cmn7w"
Sep 17 15:55:23.627147 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:23.627109    2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 15:55:23.627147 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:23.627121    2585 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 15:55:23.627147 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:23.627127    2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f65c965db-7f4nh: secret "image-registry-tls" not found
Sep 17 15:55:23.627275 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:23.627181    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d059980-8f58-483c-973a-1181ef86e82b-samples-operator-tls podName:8d059980-8f58-483c-973a-1181ef86e82b nodeName:}" failed. No retries permitted until 2026-09-17 15:55:24.627164015 +0000 UTC m=+80.397495794 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8d059980-8f58-483c-973a-1181ef86e82b-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-cmn7w" (UID: "8d059980-8f58-483c-973a-1181ef86e82b") : secret "samples-operator-tls" not found
Sep 17 15:55:23.627275 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:23.627200    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0a58c2f-ea54-489b-86a9-d9c1d887b713-registry-tls podName:e0a58c2f-ea54-489b-86a9-d9c1d887b713 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:24.627192212 +0000 UTC m=+80.397523987 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e0a58c2f-ea54-489b-86a9-d9c1d887b713-registry-tls") pod "image-registry-f65c965db-7f4nh" (UID: "e0a58c2f-ea54-489b-86a9-d9c1d887b713") : secret "image-registry-tls" not found
Sep 17 15:55:23.732182 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.732109    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-mb9mz"]
Sep 17 15:55:23.735120 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:55:23.735094    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84d8c186_5cbe_4c2e_9465_cc6729bf2b21.slice/crio-2c1cafdc165e5e5fb2ab1dae8cb57b483ac16e80df5e3c55f49e92dba547422a WatchSource:0}: Error finding container 2c1cafdc165e5e5fb2ab1dae8cb57b483ac16e80df5e3c55f49e92dba547422a: Status 404 returned error can't find the container with id 2c1cafdc165e5e5fb2ab1dae8cb57b483ac16e80df5e3c55f49e92dba547422a
Sep 17 15:55:23.914351 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.914318    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-mb9mz" event={"ID":"84d8c186-5cbe-4c2e-9465-cc6729bf2b21","Type":"ContainerStarted","Data":"2c1cafdc165e5e5fb2ab1dae8cb57b483ac16e80df5e3c55f49e92dba547422a"}
Sep 17 15:55:23.915394 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:23.915373    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-hzd4v" event={"ID":"81d65819-6f7d-4d1e-a851-ef3ae13b5535","Type":"ContainerStarted","Data":"e79972d7433585ea7e965a2cb668c9ff8914f1e0a879e255fe1ae4c3fa737076"}
Sep 17 15:55:24.535568 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:24.535484    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0b74e69e-14e3-41ab-a94c-44487eafb173-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-gsbw2\" (UID: \"0b74e69e-14e3-41ab-a94c-44487eafb173\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-gsbw2"
Sep 17 15:55:24.535772 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:24.535670    2585 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 15:55:24.535772 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:24.535764    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b74e69e-14e3-41ab-a94c-44487eafb173-cluster-monitoring-operator-tls podName:0b74e69e-14e3-41ab-a94c-44487eafb173 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:26.535743599 +0000 UTC m=+82.306075381 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0b74e69e-14e3-41ab-a94c-44487eafb173-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-gsbw2" (UID: "0b74e69e-14e3-41ab-a94c-44487eafb173") : secret "cluster-monitoring-operator-tls" not found
Sep 17 15:55:24.636181 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:24.636143    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d059980-8f58-483c-973a-1181ef86e82b-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-cmn7w\" (UID: \"8d059980-8f58-483c-973a-1181ef86e82b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-cmn7w"
Sep 17 15:55:24.636327 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:24.636245    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e0a58c2f-ea54-489b-86a9-d9c1d887b713-registry-tls\") pod \"image-registry-f65c965db-7f4nh\" (UID: \"e0a58c2f-ea54-489b-86a9-d9c1d887b713\") " pod="openshift-image-registry/image-registry-f65c965db-7f4nh"
Sep 17 15:55:24.636327 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:24.636303    2585 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 15:55:24.636399 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:24.636336    2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 15:55:24.636399 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:24.636348    2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f65c965db-7f4nh: secret "image-registry-tls" not found
Sep 17 15:55:24.636399 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:24.636389    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d059980-8f58-483c-973a-1181ef86e82b-samples-operator-tls podName:8d059980-8f58-483c-973a-1181ef86e82b nodeName:}" failed. No retries permitted until 2026-09-17 15:55:26.636363627 +0000 UTC m=+82.406695418 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8d059980-8f58-483c-973a-1181ef86e82b-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-cmn7w" (UID: "8d059980-8f58-483c-973a-1181ef86e82b") : secret "samples-operator-tls" not found
Sep 17 15:55:24.636565 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:24.636407    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0a58c2f-ea54-489b-86a9-d9c1d887b713-registry-tls podName:e0a58c2f-ea54-489b-86a9-d9c1d887b713 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:26.636398638 +0000 UTC m=+82.406730415 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e0a58c2f-ea54-489b-86a9-d9c1d887b713-registry-tls") pod "image-registry-f65c965db-7f4nh" (UID: "e0a58c2f-ea54-489b-86a9-d9c1d887b713") : secret "image-registry-tls" not found
Sep 17 15:55:24.920165 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:24.920073    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-hzd4v" event={"ID":"81d65819-6f7d-4d1e-a851-ef3ae13b5535","Type":"ContainerStarted","Data":"f24c8939b9aedf0773dc73dc88ff2aacc984be64538ee65305f6fb69d5b044f5"}
Sep 17 15:55:24.940859 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:24.940653    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-hzd4v" podStartSLOduration=1.858117603 podStartE2EDuration="2.940638311s" totalImagesPullingTime="1.082520708s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:55:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:55:24.940286827 +0000 UTC m=+80.710618626" watchObservedRunningTime="2026-09-17 15:55:24.940638311 +0000 UTC m=+80.710970108"
Sep 17 15:55:25.928357 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:25.928279    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-mb9mz" event={"ID":"84d8c186-5cbe-4c2e-9465-cc6729bf2b21","Type":"ContainerStarted","Data":"07130c1280f1861b4cd533a3778b7c1cce0296e0c791a58b793318490fca1a4b"}
Sep 17 15:55:25.958358 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:25.958307    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-mb9mz" podStartSLOduration=1.115368291 podStartE2EDuration="2.958291094s" totalImagesPullingTime="1.842922803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:55:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:55:25.956773603 +0000 UTC m=+81.727105403" watchObservedRunningTime="2026-09-17 15:55:25.958291094 +0000 UTC m=+81.728622891"
Sep 17 15:55:26.553824 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:26.553774    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0b74e69e-14e3-41ab-a94c-44487eafb173-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-gsbw2\" (UID: \"0b74e69e-14e3-41ab-a94c-44487eafb173\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-gsbw2"
Sep 17 15:55:26.554036 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:26.553896    2585 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 15:55:26.554036 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:26.553953    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b74e69e-14e3-41ab-a94c-44487eafb173-cluster-monitoring-operator-tls podName:0b74e69e-14e3-41ab-a94c-44487eafb173 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:30.553935896 +0000 UTC m=+86.324267689 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0b74e69e-14e3-41ab-a94c-44487eafb173-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-gsbw2" (UID: "0b74e69e-14e3-41ab-a94c-44487eafb173") : secret "cluster-monitoring-operator-tls" not found
Sep 17 15:55:26.654974 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:26.654925    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e0a58c2f-ea54-489b-86a9-d9c1d887b713-registry-tls\") pod \"image-registry-f65c965db-7f4nh\" (UID: \"e0a58c2f-ea54-489b-86a9-d9c1d887b713\") " pod="openshift-image-registry/image-registry-f65c965db-7f4nh"
Sep 17 15:55:26.655137 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:26.655009    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d059980-8f58-483c-973a-1181ef86e82b-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-cmn7w\" (UID: \"8d059980-8f58-483c-973a-1181ef86e82b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-cmn7w"
Sep 17 15:55:26.655137 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:26.655076    2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 15:55:26.655137 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:26.655096    2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f65c965db-7f4nh: secret "image-registry-tls" not found
Sep 17 15:55:26.655243 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:26.655146    2585 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 15:55:26.655243 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:26.655152    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0a58c2f-ea54-489b-86a9-d9c1d887b713-registry-tls podName:e0a58c2f-ea54-489b-86a9-d9c1d887b713 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:30.655137688 +0000 UTC m=+86.425469464 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e0a58c2f-ea54-489b-86a9-d9c1d887b713-registry-tls") pod "image-registry-f65c965db-7f4nh" (UID: "e0a58c2f-ea54-489b-86a9-d9c1d887b713") : secret "image-registry-tls" not found
Sep 17 15:55:26.655243 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:26.655219    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d059980-8f58-483c-973a-1181ef86e82b-samples-operator-tls podName:8d059980-8f58-483c-973a-1181ef86e82b nodeName:}" failed. No retries permitted until 2026-09-17 15:55:30.655203005 +0000 UTC m=+86.425534786 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8d059980-8f58-483c-973a-1181ef86e82b-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-cmn7w" (UID: "8d059980-8f58-483c-973a-1181ef86e82b") : secret "samples-operator-tls" not found
Sep 17 15:55:26.897933 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:26.897837    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-n84xc"]
Sep 17 15:55:26.899869 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:26.899852    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-n84xc"
Sep 17 15:55:26.908596 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:26.908568    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 15:55:26.908596 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:26.908584    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 15:55:26.908764 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:26.908572    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 15:55:26.909871 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:26.909853    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 15:55:26.913700 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:26.913679    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-d9ppb\""
Sep 17 15:55:26.914999 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:26.914978    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-n84xc"]
Sep 17 15:55:26.957604 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:26.957564    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/53c4c887-7ae3-491b-bcec-246b29c06a84-crio-socket\") pod \"insights-runtime-extractor-n84xc\" (UID: \"53c4c887-7ae3-491b-bcec-246b29c06a84\") " pod="openshift-insights/insights-runtime-extractor-n84xc"
Sep 17 15:55:26.958035 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:26.957615    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/53c4c887-7ae3-491b-bcec-246b29c06a84-data-volume\") pod \"insights-runtime-extractor-n84xc\" (UID: \"53c4c887-7ae3-491b-bcec-246b29c06a84\") " pod="openshift-insights/insights-runtime-extractor-n84xc"
Sep 17 15:55:26.958035 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:26.957684    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66xjk\" (UniqueName: \"kubernetes.io/projected/53c4c887-7ae3-491b-bcec-246b29c06a84-kube-api-access-66xjk\") pod \"insights-runtime-extractor-n84xc\" (UID: \"53c4c887-7ae3-491b-bcec-246b29c06a84\") " pod="openshift-insights/insights-runtime-extractor-n84xc"
Sep 17 15:55:26.958035 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:26.957746    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53c4c887-7ae3-491b-bcec-246b29c06a84-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n84xc\" (UID: \"53c4c887-7ae3-491b-bcec-246b29c06a84\") " pod="openshift-insights/insights-runtime-extractor-n84xc"
Sep 17 15:55:26.958035 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:26.957837    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53c4c887-7ae3-491b-bcec-246b29c06a84-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n84xc\" (UID: \"53c4c887-7ae3-491b-bcec-246b29c06a84\") " pod="openshift-insights/insights-runtime-extractor-n84xc"
Sep 17 15:55:27.058103 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:27.058064    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66xjk\" (UniqueName: \"kubernetes.io/projected/53c4c887-7ae3-491b-bcec-246b29c06a84-kube-api-access-66xjk\") pod \"insights-runtime-extractor-n84xc\" (UID: \"53c4c887-7ae3-491b-bcec-246b29c06a84\") " pod="openshift-insights/insights-runtime-extractor-n84xc"
Sep 17 15:55:27.058304 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:27.058119    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53c4c887-7ae3-491b-bcec-246b29c06a84-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n84xc\" (UID: \"53c4c887-7ae3-491b-bcec-246b29c06a84\") " pod="openshift-insights/insights-runtime-extractor-n84xc"
Sep 17 15:55:27.058304 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:27.058174    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53c4c887-7ae3-491b-bcec-246b29c06a84-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n84xc\" (UID: \"53c4c887-7ae3-491b-bcec-246b29c06a84\") " pod="openshift-insights/insights-runtime-extractor-n84xc"
Sep 17 15:55:27.058304 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:27.058210    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/53c4c887-7ae3-491b-bcec-246b29c06a84-crio-socket\") pod \"insights-runtime-extractor-n84xc\" (UID: \"53c4c887-7ae3-491b-bcec-246b29c06a84\") " pod="openshift-insights/insights-runtime-extractor-n84xc"
Sep 17 15:55:27.058304 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:27.058256    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/53c4c887-7ae3-491b-bcec-246b29c06a84-data-volume\") pod \"insights-runtime-extractor-n84xc\" (UID: \"53c4c887-7ae3-491b-bcec-246b29c06a84\") " pod="openshift-insights/insights-runtime-extractor-n84xc"
Sep 17 15:55:27.058543 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:27.058315    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:55:27.058543 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:27.058351    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/53c4c887-7ae3-491b-bcec-246b29c06a84-crio-socket\") pod \"insights-runtime-extractor-n84xc\" (UID: \"53c4c887-7ae3-491b-bcec-246b29c06a84\") " pod="openshift-insights/insights-runtime-extractor-n84xc"
Sep 17 15:55:27.058543 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:27.058378    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53c4c887-7ae3-491b-bcec-246b29c06a84-insights-runtime-extractor-tls podName:53c4c887-7ae3-491b-bcec-246b29c06a84 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:27.558361137 +0000 UTC m=+83.328692934 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/53c4c887-7ae3-491b-bcec-246b29c06a84-insights-runtime-extractor-tls") pod "insights-runtime-extractor-n84xc" (UID: "53c4c887-7ae3-491b-bcec-246b29c06a84") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:55:27.058676 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:27.058550    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 15:55:27.058676 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:27.058616    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/53c4c887-7ae3-491b-bcec-246b29c06a84-kube-rbac-proxy-cm podName:53c4c887-7ae3-491b-bcec-246b29c06a84 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:27.558597263 +0000 UTC m=+83.328929052 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/53c4c887-7ae3-491b-bcec-246b29c06a84-kube-rbac-proxy-cm") pod "insights-runtime-extractor-n84xc" (UID: "53c4c887-7ae3-491b-bcec-246b29c06a84") : configmap "kube-rbac-proxy" not found
Sep 17 15:55:27.058676 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:27.058617    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/53c4c887-7ae3-491b-bcec-246b29c06a84-data-volume\") pod \"insights-runtime-extractor-n84xc\" (UID: \"53c4c887-7ae3-491b-bcec-246b29c06a84\") " pod="openshift-insights/insights-runtime-extractor-n84xc"
Sep 17 15:55:27.070477 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:27.070448    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66xjk\" (UniqueName: \"kubernetes.io/projected/53c4c887-7ae3-491b-bcec-246b29c06a84-kube-api-access-66xjk\") pod \"insights-runtime-extractor-n84xc\" (UID: \"53c4c887-7ae3-491b-bcec-246b29c06a84\") " pod="openshift-insights/insights-runtime-extractor-n84xc"
Sep 17 15:55:27.562834 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:27.562787    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53c4c887-7ae3-491b-bcec-246b29c06a84-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n84xc\" (UID: \"53c4c887-7ae3-491b-bcec-246b29c06a84\") " pod="openshift-insights/insights-runtime-extractor-n84xc"
Sep 17 15:55:27.563145 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:27.562869    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53c4c887-7ae3-491b-bcec-246b29c06a84-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n84xc\" (UID: \"53c4c887-7ae3-491b-bcec-246b29c06a84\") " pod="openshift-insights/insights-runtime-extractor-n84xc"
Sep 17 15:55:27.563145 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:27.562922    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 15:55:27.563145 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:27.563001    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:55:27.563145 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:27.563014    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/53c4c887-7ae3-491b-bcec-246b29c06a84-kube-rbac-proxy-cm podName:53c4c887-7ae3-491b-bcec-246b29c06a84 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:28.562993879 +0000 UTC m=+84.333325654 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/53c4c887-7ae3-491b-bcec-246b29c06a84-kube-rbac-proxy-cm") pod "insights-runtime-extractor-n84xc" (UID: "53c4c887-7ae3-491b-bcec-246b29c06a84") : configmap "kube-rbac-proxy" not found
Sep 17 15:55:27.563145 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:27.563047    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53c4c887-7ae3-491b-bcec-246b29c06a84-insights-runtime-extractor-tls podName:53c4c887-7ae3-491b-bcec-246b29c06a84 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:28.563035723 +0000 UTC m=+84.333367499 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/53c4c887-7ae3-491b-bcec-246b29c06a84-insights-runtime-extractor-tls") pod "insights-runtime-extractor-n84xc" (UID: "53c4c887-7ae3-491b-bcec-246b29c06a84") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:55:28.570934 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:28.570895    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53c4c887-7ae3-491b-bcec-246b29c06a84-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n84xc\" (UID: \"53c4c887-7ae3-491b-bcec-246b29c06a84\") " pod="openshift-insights/insights-runtime-extractor-n84xc"
Sep 17 15:55:28.571346 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:28.570957    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53c4c887-7ae3-491b-bcec-246b29c06a84-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n84xc\" (UID: \"53c4c887-7ae3-491b-bcec-246b29c06a84\") " pod="openshift-insights/insights-runtime-extractor-n84xc"
Sep 17 15:55:28.571346 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:28.570999    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 15:55:28.571346 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:28.571061    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/53c4c887-7ae3-491b-bcec-246b29c06a84-kube-rbac-proxy-cm podName:53c4c887-7ae3-491b-bcec-246b29c06a84 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:30.571045788 +0000 UTC m=+86.341377564 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/53c4c887-7ae3-491b-bcec-246b29c06a84-kube-rbac-proxy-cm") pod "insights-runtime-extractor-n84xc" (UID: "53c4c887-7ae3-491b-bcec-246b29c06a84") : configmap "kube-rbac-proxy" not found
Sep 17 15:55:28.571346 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:28.571065    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:55:28.571346 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:28.571111    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53c4c887-7ae3-491b-bcec-246b29c06a84-insights-runtime-extractor-tls podName:53c4c887-7ae3-491b-bcec-246b29c06a84 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:30.571100039 +0000 UTC m=+86.341431815 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/53c4c887-7ae3-491b-bcec-246b29c06a84-insights-runtime-extractor-tls") pod "insights-runtime-extractor-n84xc" (UID: "53c4c887-7ae3-491b-bcec-246b29c06a84") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:55:29.000873 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:29.000848    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xhtm4_c4438a49-8ea1-40aa-bdd9-2707192617af/dns-node-resolver/0.log"
Sep 17 15:55:29.803144 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:29.803115    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dtbtf_28231271-38ce-4985-9020-e3494fd366c7/node-ca/0.log"
Sep 17 15:55:30.404734 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:30.404699    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6qpcq_1f3d6b60-625e-4015-942c-6c8150fa364f/serve-healthcheck-canary/0.log"
Sep 17 15:55:30.586255 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:30.586211    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53c4c887-7ae3-491b-bcec-246b29c06a84-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n84xc\" (UID: \"53c4c887-7ae3-491b-bcec-246b29c06a84\") " pod="openshift-insights/insights-runtime-extractor-n84xc"
Sep 17 15:55:30.586483 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:30.586273    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0b74e69e-14e3-41ab-a94c-44487eafb173-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-gsbw2\" (UID: \"0b74e69e-14e3-41ab-a94c-44487eafb173\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-gsbw2"
Sep 17 15:55:30.586483 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:30.586309    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53c4c887-7ae3-491b-bcec-246b29c06a84-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n84xc\" (UID: \"53c4c887-7ae3-491b-bcec-246b29c06a84\") " pod="openshift-insights/insights-runtime-extractor-n84xc"
Sep 17 15:55:30.586483 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:30.586344    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 15:55:30.586483 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:30.586403    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/53c4c887-7ae3-491b-bcec-246b29c06a84-kube-rbac-proxy-cm podName:53c4c887-7ae3-491b-bcec-246b29c06a84 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:34.586387577 +0000 UTC m=+90.356719352 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/53c4c887-7ae3-491b-bcec-246b29c06a84-kube-rbac-proxy-cm") pod "insights-runtime-extractor-n84xc" (UID: "53c4c887-7ae3-491b-bcec-246b29c06a84") : configmap "kube-rbac-proxy" not found
Sep 17 15:55:30.586483 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:30.586452    2585 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 15:55:30.586483 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:30.586476    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:55:30.586737 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:30.586512    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b74e69e-14e3-41ab-a94c-44487eafb173-cluster-monitoring-operator-tls podName:0b74e69e-14e3-41ab-a94c-44487eafb173 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:38.586497355 +0000 UTC m=+94.356829151 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0b74e69e-14e3-41ab-a94c-44487eafb173-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-gsbw2" (UID: "0b74e69e-14e3-41ab-a94c-44487eafb173") : secret "cluster-monitoring-operator-tls" not found
Sep 17 15:55:30.586737 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:30.586532    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53c4c887-7ae3-491b-bcec-246b29c06a84-insights-runtime-extractor-tls podName:53c4c887-7ae3-491b-bcec-246b29c06a84 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:34.586522832 +0000 UTC m=+90.356854608 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/53c4c887-7ae3-491b-bcec-246b29c06a84-insights-runtime-extractor-tls") pod "insights-runtime-extractor-n84xc" (UID: "53c4c887-7ae3-491b-bcec-246b29c06a84") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:55:30.687490 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:30.687379    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e0a58c2f-ea54-489b-86a9-d9c1d887b713-registry-tls\") pod \"image-registry-f65c965db-7f4nh\" (UID: \"e0a58c2f-ea54-489b-86a9-d9c1d887b713\") " pod="openshift-image-registry/image-registry-f65c965db-7f4nh"
Sep 17 15:55:30.687490 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:30.687467    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d059980-8f58-483c-973a-1181ef86e82b-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-cmn7w\" (UID: \"8d059980-8f58-483c-973a-1181ef86e82b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-cmn7w"
Sep 17 15:55:30.687661 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:30.687528    2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 15:55:30.687661 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:30.687550    2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f65c965db-7f4nh: secret "image-registry-tls" not found
Sep 17 15:55:30.687661 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:30.687553    2585 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 17 15:55:30.687661 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:30.687604    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d059980-8f58-483c-973a-1181ef86e82b-samples-operator-tls podName:8d059980-8f58-483c-973a-1181ef86e82b nodeName:}" failed. No retries permitted until 2026-09-17 15:55:38.687589986 +0000 UTC m=+94.457921762 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8d059980-8f58-483c-973a-1181ef86e82b-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-cmn7w" (UID: "8d059980-8f58-483c-973a-1181ef86e82b") : secret "samples-operator-tls" not found
Sep 17 15:55:30.687661 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:30.687618    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e0a58c2f-ea54-489b-86a9-d9c1d887b713-registry-tls podName:e0a58c2f-ea54-489b-86a9-d9c1d887b713 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:38.687610876 +0000 UTC m=+94.457942651 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e0a58c2f-ea54-489b-86a9-d9c1d887b713-registry-tls") pod "image-registry-f65c965db-7f4nh" (UID: "e0a58c2f-ea54-489b-86a9-d9c1d887b713") : secret "image-registry-tls" not found
Sep 17 15:55:34.619446 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:34.619377    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53c4c887-7ae3-491b-bcec-246b29c06a84-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n84xc\" (UID: \"53c4c887-7ae3-491b-bcec-246b29c06a84\") " pod="openshift-insights/insights-runtime-extractor-n84xc"
Sep 17 15:55:34.619908 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:34.619491    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53c4c887-7ae3-491b-bcec-246b29c06a84-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n84xc\" (UID: \"53c4c887-7ae3-491b-bcec-246b29c06a84\") " pod="openshift-insights/insights-runtime-extractor-n84xc"
Sep 17 15:55:34.619908 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:34.619510    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 15:55:34.619908 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:34.619591    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/53c4c887-7ae3-491b-bcec-246b29c06a84-kube-rbac-proxy-cm podName:53c4c887-7ae3-491b-bcec-246b29c06a84 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:42.619572754 +0000 UTC m=+98.389904530 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/53c4c887-7ae3-491b-bcec-246b29c06a84-kube-rbac-proxy-cm") pod "insights-runtime-extractor-n84xc" (UID: "53c4c887-7ae3-491b-bcec-246b29c06a84") : configmap "kube-rbac-proxy" not found
Sep 17 15:55:34.619908 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:34.619643    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:55:34.619908 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:34.619701    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53c4c887-7ae3-491b-bcec-246b29c06a84-insights-runtime-extractor-tls podName:53c4c887-7ae3-491b-bcec-246b29c06a84 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:42.619689296 +0000 UTC m=+98.390021071 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/53c4c887-7ae3-491b-bcec-246b29c06a84-insights-runtime-extractor-tls") pod "insights-runtime-extractor-n84xc" (UID: "53c4c887-7ae3-491b-bcec-246b29c06a84") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:55:38.656627 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:38.656565    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0b74e69e-14e3-41ab-a94c-44487eafb173-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-gsbw2\" (UID: \"0b74e69e-14e3-41ab-a94c-44487eafb173\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-gsbw2"
Sep 17 15:55:38.657041 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:38.656720    2585 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 15:55:38.657041 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:38.656785    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b74e69e-14e3-41ab-a94c-44487eafb173-cluster-monitoring-operator-tls podName:0b74e69e-14e3-41ab-a94c-44487eafb173 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:54.656770134 +0000 UTC m=+110.427101926 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0b74e69e-14e3-41ab-a94c-44487eafb173-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-gsbw2" (UID: "0b74e69e-14e3-41ab-a94c-44487eafb173") : secret "cluster-monitoring-operator-tls" not found
Sep 17 15:55:38.757501 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:38.757461    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e0a58c2f-ea54-489b-86a9-d9c1d887b713-registry-tls\") pod \"image-registry-f65c965db-7f4nh\" (UID: \"e0a58c2f-ea54-489b-86a9-d9c1d887b713\") " pod="openshift-image-registry/image-registry-f65c965db-7f4nh"
Sep 17 15:55:38.757668 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:38.757523    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d059980-8f58-483c-973a-1181ef86e82b-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-cmn7w\" (UID: \"8d059980-8f58-483c-973a-1181ef86e82b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-cmn7w"
Sep 17 15:55:38.759918 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:38.759895    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d059980-8f58-483c-973a-1181ef86e82b-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-cmn7w\" (UID: \"8d059980-8f58-483c-973a-1181ef86e82b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-cmn7w"
Sep 17 15:55:38.759983 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:38.759920    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e0a58c2f-ea54-489b-86a9-d9c1d887b713-registry-tls\") pod \"image-registry-f65c965db-7f4nh\" (UID: \"e0a58c2f-ea54-489b-86a9-d9c1d887b713\") " pod="openshift-image-registry/image-registry-f65c965db-7f4nh"
Sep 17 15:55:38.859748 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:38.859690    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f65c965db-7f4nh"
Sep 17 15:55:38.893450 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:38.893406    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-cmn7w"
Sep 17 15:55:39.152972 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:39.152947    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-f65c965db-7f4nh"]
Sep 17 15:55:39.154498 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:55:39.154473    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0a58c2f_ea54_489b_86a9_d9c1d887b713.slice/crio-13ef5d356ff5a3c2c8091e5c9b524c087578929c49e28b79316301d9ffb98715 WatchSource:0}: Error finding container 13ef5d356ff5a3c2c8091e5c9b524c087578929c49e28b79316301d9ffb98715: Status 404 returned error can't find the container with id 13ef5d356ff5a3c2c8091e5c9b524c087578929c49e28b79316301d9ffb98715
Sep 17 15:55:39.172569 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:39.172494    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-cmn7w"]
Sep 17 15:55:39.973155 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:39.973113    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f65c965db-7f4nh" event={"ID":"e0a58c2f-ea54-489b-86a9-d9c1d887b713","Type":"ContainerStarted","Data":"362e5e89ae4e22726307028bd2eb19f33f8c5739265d4669a17c427a5f6d5a69"}
Sep 17 15:55:39.973660 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:39.973175    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-f65c965db-7f4nh"
Sep 17 15:55:39.973660 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:39.973195    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f65c965db-7f4nh" event={"ID":"e0a58c2f-ea54-489b-86a9-d9c1d887b713","Type":"ContainerStarted","Data":"13ef5d356ff5a3c2c8091e5c9b524c087578929c49e28b79316301d9ffb98715"}
Sep 17 15:55:39.974293 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:39.974270    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-cmn7w" event={"ID":"8d059980-8f58-483c-973a-1181ef86e82b","Type":"ContainerStarted","Data":"d9f7efdea71569b5b032dcbea2a9417371b48db6823c1fae3521b9913a7fac9f"}
Sep 17 15:55:40.000844 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:40.000785    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-f65c965db-7f4nh" podStartSLOduration=18.000768113 podStartE2EDuration="18.000768113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:55:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:55:39.999301795 +0000 UTC m=+95.769633595" watchObservedRunningTime="2026-09-17 15:55:40.000768113 +0000 UTC m=+95.771099924"
Sep 17 15:55:41.983210 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:41.983167    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-cmn7w" event={"ID":"8d059980-8f58-483c-973a-1181ef86e82b","Type":"ContainerStarted","Data":"da560f0cd0480bf3f35f04d635c310513ef8fea8012465ec7a80cc11a3393ef7"}
Sep 17 15:55:41.983606 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:41.983229    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-cmn7w" event={"ID":"8d059980-8f58-483c-973a-1181ef86e82b","Type":"ContainerStarted","Data":"30d9cfa5e1a2689dfe8be7315d5f5ca6de34f1de183aad6f1a32fb0b286f2969"}
Sep 17 15:55:42.003670 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:42.003622    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-cmn7w" podStartSLOduration=18.244946407 podStartE2EDuration="20.003606723s" totalImagesPullingTime="1.758660316s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:55:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:55:42.002936868 +0000 UTC m=+97.773268666" watchObservedRunningTime="2026-09-17 15:55:42.003606723 +0000 UTC m=+97.773938520"
Sep 17 15:55:42.692699 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:42.692649    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53c4c887-7ae3-491b-bcec-246b29c06a84-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n84xc\" (UID: \"53c4c887-7ae3-491b-bcec-246b29c06a84\") " pod="openshift-insights/insights-runtime-extractor-n84xc"
Sep 17 15:55:42.692885 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:42.692819    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53c4c887-7ae3-491b-bcec-246b29c06a84-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n84xc\" (UID: \"53c4c887-7ae3-491b-bcec-246b29c06a84\") " pod="openshift-insights/insights-runtime-extractor-n84xc"
Sep 17 15:55:42.692966 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:42.692948    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:55:42.693020 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:55:42.693011    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53c4c887-7ae3-491b-bcec-246b29c06a84-insights-runtime-extractor-tls podName:53c4c887-7ae3-491b-bcec-246b29c06a84 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:58.692997913 +0000 UTC m=+114.463329688 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/53c4c887-7ae3-491b-bcec-246b29c06a84-insights-runtime-extractor-tls") pod "insights-runtime-extractor-n84xc" (UID: "53c4c887-7ae3-491b-bcec-246b29c06a84") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:55:42.693180 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:42.693162    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53c4c887-7ae3-491b-bcec-246b29c06a84-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n84xc\" (UID: \"53c4c887-7ae3-491b-bcec-246b29c06a84\") " pod="openshift-insights/insights-runtime-extractor-n84xc"
Sep 17 15:55:44.204097 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:44.204071    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5cad5942-b49d-4bad-b6c4-d881654b6ef5-metrics-tls\") pod \"dns-default-cvmck\" (UID: \"5cad5942-b49d-4bad-b6c4-d881654b6ef5\") " pod="openshift-dns/dns-default-cvmck"
Sep 17 15:55:44.206876 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:44.206845    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5cad5942-b49d-4bad-b6c4-d881654b6ef5-metrics-tls\") pod \"dns-default-cvmck\" (UID: \"5cad5942-b49d-4bad-b6c4-d881654b6ef5\") " pod="openshift-dns/dns-default-cvmck"
Sep 17 15:55:44.473935 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:44.473862    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qcdnd\""
Sep 17 15:55:44.480337 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:44.480316    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cvmck"
Sep 17 15:55:44.770022 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:44.769927    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cvmck"]
Sep 17 15:55:44.773792 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:55:44.773759    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cad5942_b49d_4bad_b6c4_d881654b6ef5.slice/crio-7a1388a0de69ecdc136c49e73b381ef23a65244e2b24fd91476bc51591bde5bc WatchSource:0}: Error finding container 7a1388a0de69ecdc136c49e73b381ef23a65244e2b24fd91476bc51591bde5bc: Status 404 returned error can't find the container with id 7a1388a0de69ecdc136c49e73b381ef23a65244e2b24fd91476bc51591bde5bc
Sep 17 15:55:44.993921 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:44.993884    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cvmck" event={"ID":"5cad5942-b49d-4bad-b6c4-d881654b6ef5","Type":"ContainerStarted","Data":"7a1388a0de69ecdc136c49e73b381ef23a65244e2b24fd91476bc51591bde5bc"}
Sep 17 15:55:47.002004 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:47.001967    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cvmck" event={"ID":"5cad5942-b49d-4bad-b6c4-d881654b6ef5","Type":"ContainerStarted","Data":"b3a7b41386ca8efef4b341eb256c239e203b03372c3d17d894995d8ce6657f4c"}
Sep 17 15:55:47.002394 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:47.002033    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cvmck" event={"ID":"5cad5942-b49d-4bad-b6c4-d881654b6ef5","Type":"ContainerStarted","Data":"63866d053b9e3e7182c7847b69cd4335fba409e686db6ab6e95278fbda67664c"}
Sep 17 15:55:47.002394 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:47.002091    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-cvmck"
Sep 17 15:55:47.040749 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:47.040685    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-cvmck" podStartSLOduration=65.464688558 podStartE2EDuration="1m7.040665136s" totalImagesPullingTime="1.575976578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:54:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:55:47.039115305 +0000 UTC m=+102.809447115" watchObservedRunningTime="2026-09-17 15:55:47.040665136 +0000 UTC m=+102.810996937"
Sep 17 15:55:48.903939 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:48.903910    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9mgwn"
Sep 17 15:55:54.685237 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:54.685192    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0b74e69e-14e3-41ab-a94c-44487eafb173-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-gsbw2\" (UID: \"0b74e69e-14e3-41ab-a94c-44487eafb173\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-gsbw2"
Sep 17 15:55:54.687769 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:54.687738    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0b74e69e-14e3-41ab-a94c-44487eafb173-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-gsbw2\" (UID: \"0b74e69e-14e3-41ab-a94c-44487eafb173\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-gsbw2"
Sep 17 15:55:54.958649 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:54.958613    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-gsbw2"
Sep 17 15:55:55.233484 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:55.233380    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-gsbw2"]
Sep 17 15:55:55.236333 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:55:55.236303    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b74e69e_14e3_41ab_a94c_44487eafb173.slice/crio-2ea028fb80aef371de87800f26105a79657130c54e069d3139db1885c854e044 WatchSource:0}: Error finding container 2ea028fb80aef371de87800f26105a79657130c54e069d3139db1885c854e044: Status 404 returned error can't find the container with id 2ea028fb80aef371de87800f26105a79657130c54e069d3139db1885c854e044
Sep 17 15:55:56.033178 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:56.033136    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-gsbw2" event={"ID":"0b74e69e-14e3-41ab-a94c-44487eafb173","Type":"ContainerStarted","Data":"2ea028fb80aef371de87800f26105a79657130c54e069d3139db1885c854e044"}
Sep 17 15:55:57.008134 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:57.008098    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-cvmck"
Sep 17 15:55:57.042986 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:57.042947    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-gsbw2" event={"ID":"0b74e69e-14e3-41ab-a94c-44487eafb173","Type":"ContainerStarted","Data":"daa55e3c7ba37d6d11ff66e2698d55191811a17771aa29ccb9d13311081b42bf"}
Sep 17 15:55:57.062461 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:57.062392    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-gsbw2" podStartSLOduration=33.379021063 podStartE2EDuration="35.062369265s" totalImagesPullingTime="1.683348202s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:55:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:55:57.062236573 +0000 UTC m=+112.832568372" watchObservedRunningTime="2026-09-17 15:55:57.062369265 +0000 UTC m=+112.832701064"
Sep 17 15:55:58.715118 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:58.715058    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53c4c887-7ae3-491b-bcec-246b29c06a84-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n84xc\" (UID: \"53c4c887-7ae3-491b-bcec-246b29c06a84\") " pod="openshift-insights/insights-runtime-extractor-n84xc"
Sep 17 15:55:58.717464 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:58.717442    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53c4c887-7ae3-491b-bcec-246b29c06a84-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n84xc\" (UID: \"53c4c887-7ae3-491b-bcec-246b29c06a84\") " pod="openshift-insights/insights-runtime-extractor-n84xc"
Sep 17 15:55:58.864074 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:58.864022    2585 patch_prober.go:28] interesting pod/image-registry-f65c965db-7f4nh container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 17 15:55:58.864232 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:58.864085    2585 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-f65c965db-7f4nh" podUID="e0a58c2f-ea54-489b-86a9-d9c1d887b713" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:55:59.010646 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:59.010556    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-n84xc"
Sep 17 15:55:59.284184 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:55:59.284110    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-n84xc"]
Sep 17 15:55:59.287271 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:55:59.287246    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53c4c887_7ae3_491b_bcec_246b29c06a84.slice/crio-98e0984493d06ff7d9177e49ee722102bf67a42b2f28fa2598b66bbb900184e5 WatchSource:0}: Error finding container 98e0984493d06ff7d9177e49ee722102bf67a42b2f28fa2598b66bbb900184e5: Status 404 returned error can't find the container with id 98e0984493d06ff7d9177e49ee722102bf67a42b2f28fa2598b66bbb900184e5
Sep 17 15:56:00.056301 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:00.056261    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n84xc" event={"ID":"53c4c887-7ae3-491b-bcec-246b29c06a84","Type":"ContainerStarted","Data":"7ec52081addb4868e36fd33d0e19820ccd8a3070f700a71e2efba6d4d3fa4106"}
Sep 17 15:56:00.056754 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:00.056333    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n84xc" event={"ID":"53c4c887-7ae3-491b-bcec-246b29c06a84","Type":"ContainerStarted","Data":"9c519241dba15a8cfb78c0365ab3196236b25083d99aa7b9985ad11f22c2ddd9"}
Sep 17 15:56:00.056754 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:00.056360    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n84xc" event={"ID":"53c4c887-7ae3-491b-bcec-246b29c06a84","Type":"ContainerStarted","Data":"98e0984493d06ff7d9177e49ee722102bf67a42b2f28fa2598b66bbb900184e5"}
Sep 17 15:56:00.982891 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:00.982858    2585 patch_prober.go:28] interesting pod/image-registry-f65c965db-7f4nh container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 17 15:56:00.983073 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:00.982914    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-f65c965db-7f4nh" podUID="e0a58c2f-ea54-489b-86a9-d9c1d887b713" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:56:02.065181 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:02.065145    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n84xc" event={"ID":"53c4c887-7ae3-491b-bcec-246b29c06a84","Type":"ContainerStarted","Data":"853d806320ebcb33a16856faba0c5cd1c10c67b3c5c6eb47a7d92d5de7427456"}
Sep 17 15:56:02.088627 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:02.088571    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-n84xc" podStartSLOduration=34.15793853 podStartE2EDuration="36.088555088s" totalImagesPullingTime="1.930616558s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:55:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:56:02.087632515 +0000 UTC m=+117.857964317" watchObservedRunningTime="2026-09-17 15:56:02.088555088 +0000 UTC m=+117.858886878"
Sep 17 15:56:03.756405 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:03.756371    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-59669fb66b-sjgjs"]
Sep 17 15:56:03.759527 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:03.759507    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59669fb66b-sjgjs"
Sep 17 15:56:03.762791 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:03.762773    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 15:56:03.763965 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:03.763947    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 17 15:56:03.764067 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:03.763964    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 15:56:03.764067 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:03.763948    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 15:56:03.774024 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:03.774000    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-f65c965db-7f4nh"]
Sep 17 15:56:03.780670 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:03.780640    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-59669fb66b-sjgjs"]
Sep 17 15:56:03.781702 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:03.781662    2585 patch_prober.go:28] interesting pod/image-registry-f65c965db-7f4nh container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 17 15:56:03.781945 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:03.781916    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-f65c965db-7f4nh" podUID="e0a58c2f-ea54-489b-86a9-d9c1d887b713" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:56:03.856319 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:03.856288    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ad94a5d3-6f44-4e5d-bf10-fe69a34e67b9-klusterlet-config\") pod \"klusterlet-addon-workmgr-59669fb66b-sjgjs\" (UID: \"ad94a5d3-6f44-4e5d-bf10-fe69a34e67b9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59669fb66b-sjgjs"
Sep 17 15:56:03.856531 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:03.856327    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad94a5d3-6f44-4e5d-bf10-fe69a34e67b9-tmp\") pod \"klusterlet-addon-workmgr-59669fb66b-sjgjs\" (UID: \"ad94a5d3-6f44-4e5d-bf10-fe69a34e67b9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59669fb66b-sjgjs"
Sep 17 15:56:03.856531 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:03.856414    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwr5c\" (UniqueName: \"kubernetes.io/projected/ad94a5d3-6f44-4e5d-bf10-fe69a34e67b9-kube-api-access-gwr5c\") pod \"klusterlet-addon-workmgr-59669fb66b-sjgjs\" (UID: \"ad94a5d3-6f44-4e5d-bf10-fe69a34e67b9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59669fb66b-sjgjs"
Sep 17 15:56:03.957756 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:03.957725    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwr5c\" (UniqueName: \"kubernetes.io/projected/ad94a5d3-6f44-4e5d-bf10-fe69a34e67b9-kube-api-access-gwr5c\") pod \"klusterlet-addon-workmgr-59669fb66b-sjgjs\" (UID: \"ad94a5d3-6f44-4e5d-bf10-fe69a34e67b9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59669fb66b-sjgjs"
Sep 17 15:56:03.957934 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:03.957771    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ad94a5d3-6f44-4e5d-bf10-fe69a34e67b9-klusterlet-config\") pod \"klusterlet-addon-workmgr-59669fb66b-sjgjs\" (UID: \"ad94a5d3-6f44-4e5d-bf10-fe69a34e67b9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59669fb66b-sjgjs"
Sep 17 15:56:03.957934 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:03.957807    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad94a5d3-6f44-4e5d-bf10-fe69a34e67b9-tmp\") pod \"klusterlet-addon-workmgr-59669fb66b-sjgjs\" (UID: \"ad94a5d3-6f44-4e5d-bf10-fe69a34e67b9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59669fb66b-sjgjs"
Sep 17 15:56:03.958267 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:03.958237    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad94a5d3-6f44-4e5d-bf10-fe69a34e67b9-tmp\") pod \"klusterlet-addon-workmgr-59669fb66b-sjgjs\" (UID: \"ad94a5d3-6f44-4e5d-bf10-fe69a34e67b9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59669fb66b-sjgjs"
Sep 17 15:56:03.960254 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:03.960232    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ad94a5d3-6f44-4e5d-bf10-fe69a34e67b9-klusterlet-config\") pod \"klusterlet-addon-workmgr-59669fb66b-sjgjs\" (UID: \"ad94a5d3-6f44-4e5d-bf10-fe69a34e67b9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59669fb66b-sjgjs"
Sep 17 15:56:03.977374 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:03.977344    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6ffcc66d5f-h4njt"]
Sep 17 15:56:03.980558 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:03.980537    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6ffcc66d5f-h4njt"
Sep 17 15:56:03.983044 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:56:03.983005    2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"image-registry-6ffcc66d5f-h4njt\" is forbidden: User \"system:node:ip-10-0-135-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-image-registry\": no relationship found between node 'ip-10-0-135-43.ec2.internal' and this object" podUID="8d471801-4e18-4b24-beb7-e1e8f56c0a0e" pod="openshift-image-registry/image-registry-6ffcc66d5f-h4njt"
Sep 17 15:56:03.984004 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:03.983983    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwr5c\" (UniqueName: \"kubernetes.io/projected/ad94a5d3-6f44-4e5d-bf10-fe69a34e67b9-kube-api-access-gwr5c\") pod \"klusterlet-addon-workmgr-59669fb66b-sjgjs\" (UID: \"ad94a5d3-6f44-4e5d-bf10-fe69a34e67b9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59669fb66b-sjgjs"
Sep 17 15:56:04.016480 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:04.016393    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6ffcc66d5f-h4njt"]
Sep 17 15:56:04.058267 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:04.058233    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8d471801-4e18-4b24-beb7-e1e8f56c0a0e-ca-trust-extracted\") pod \"image-registry-6ffcc66d5f-h4njt\" (UID: \"8d471801-4e18-4b24-beb7-e1e8f56c0a0e\") " pod="openshift-image-registry/image-registry-6ffcc66d5f-h4njt"
Sep 17 15:56:04.058433 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:04.058293    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8d471801-4e18-4b24-beb7-e1e8f56c0a0e-trusted-ca\") pod \"image-registry-6ffcc66d5f-h4njt\" (UID: \"8d471801-4e18-4b24-beb7-e1e8f56c0a0e\") " pod="openshift-image-registry/image-registry-6ffcc66d5f-h4njt"
Sep 17 15:56:04.058433 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:04.058311    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8d471801-4e18-4b24-beb7-e1e8f56c0a0e-bound-sa-token\") pod \"image-registry-6ffcc66d5f-h4njt\" (UID: \"8d471801-4e18-4b24-beb7-e1e8f56c0a0e\") " pod="openshift-image-registry/image-registry-6ffcc66d5f-h4njt"
Sep 17 15:56:04.058433 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:04.058391    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8d471801-4e18-4b24-beb7-e1e8f56c0a0e-registry-tls\") pod \"image-registry-6ffcc66d5f-h4njt\" (UID: \"8d471801-4e18-4b24-beb7-e1e8f56c0a0e\") " pod="openshift-image-registry/image-registry-6ffcc66d5f-h4njt"
Sep 17 15:56:04.058556 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:04.058440    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvbp7\" (UniqueName: \"kubernetes.io/projected/8d471801-4e18-4b24-beb7-e1e8f56c0a0e-kube-api-access-bvbp7\") pod \"image-registry-6ffcc66d5f-h4njt\" (UID: \"8d471801-4e18-4b24-beb7-e1e8f56c0a0e\") " pod="openshift-image-registry/image-registry-6ffcc66d5f-h4njt"
Sep 17 15:56:04.058556 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:04.058472    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8d471801-4e18-4b24-beb7-e1e8f56c0a0e-installation-pull-secrets\") pod \"image-registry-6ffcc66d5f-h4njt\" (UID: \"8d471801-4e18-4b24-beb7-e1e8f56c0a0e\") " pod="openshift-image-registry/image-registry-6ffcc66d5f-h4njt"
Sep 17 15:56:04.058556 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:04.058494    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8d471801-4e18-4b24-beb7-e1e8f56c0a0e-image-registry-private-configuration\") pod \"image-registry-6ffcc66d5f-h4njt\" (UID: \"8d471801-4e18-4b24-beb7-e1e8f56c0a0e\") " pod="openshift-image-registry/image-registry-6ffcc66d5f-h4njt"
Sep 17 15:56:04.058556 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:04.058515    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8d471801-4e18-4b24-beb7-e1e8f56c0a0e-registry-certificates\") pod \"image-registry-6ffcc66d5f-h4njt\" (UID: \"8d471801-4e18-4b24-beb7-e1e8f56c0a0e\") " pod="openshift-image-registry/image-registry-6ffcc66d5f-h4njt"
Sep 17 15:56:04.069283 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:04.069252    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59669fb66b-sjgjs"
Sep 17 15:56:04.159354 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:04.159322    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8d471801-4e18-4b24-beb7-e1e8f56c0a0e-registry-tls\") pod \"image-registry-6ffcc66d5f-h4njt\" (UID: \"8d471801-4e18-4b24-beb7-e1e8f56c0a0e\") " pod="openshift-image-registry/image-registry-6ffcc66d5f-h4njt"
Sep 17 15:56:04.159354 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:04.159358    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvbp7\" (UniqueName: \"kubernetes.io/projected/8d471801-4e18-4b24-beb7-e1e8f56c0a0e-kube-api-access-bvbp7\") pod \"image-registry-6ffcc66d5f-h4njt\" (UID: \"8d471801-4e18-4b24-beb7-e1e8f56c0a0e\") " pod="openshift-image-registry/image-registry-6ffcc66d5f-h4njt"
Sep 17 15:56:04.159596 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:04.159375    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8d471801-4e18-4b24-beb7-e1e8f56c0a0e-installation-pull-secrets\") pod \"image-registry-6ffcc66d5f-h4njt\" (UID: \"8d471801-4e18-4b24-beb7-e1e8f56c0a0e\") " pod="openshift-image-registry/image-registry-6ffcc66d5f-h4njt"
Sep 17 15:56:04.159596 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:04.159536    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8d471801-4e18-4b24-beb7-e1e8f56c0a0e-image-registry-private-configuration\") pod \"image-registry-6ffcc66d5f-h4njt\" (UID: \"8d471801-4e18-4b24-beb7-e1e8f56c0a0e\") " pod="openshift-image-registry/image-registry-6ffcc66d5f-h4njt"
Sep 17 15:56:04.159596 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:04.159587    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8d471801-4e18-4b24-beb7-e1e8f56c0a0e-registry-certificates\") pod \"image-registry-6ffcc66d5f-h4njt\" (UID: \"8d471801-4e18-4b24-beb7-e1e8f56c0a0e\") " pod="openshift-image-registry/image-registry-6ffcc66d5f-h4njt"
Sep 17 15:56:04.159744 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:04.159625    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8d471801-4e18-4b24-beb7-e1e8f56c0a0e-ca-trust-extracted\") pod \"image-registry-6ffcc66d5f-h4njt\" (UID: \"8d471801-4e18-4b24-beb7-e1e8f56c0a0e\") " pod="openshift-image-registry/image-registry-6ffcc66d5f-h4njt"
Sep 17 15:56:04.159744 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:04.159696    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8d471801-4e18-4b24-beb7-e1e8f56c0a0e-trusted-ca\") pod \"image-registry-6ffcc66d5f-h4njt\" (UID: \"8d471801-4e18-4b24-beb7-e1e8f56c0a0e\") " pod="openshift-image-registry/image-registry-6ffcc66d5f-h4njt"
Sep 17 15:56:04.159744 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:04.159726    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8d471801-4e18-4b24-beb7-e1e8f56c0a0e-bound-sa-token\") pod \"image-registry-6ffcc66d5f-h4njt\" (UID: \"8d471801-4e18-4b24-beb7-e1e8f56c0a0e\") " pod="openshift-image-registry/image-registry-6ffcc66d5f-h4njt"
Sep 17 15:56:04.160175 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:04.160148    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8d471801-4e18-4b24-beb7-e1e8f56c0a0e-ca-trust-extracted\") pod \"image-registry-6ffcc66d5f-h4njt\" (UID: \"8d471801-4e18-4b24-beb7-e1e8f56c0a0e\") " pod="openshift-image-registry/image-registry-6ffcc66d5f-h4njt"
Sep 17 15:56:04.160610 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:04.160588    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8d471801-4e18-4b24-beb7-e1e8f56c0a0e-registry-certificates\") pod \"image-registry-6ffcc66d5f-h4njt\" (UID: \"8d471801-4e18-4b24-beb7-e1e8f56c0a0e\") " pod="openshift-image-registry/image-registry-6ffcc66d5f-h4njt"
Sep 17 15:56:04.161042 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:04.161018    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8d471801-4e18-4b24-beb7-e1e8f56c0a0e-trusted-ca\") pod \"image-registry-6ffcc66d5f-h4njt\" (UID: \"8d471801-4e18-4b24-beb7-e1e8f56c0a0e\") " pod="openshift-image-registry/image-registry-6ffcc66d5f-h4njt"
Sep 17 15:56:04.161891 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:04.161873    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8d471801-4e18-4b24-beb7-e1e8f56c0a0e-installation-pull-secrets\") pod \"image-registry-6ffcc66d5f-h4njt\" (UID: \"8d471801-4e18-4b24-beb7-e1e8f56c0a0e\") " pod="openshift-image-registry/image-registry-6ffcc66d5f-h4njt"
Sep 17 15:56:04.162002 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:04.161940    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8d471801-4e18-4b24-beb7-e1e8f56c0a0e-image-registry-private-configuration\") pod \"image-registry-6ffcc66d5f-h4njt\" (UID: \"8d471801-4e18-4b24-beb7-e1e8f56c0a0e\") " pod="openshift-image-registry/image-registry-6ffcc66d5f-h4njt"
Sep 17 15:56:04.162061 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:04.162020    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8d471801-4e18-4b24-beb7-e1e8f56c0a0e-registry-tls\") pod \"image-registry-6ffcc66d5f-h4njt\" (UID: \"8d471801-4e18-4b24-beb7-e1e8f56c0a0e\") " pod="openshift-image-registry/image-registry-6ffcc66d5f-h4njt"
Sep 17 15:56:04.186701 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:04.186669    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8d471801-4e18-4b24-beb7-e1e8f56c0a0e-bound-sa-token\") pod \"image-registry-6ffcc66d5f-h4njt\" (UID: \"8d471801-4e18-4b24-beb7-e1e8f56c0a0e\") " pod="openshift-image-registry/image-registry-6ffcc66d5f-h4njt"
Sep 17 15:56:04.186869 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:04.186790    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvbp7\" (UniqueName: \"kubernetes.io/projected/8d471801-4e18-4b24-beb7-e1e8f56c0a0e-kube-api-access-bvbp7\") pod \"image-registry-6ffcc66d5f-h4njt\" (UID: \"8d471801-4e18-4b24-beb7-e1e8f56c0a0e\") " pod="openshift-image-registry/image-registry-6ffcc66d5f-h4njt"
Sep 17 15:56:04.294467 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:04.294345    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6ffcc66d5f-h4njt"
Sep 17 15:56:04.388654 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:04.388623    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-59669fb66b-sjgjs"]
Sep 17 15:56:04.389404 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:56:04.389382    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad94a5d3_6f44_4e5d_bf10_fe69a34e67b9.slice/crio-36c458ad7f865ea14a6806428e88946974005631fb8995890cf4fe988e80722b WatchSource:0}: Error finding container 36c458ad7f865ea14a6806428e88946974005631fb8995890cf4fe988e80722b: Status 404 returned error can't find the container with id 36c458ad7f865ea14a6806428e88946974005631fb8995890cf4fe988e80722b
Sep 17 15:56:04.597625 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:04.597550    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6ffcc66d5f-h4njt"]
Sep 17 15:56:04.600805 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:56:04.600780    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d471801_4e18_4b24_beb7_e1e8f56c0a0e.slice/crio-7d49fad7c411dca299303bc428e230a7c4e9948c86d7770cea5a084110de873d WatchSource:0}: Error finding container 7d49fad7c411dca299303bc428e230a7c4e9948c86d7770cea5a084110de873d: Status 404 returned error can't find the container with id 7d49fad7c411dca299303bc428e230a7c4e9948c86d7770cea5a084110de873d
Sep 17 15:56:05.076804 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:05.076763    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6ffcc66d5f-h4njt" event={"ID":"8d471801-4e18-4b24-beb7-e1e8f56c0a0e","Type":"ContainerStarted","Data":"1d2398bb112d03b4f3da290d40bde4c0affa2806c1ea0613d25aeb99831f3794"}
Sep 17 15:56:05.077305 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:05.076826    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6ffcc66d5f-h4njt" event={"ID":"8d471801-4e18-4b24-beb7-e1e8f56c0a0e","Type":"ContainerStarted","Data":"7d49fad7c411dca299303bc428e230a7c4e9948c86d7770cea5a084110de873d"}
Sep 17 15:56:05.077305 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:05.076875    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6ffcc66d5f-h4njt"
Sep 17 15:56:05.077984 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:05.077961    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59669fb66b-sjgjs" event={"ID":"ad94a5d3-6f44-4e5d-bf10-fe69a34e67b9","Type":"ContainerStarted","Data":"36c458ad7f865ea14a6806428e88946974005631fb8995890cf4fe988e80722b"}
Sep 17 15:56:07.580402 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:07.580334    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6ffcc66d5f-h4njt" podStartSLOduration=4.580314648 podStartE2EDuration="4.580314648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:56:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:56:05.104464793 +0000 UTC m=+120.874796590" watchObservedRunningTime="2026-09-17 15:56:07.580314648 +0000 UTC m=+123.350646445"
Sep 17 15:56:07.581380 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:07.581363    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-fch49"]
Sep 17 15:56:07.586151 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:07.586133    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-fch49"
Sep 17 15:56:07.589093 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:07.589064    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 17 15:56:07.589093 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:07.589078    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 17 15:56:07.590183 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:07.590165    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-dw7xf\""
Sep 17 15:56:07.590183 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:07.590178    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 15:56:07.598271 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:07.598244    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-fch49"]
Sep 17 15:56:07.687624 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:07.687587    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cf657178-2022-4f3b-a53e-c45eb342c5e6-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-fch49\" (UID: \"cf657178-2022-4f3b-a53e-c45eb342c5e6\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-fch49"
Sep 17 15:56:07.687789 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:07.687660    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf657178-2022-4f3b-a53e-c45eb342c5e6-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-fch49\" (UID: \"cf657178-2022-4f3b-a53e-c45eb342c5e6\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-fch49"
Sep 17 15:56:07.687789 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:07.687686    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z47h\" (UniqueName: \"kubernetes.io/projected/cf657178-2022-4f3b-a53e-c45eb342c5e6-kube-api-access-8z47h\") pod \"prometheus-operator-7bd5d56d6-fch49\" (UID: \"cf657178-2022-4f3b-a53e-c45eb342c5e6\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-fch49"
Sep 17 15:56:07.687789 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:07.687703    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cf657178-2022-4f3b-a53e-c45eb342c5e6-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-fch49\" (UID: \"cf657178-2022-4f3b-a53e-c45eb342c5e6\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-fch49"
Sep 17 15:56:07.788052 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:07.788007    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cf657178-2022-4f3b-a53e-c45eb342c5e6-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-fch49\" (UID: \"cf657178-2022-4f3b-a53e-c45eb342c5e6\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-fch49"
Sep 17 15:56:07.788245 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:07.788072    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf657178-2022-4f3b-a53e-c45eb342c5e6-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-fch49\" (UID: \"cf657178-2022-4f3b-a53e-c45eb342c5e6\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-fch49"
Sep 17 15:56:07.788245 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:07.788091    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z47h\" (UniqueName: \"kubernetes.io/projected/cf657178-2022-4f3b-a53e-c45eb342c5e6-kube-api-access-8z47h\") pod \"prometheus-operator-7bd5d56d6-fch49\" (UID: \"cf657178-2022-4f3b-a53e-c45eb342c5e6\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-fch49"
Sep 17 15:56:07.788245 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:07.788111    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cf657178-2022-4f3b-a53e-c45eb342c5e6-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-fch49\" (UID: \"cf657178-2022-4f3b-a53e-c45eb342c5e6\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-fch49"
Sep 17 15:56:07.788405 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:56:07.788273    2585 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 17 15:56:07.788405 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:56:07.788333    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf657178-2022-4f3b-a53e-c45eb342c5e6-prometheus-operator-tls podName:cf657178-2022-4f3b-a53e-c45eb342c5e6 nodeName:}" failed. No retries permitted until 2026-09-17 15:56:08.28831736 +0000 UTC m=+124.058649135 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/cf657178-2022-4f3b-a53e-c45eb342c5e6-prometheus-operator-tls") pod "prometheus-operator-7bd5d56d6-fch49" (UID: "cf657178-2022-4f3b-a53e-c45eb342c5e6") : secret "prometheus-operator-tls" not found
Sep 17 15:56:07.788829 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:07.788809    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf657178-2022-4f3b-a53e-c45eb342c5e6-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-fch49\" (UID: \"cf657178-2022-4f3b-a53e-c45eb342c5e6\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-fch49"
Sep 17 15:56:07.790518 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:07.790495    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cf657178-2022-4f3b-a53e-c45eb342c5e6-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-fch49\" (UID: \"cf657178-2022-4f3b-a53e-c45eb342c5e6\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-fch49"
Sep 17 15:56:07.799803 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:07.799773    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z47h\" (UniqueName: \"kubernetes.io/projected/cf657178-2022-4f3b-a53e-c45eb342c5e6-kube-api-access-8z47h\") pod \"prometheus-operator-7bd5d56d6-fch49\" (UID: \"cf657178-2022-4f3b-a53e-c45eb342c5e6\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-fch49"
Sep 17 15:56:08.292377 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:08.292328    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cf657178-2022-4f3b-a53e-c45eb342c5e6-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-fch49\" (UID: \"cf657178-2022-4f3b-a53e-c45eb342c5e6\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-fch49"
Sep 17 15:56:08.294705 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:08.294676    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cf657178-2022-4f3b-a53e-c45eb342c5e6-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-fch49\" (UID: \"cf657178-2022-4f3b-a53e-c45eb342c5e6\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-fch49"
Sep 17 15:56:08.494475 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:08.494436    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-fch49"
Sep 17 15:56:08.777894 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:08.777860    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-fch49"]
Sep 17 15:56:08.781300 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:56:08.781273    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf657178_2022_4f3b_a53e_c45eb342c5e6.slice/crio-36b46c85adad7d47ace1268635b23e334eb406743f68322526e020ddc6c530bd WatchSource:0}: Error finding container 36b46c85adad7d47ace1268635b23e334eb406743f68322526e020ddc6c530bd: Status 404 returned error can't find the container with id 36b46c85adad7d47ace1268635b23e334eb406743f68322526e020ddc6c530bd
Sep 17 15:56:09.092340 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:09.092299    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-fch49" event={"ID":"cf657178-2022-4f3b-a53e-c45eb342c5e6","Type":"ContainerStarted","Data":"36b46c85adad7d47ace1268635b23e334eb406743f68322526e020ddc6c530bd"}
Sep 17 15:56:11.101493 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:11.101454    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-fch49" event={"ID":"cf657178-2022-4f3b-a53e-c45eb342c5e6","Type":"ContainerStarted","Data":"220bca99f22b622f9e22699a7d0e66358fdcbf08d9cae3eed6ece7309a5eecf7"}
Sep 17 15:56:11.101915 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:11.101517    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-fch49" event={"ID":"cf657178-2022-4f3b-a53e-c45eb342c5e6","Type":"ContainerStarted","Data":"237cc629257e7c40cd47d1a97175945236004b7b5e28bfe17e7a4553615ef6fc"}
Sep 17 15:56:11.125802 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:11.125735    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-fch49" podStartSLOduration=2.8124518370000002 podStartE2EDuration="4.125715047s" totalImagesPullingTime="1.31326321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:56:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:56:11.125051335 +0000 UTC m=+126.895383133" watchObservedRunningTime="2026-09-17 15:56:11.125715047 +0000 UTC m=+126.896046856"
Sep 17 15:56:13.136328 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.136292    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-5bcr2"]
Sep 17 15:56:13.139517 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.139497    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-5bcr2"
Sep 17 15:56:13.149628 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.149612    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 17 15:56:13.151020 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.151003    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 17 15:56:13.151076 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.151040    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 17 15:56:13.152179 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.152166    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-m877r\""
Sep 17 15:56:13.171336 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.171307    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-5bcr2"]
Sep 17 15:56:13.202640 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.202605    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-99ndh"]
Sep 17 15:56:13.206106 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.206078    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-99ndh"
Sep 17 15:56:13.218177 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.218157    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 15:56:13.218750 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.218734    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-w9x66\""
Sep 17 15:56:13.218858 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.218801    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 15:56:13.219220 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.219208    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 15:56:13.233040 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.233008    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8368d252-ce4a-4aea-94d4-7956729c3219-root\") pod \"node-exporter-99ndh\" (UID: \"8368d252-ce4a-4aea-94d4-7956729c3219\") " pod="openshift-monitoring/node-exporter-99ndh"
Sep 17 15:56:13.233146 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.233057    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8368d252-ce4a-4aea-94d4-7956729c3219-node-exporter-wtmp\") pod \"node-exporter-99ndh\" (UID: \"8368d252-ce4a-4aea-94d4-7956729c3219\") " pod="openshift-monitoring/node-exporter-99ndh"
Sep 17 15:56:13.233146 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.233085    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c1fb615a-b839-4e47-b63b-47eec13d57cc-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-5bcr2\" (UID: \"c1fb615a-b839-4e47-b63b-47eec13d57cc\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-5bcr2"
Sep 17 15:56:13.233146 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.233109    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8368d252-ce4a-4aea-94d4-7956729c3219-node-exporter-textfile\") pod \"node-exporter-99ndh\" (UID: \"8368d252-ce4a-4aea-94d4-7956729c3219\") " pod="openshift-monitoring/node-exporter-99ndh"
Sep 17 15:56:13.233276 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.233148    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8368d252-ce4a-4aea-94d4-7956729c3219-sys\") pod \"node-exporter-99ndh\" (UID: \"8368d252-ce4a-4aea-94d4-7956729c3219\") " pod="openshift-monitoring/node-exporter-99ndh"
Sep 17 15:56:13.233276 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.233221    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b4fg\" (UniqueName: \"kubernetes.io/projected/c1fb615a-b839-4e47-b63b-47eec13d57cc-kube-api-access-4b4fg\") pod \"kube-state-metrics-84c885658c-5bcr2\" (UID: \"c1fb615a-b839-4e47-b63b-47eec13d57cc\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-5bcr2"
Sep 17 15:56:13.233276 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.233260    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8368d252-ce4a-4aea-94d4-7956729c3219-metrics-client-ca\") pod \"node-exporter-99ndh\" (UID: \"8368d252-ce4a-4aea-94d4-7956729c3219\") " pod="openshift-monitoring/node-exporter-99ndh"
Sep 17 15:56:13.233404 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.233295    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dz7v\" (UniqueName: \"kubernetes.io/projected/8368d252-ce4a-4aea-94d4-7956729c3219-kube-api-access-9dz7v\") pod \"node-exporter-99ndh\" (UID: \"8368d252-ce4a-4aea-94d4-7956729c3219\") " pod="openshift-monitoring/node-exporter-99ndh"
Sep 17 15:56:13.233404 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.233316    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8368d252-ce4a-4aea-94d4-7956729c3219-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-99ndh\" (UID: \"8368d252-ce4a-4aea-94d4-7956729c3219\") " pod="openshift-monitoring/node-exporter-99ndh"
Sep 17 15:56:13.233404 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.233366    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c1fb615a-b839-4e47-b63b-47eec13d57cc-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-5bcr2\" (UID: \"c1fb615a-b839-4e47-b63b-47eec13d57cc\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-5bcr2"
Sep 17 15:56:13.233646 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.233407    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8368d252-ce4a-4aea-94d4-7956729c3219-node-exporter-tls\") pod \"node-exporter-99ndh\" (UID: \"8368d252-ce4a-4aea-94d4-7956729c3219\") " pod="openshift-monitoring/node-exporter-99ndh"
Sep 17 15:56:13.233646 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.233476    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8368d252-ce4a-4aea-94d4-7956729c3219-node-exporter-accelerators-collector-config\") pod \"node-exporter-99ndh\" (UID: \"8368d252-ce4a-4aea-94d4-7956729c3219\") " pod="openshift-monitoring/node-exporter-99ndh"
Sep 17 15:56:13.233646 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.233509    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c1fb615a-b839-4e47-b63b-47eec13d57cc-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-5bcr2\" (UID: \"c1fb615a-b839-4e47-b63b-47eec13d57cc\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-5bcr2"
Sep 17 15:56:13.233646 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.233543    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c1fb615a-b839-4e47-b63b-47eec13d57cc-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-5bcr2\" (UID: \"c1fb615a-b839-4e47-b63b-47eec13d57cc\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-5bcr2"
Sep 17 15:56:13.233646 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.233577    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c1fb615a-b839-4e47-b63b-47eec13d57cc-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-5bcr2\" (UID: \"c1fb615a-b839-4e47-b63b-47eec13d57cc\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-5bcr2"
Sep 17 15:56:13.334752 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.334717    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dz7v\" (UniqueName: \"kubernetes.io/projected/8368d252-ce4a-4aea-94d4-7956729c3219-kube-api-access-9dz7v\") pod \"node-exporter-99ndh\" (UID: \"8368d252-ce4a-4aea-94d4-7956729c3219\") " pod="openshift-monitoring/node-exporter-99ndh"
Sep 17 15:56:13.334752 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.334755    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8368d252-ce4a-4aea-94d4-7956729c3219-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-99ndh\" (UID: \"8368d252-ce4a-4aea-94d4-7956729c3219\") " pod="openshift-monitoring/node-exporter-99ndh"
Sep 17 15:56:13.334951 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.334806    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c1fb615a-b839-4e47-b63b-47eec13d57cc-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-5bcr2\" (UID: \"c1fb615a-b839-4e47-b63b-47eec13d57cc\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-5bcr2"
Sep 17 15:56:13.334951 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.334866    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8368d252-ce4a-4aea-94d4-7956729c3219-node-exporter-tls\") pod \"node-exporter-99ndh\" (UID: \"8368d252-ce4a-4aea-94d4-7956729c3219\") " pod="openshift-monitoring/node-exporter-99ndh"
Sep 17 15:56:13.334951 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.334902    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8368d252-ce4a-4aea-94d4-7956729c3219-node-exporter-accelerators-collector-config\") pod \"node-exporter-99ndh\" (UID: \"8368d252-ce4a-4aea-94d4-7956729c3219\") " pod="openshift-monitoring/node-exporter-99ndh"
Sep 17 15:56:13.334951 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.334933    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c1fb615a-b839-4e47-b63b-47eec13d57cc-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-5bcr2\" (UID: \"c1fb615a-b839-4e47-b63b-47eec13d57cc\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-5bcr2"
Sep 17 15:56:13.335116 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.334976    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c1fb615a-b839-4e47-b63b-47eec13d57cc-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-5bcr2\" (UID: \"c1fb615a-b839-4e47-b63b-47eec13d57cc\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-5bcr2"
Sep 17 15:56:13.335116 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:56:13.335000    2585 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 17 15:56:13.335116 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.335004    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c1fb615a-b839-4e47-b63b-47eec13d57cc-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-5bcr2\" (UID: \"c1fb615a-b839-4e47-b63b-47eec13d57cc\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-5bcr2"
Sep 17 15:56:13.335116 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:56:13.335062    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8368d252-ce4a-4aea-94d4-7956729c3219-node-exporter-tls podName:8368d252-ce4a-4aea-94d4-7956729c3219 nodeName:}" failed. No retries permitted until 2026-09-17 15:56:13.835041113 +0000 UTC m=+129.605372903 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/8368d252-ce4a-4aea-94d4-7956729c3219-node-exporter-tls") pod "node-exporter-99ndh" (UID: "8368d252-ce4a-4aea-94d4-7956729c3219") : secret "node-exporter-tls" not found
Sep 17 15:56:13.335116 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.335088    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8368d252-ce4a-4aea-94d4-7956729c3219-root\") pod \"node-exporter-99ndh\" (UID: \"8368d252-ce4a-4aea-94d4-7956729c3219\") " pod="openshift-monitoring/node-exporter-99ndh"
Sep 17 15:56:13.335116 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.335114    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8368d252-ce4a-4aea-94d4-7956729c3219-node-exporter-wtmp\") pod \"node-exporter-99ndh\" (UID: \"8368d252-ce4a-4aea-94d4-7956729c3219\") " pod="openshift-monitoring/node-exporter-99ndh"
Sep 17 15:56:13.335445 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.335143    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c1fb615a-b839-4e47-b63b-47eec13d57cc-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-5bcr2\" (UID: \"c1fb615a-b839-4e47-b63b-47eec13d57cc\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-5bcr2"
Sep 17 15:56:13.335445 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.335166    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8368d252-ce4a-4aea-94d4-7956729c3219-node-exporter-textfile\") pod \"node-exporter-99ndh\" (UID: \"8368d252-ce4a-4aea-94d4-7956729c3219\") " pod="openshift-monitoring/node-exporter-99ndh"
Sep 17 15:56:13.335445 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.335205    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8368d252-ce4a-4aea-94d4-7956729c3219-sys\") pod \"node-exporter-99ndh\" (UID: \"8368d252-ce4a-4aea-94d4-7956729c3219\") " pod="openshift-monitoring/node-exporter-99ndh"
Sep 17 15:56:13.335445 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.335243    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b4fg\" (UniqueName: \"kubernetes.io/projected/c1fb615a-b839-4e47-b63b-47eec13d57cc-kube-api-access-4b4fg\") pod \"kube-state-metrics-84c885658c-5bcr2\" (UID: \"c1fb615a-b839-4e47-b63b-47eec13d57cc\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-5bcr2"
Sep 17 15:56:13.335445 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.335265    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8368d252-ce4a-4aea-94d4-7956729c3219-metrics-client-ca\") pod \"node-exporter-99ndh\" (UID: \"8368d252-ce4a-4aea-94d4-7956729c3219\") " pod="openshift-monitoring/node-exporter-99ndh"
Sep 17 15:56:13.335445 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.335279    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8368d252-ce4a-4aea-94d4-7956729c3219-node-exporter-wtmp\") pod \"node-exporter-99ndh\" (UID: \"8368d252-ce4a-4aea-94d4-7956729c3219\") " pod="openshift-monitoring/node-exporter-99ndh"
Sep 17 15:56:13.335445 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.335165    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8368d252-ce4a-4aea-94d4-7956729c3219-root\") pod \"node-exporter-99ndh\" (UID: \"8368d252-ce4a-4aea-94d4-7956729c3219\") " pod="openshift-monitoring/node-exporter-99ndh"
Sep 17 15:56:13.335796 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.335480    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8368d252-ce4a-4aea-94d4-7956729c3219-sys\") pod \"node-exporter-99ndh\" (UID: \"8368d252-ce4a-4aea-94d4-7956729c3219\") " pod="openshift-monitoring/node-exporter-99ndh"
Sep 17 15:56:13.335796 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.335640    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8368d252-ce4a-4aea-94d4-7956729c3219-node-exporter-accelerators-collector-config\") pod \"node-exporter-99ndh\" (UID: \"8368d252-ce4a-4aea-94d4-7956729c3219\") " pod="openshift-monitoring/node-exporter-99ndh"
Sep 17 15:56:13.335796 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.335640    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c1fb615a-b839-4e47-b63b-47eec13d57cc-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-5bcr2\" (UID: \"c1fb615a-b839-4e47-b63b-47eec13d57cc\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-5bcr2"
Sep 17 15:56:13.335951 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.335838    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8368d252-ce4a-4aea-94d4-7956729c3219-metrics-client-ca\") pod \"node-exporter-99ndh\" (UID: \"8368d252-ce4a-4aea-94d4-7956729c3219\") " pod="openshift-monitoring/node-exporter-99ndh"
Sep 17 15:56:13.335951 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.335864    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8368d252-ce4a-4aea-94d4-7956729c3219-node-exporter-textfile\") pod \"node-exporter-99ndh\" (UID: \"8368d252-ce4a-4aea-94d4-7956729c3219\") " pod="openshift-monitoring/node-exporter-99ndh"
Sep 17 15:56:13.335951 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.335891    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c1fb615a-b839-4e47-b63b-47eec13d57cc-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-5bcr2\" (UID: \"c1fb615a-b839-4e47-b63b-47eec13d57cc\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-5bcr2"
Sep 17 15:56:13.336225 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.336199    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c1fb615a-b839-4e47-b63b-47eec13d57cc-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-5bcr2\" (UID: \"c1fb615a-b839-4e47-b63b-47eec13d57cc\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-5bcr2"
Sep 17 15:56:13.337758 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.337727    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8368d252-ce4a-4aea-94d4-7956729c3219-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-99ndh\" (UID: \"8368d252-ce4a-4aea-94d4-7956729c3219\") " pod="openshift-monitoring/node-exporter-99ndh"
Sep 17 15:56:13.337857 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.337832    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c1fb615a-b839-4e47-b63b-47eec13d57cc-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-5bcr2\" (UID: \"c1fb615a-b839-4e47-b63b-47eec13d57cc\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-5bcr2"
Sep 17 15:56:13.338017 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.338001    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c1fb615a-b839-4e47-b63b-47eec13d57cc-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-5bcr2\" (UID: \"c1fb615a-b839-4e47-b63b-47eec13d57cc\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-5bcr2"
Sep 17 15:56:13.345716 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.345691    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dz7v\" (UniqueName: \"kubernetes.io/projected/8368d252-ce4a-4aea-94d4-7956729c3219-kube-api-access-9dz7v\") pod \"node-exporter-99ndh\" (UID: \"8368d252-ce4a-4aea-94d4-7956729c3219\") " pod="openshift-monitoring/node-exporter-99ndh"
Sep 17 15:56:13.345916 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.345898    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b4fg\" (UniqueName: \"kubernetes.io/projected/c1fb615a-b839-4e47-b63b-47eec13d57cc-kube-api-access-4b4fg\") pod \"kube-state-metrics-84c885658c-5bcr2\" (UID: \"c1fb615a-b839-4e47-b63b-47eec13d57cc\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-5bcr2"
Sep 17 15:56:13.448651 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.448622    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-5bcr2"
Sep 17 15:56:13.735973 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.735891    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-5bcr2"]
Sep 17 15:56:13.739468 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:56:13.739444    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1fb615a_b839_4e47_b63b_47eec13d57cc.slice/crio-6990fe3f424ef746bfdf9fcab826ac70eb885fcb99823670b945b87454b4ec96 WatchSource:0}: Error finding container 6990fe3f424ef746bfdf9fcab826ac70eb885fcb99823670b945b87454b4ec96: Status 404 returned error can't find the container with id 6990fe3f424ef746bfdf9fcab826ac70eb885fcb99823670b945b87454b4ec96
Sep 17 15:56:13.778692 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.778663    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-f65c965db-7f4nh"
Sep 17 15:56:13.839076 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.839043    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8368d252-ce4a-4aea-94d4-7956729c3219-node-exporter-tls\") pod \"node-exporter-99ndh\" (UID: \"8368d252-ce4a-4aea-94d4-7956729c3219\") " pod="openshift-monitoring/node-exporter-99ndh"
Sep 17 15:56:13.841498 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:13.841473    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8368d252-ce4a-4aea-94d4-7956729c3219-node-exporter-tls\") pod \"node-exporter-99ndh\" (UID: \"8368d252-ce4a-4aea-94d4-7956729c3219\") " pod="openshift-monitoring/node-exporter-99ndh"
Sep 17 15:56:14.114444 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:14.114321    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-99ndh"
Sep 17 15:56:14.115372 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:14.115333    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-5bcr2" event={"ID":"c1fb615a-b839-4e47-b63b-47eec13d57cc","Type":"ContainerStarted","Data":"6990fe3f424ef746bfdf9fcab826ac70eb885fcb99823670b945b87454b4ec96"}
Sep 17 15:56:14.128726 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:56:14.128697    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8368d252_ce4a_4aea_94d4_7956729c3219.slice/crio-4cbd4e9051e31817b39e1cd8e98916feadd454983ddade01a922551f002e3813 WatchSource:0}: Error finding container 4cbd4e9051e31817b39e1cd8e98916feadd454983ddade01a922551f002e3813: Status 404 returned error can't find the container with id 4cbd4e9051e31817b39e1cd8e98916feadd454983ddade01a922551f002e3813
Sep 17 15:56:15.122143 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:15.122101    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-99ndh" event={"ID":"8368d252-ce4a-4aea-94d4-7956729c3219","Type":"ContainerStarted","Data":"4cbd4e9051e31817b39e1cd8e98916feadd454983ddade01a922551f002e3813"}
Sep 17 15:56:16.262159 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:16.262121    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7adc5269-e778-473a-a6c7-04754f5e585c-metrics-certs\") pod \"network-metrics-daemon-m6w9z\" (UID: \"7adc5269-e778-473a-a6c7-04754f5e585c\") " pod="openshift-multus/network-metrics-daemon-m6w9z"
Sep 17 15:56:16.264443 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:16.264395    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7adc5269-e778-473a-a6c7-04754f5e585c-metrics-certs\") pod \"network-metrics-daemon-m6w9z\" (UID: \"7adc5269-e778-473a-a6c7-04754f5e585c\") " pod="openshift-multus/network-metrics-daemon-m6w9z"
Sep 17 15:56:16.339693 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:16.339663    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9smtj\""
Sep 17 15:56:16.347254 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:16.347225    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m6w9z"
Sep 17 15:56:17.197491 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:17.197448    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5976878f85-g6lhx"]
Sep 17 15:56:17.201544 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:17.201514    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5976878f85-g6lhx"
Sep 17 15:56:17.206886 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:17.206858    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 17 15:56:17.207024 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:17.206935    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 17 15:56:17.207100 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:17.207053    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 17 15:56:17.207100 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:17.207065    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 17 15:56:17.207234 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:17.207206    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-4nn0qnkq62jdo\""
Sep 17 15:56:17.207733 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:17.207669    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-h27mj\""
Sep 17 15:56:17.208646 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:17.208593    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 17 15:56:17.222695 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:17.222667    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5976878f85-g6lhx"]
Sep 17 15:56:17.270730 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:17.270686    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fc63c498-e311-4eac-9aba-166507716705-secret-grpc-tls\") pod \"thanos-querier-5976878f85-g6lhx\" (UID: \"fc63c498-e311-4eac-9aba-166507716705\") " pod="openshift-monitoring/thanos-querier-5976878f85-g6lhx"
Sep 17 15:56:17.271176 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:17.270753    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bqrl\" (UniqueName: \"kubernetes.io/projected/fc63c498-e311-4eac-9aba-166507716705-kube-api-access-4bqrl\") pod \"thanos-querier-5976878f85-g6lhx\" (UID: \"fc63c498-e311-4eac-9aba-166507716705\") " pod="openshift-monitoring/thanos-querier-5976878f85-g6lhx"
Sep 17 15:56:17.271176 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:17.270783    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fc63c498-e311-4eac-9aba-166507716705-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5976878f85-g6lhx\" (UID: \"fc63c498-e311-4eac-9aba-166507716705\") " pod="openshift-monitoring/thanos-querier-5976878f85-g6lhx"
Sep 17 15:56:17.271176 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:17.270872    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc63c498-e311-4eac-9aba-166507716705-metrics-client-ca\") pod \"thanos-querier-5976878f85-g6lhx\" (UID: \"fc63c498-e311-4eac-9aba-166507716705\") " pod="openshift-monitoring/thanos-querier-5976878f85-g6lhx"
Sep 17 15:56:17.271176 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:17.270947    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fc63c498-e311-4eac-9aba-166507716705-secret-thanos-querier-tls\") pod \"thanos-querier-5976878f85-g6lhx\" (UID: \"fc63c498-e311-4eac-9aba-166507716705\") " pod="openshift-monitoring/thanos-querier-5976878f85-g6lhx"
Sep 17 15:56:17.271176 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:17.271007    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fc63c498-e311-4eac-9aba-166507716705-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5976878f85-g6lhx\" (UID: \"fc63c498-e311-4eac-9aba-166507716705\") " pod="openshift-monitoring/thanos-querier-5976878f85-g6lhx"
Sep 17 15:56:17.271176 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:17.271031    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fc63c498-e311-4eac-9aba-166507716705-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5976878f85-g6lhx\" (UID: \"fc63c498-e311-4eac-9aba-166507716705\") " pod="openshift-monitoring/thanos-querier-5976878f85-g6lhx"
Sep 17 15:56:17.271176 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:17.271059    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fc63c498-e311-4eac-9aba-166507716705-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5976878f85-g6lhx\" (UID: \"fc63c498-e311-4eac-9aba-166507716705\") " pod="openshift-monitoring/thanos-querier-5976878f85-g6lhx"
Sep 17 15:56:17.372197 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:17.372152    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fc63c498-e311-4eac-9aba-166507716705-secret-thanos-querier-tls\") pod \"thanos-querier-5976878f85-g6lhx\" (UID: \"fc63c498-e311-4eac-9aba-166507716705\") " pod="openshift-monitoring/thanos-querier-5976878f85-g6lhx"
Sep 17 15:56:17.372368 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:17.372218    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fc63c498-e311-4eac-9aba-166507716705-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5976878f85-g6lhx\" (UID: \"fc63c498-e311-4eac-9aba-166507716705\") " pod="openshift-monitoring/thanos-querier-5976878f85-g6lhx"
Sep 17 15:56:17.372368 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:17.372249    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fc63c498-e311-4eac-9aba-166507716705-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5976878f85-g6lhx\" (UID: \"fc63c498-e311-4eac-9aba-166507716705\") " pod="openshift-monitoring/thanos-querier-5976878f85-g6lhx"
Sep 17 15:56:17.372368 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:17.372274    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fc63c498-e311-4eac-9aba-166507716705-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5976878f85-g6lhx\" (UID: \"fc63c498-e311-4eac-9aba-166507716705\") " pod="openshift-monitoring/thanos-querier-5976878f85-g6lhx"
Sep 17 15:56:17.372368 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:17.372310    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fc63c498-e311-4eac-9aba-166507716705-secret-grpc-tls\") pod \"thanos-querier-5976878f85-g6lhx\" (UID: \"fc63c498-e311-4eac-9aba-166507716705\") " pod="openshift-monitoring/thanos-querier-5976878f85-g6lhx"
Sep 17 15:56:17.372368 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:17.372342    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bqrl\" (UniqueName: \"kubernetes.io/projected/fc63c498-e311-4eac-9aba-166507716705-kube-api-access-4bqrl\") pod \"thanos-querier-5976878f85-g6lhx\" (UID: \"fc63c498-e311-4eac-9aba-166507716705\") " pod="openshift-monitoring/thanos-querier-5976878f85-g6lhx"
Sep 17 15:56:17.372368 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:17.372365    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fc63c498-e311-4eac-9aba-166507716705-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5976878f85-g6lhx\" (UID: \"fc63c498-e311-4eac-9aba-166507716705\") " pod="openshift-monitoring/thanos-querier-5976878f85-g6lhx"
Sep 17 15:56:17.372708 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:17.372400    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc63c498-e311-4eac-9aba-166507716705-metrics-client-ca\") pod \"thanos-querier-5976878f85-g6lhx\" (UID: \"fc63c498-e311-4eac-9aba-166507716705\") " pod="openshift-monitoring/thanos-querier-5976878f85-g6lhx"
Sep 17 15:56:17.373285 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:17.373210    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc63c498-e311-4eac-9aba-166507716705-metrics-client-ca\") pod \"thanos-querier-5976878f85-g6lhx\" (UID: \"fc63c498-e311-4eac-9aba-166507716705\") " pod="openshift-monitoring/thanos-querier-5976878f85-g6lhx"
Sep 17 15:56:17.375779 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:17.375746    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fc63c498-e311-4eac-9aba-166507716705-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5976878f85-g6lhx\" (UID: \"fc63c498-e311-4eac-9aba-166507716705\") " pod="openshift-monitoring/thanos-querier-5976878f85-g6lhx"
Sep 17 15:56:17.375909 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:17.375853    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fc63c498-e311-4eac-9aba-166507716705-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5976878f85-g6lhx\" (UID: \"fc63c498-e311-4eac-9aba-166507716705\") " pod="openshift-monitoring/thanos-querier-5976878f85-g6lhx"
Sep 17 15:56:17.375909 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:17.375862    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fc63c498-e311-4eac-9aba-166507716705-secret-thanos-querier-tls\") pod \"thanos-querier-5976878f85-g6lhx\" (UID: \"fc63c498-e311-4eac-9aba-166507716705\") " pod="openshift-monitoring/thanos-querier-5976878f85-g6lhx"
Sep 17 15:56:17.376066 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:17.376045    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fc63c498-e311-4eac-9aba-166507716705-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5976878f85-g6lhx\" (UID: \"fc63c498-e311-4eac-9aba-166507716705\") " pod="openshift-monitoring/thanos-querier-5976878f85-g6lhx"
Sep 17 15:56:17.376398 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:17.376375    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fc63c498-e311-4eac-9aba-166507716705-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5976878f85-g6lhx\" (UID: \"fc63c498-e311-4eac-9aba-166507716705\") " pod="openshift-monitoring/thanos-querier-5976878f85-g6lhx"
Sep 17 15:56:17.376633 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:17.376612    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fc63c498-e311-4eac-9aba-166507716705-secret-grpc-tls\") pod \"thanos-querier-5976878f85-g6lhx\" (UID: \"fc63c498-e311-4eac-9aba-166507716705\") " pod="openshift-monitoring/thanos-querier-5976878f85-g6lhx"
Sep 17 15:56:17.382677 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:17.382654    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bqrl\" (UniqueName: \"kubernetes.io/projected/fc63c498-e311-4eac-9aba-166507716705-kube-api-access-4bqrl\") pod \"thanos-querier-5976878f85-g6lhx\" (UID: \"fc63c498-e311-4eac-9aba-166507716705\") " pod="openshift-monitoring/thanos-querier-5976878f85-g6lhx"
Sep 17 15:56:17.513182 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:17.513153    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5976878f85-g6lhx"
Sep 17 15:56:17.772228 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:17.772197    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-m6w9z"]
Sep 17 15:56:17.774849 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:56:17.774827    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7adc5269_e778_473a_a6c7_04754f5e585c.slice/crio-5327618cf99b415ee2d91545763d405a1b05830a45731ad05c0116784d33f15a WatchSource:0}: Error finding container 5327618cf99b415ee2d91545763d405a1b05830a45731ad05c0116784d33f15a: Status 404 returned error can't find the container with id 5327618cf99b415ee2d91545763d405a1b05830a45731ad05c0116784d33f15a
Sep 17 15:56:17.816037 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:17.816003    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5976878f85-g6lhx"]
Sep 17 15:56:17.819983 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:56:17.819956    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc63c498_e311_4eac_9aba_166507716705.slice/crio-acc8dd6863a23e5fb43120a04cd63235d4d985781e74e15a9a574339c25197ae WatchSource:0}: Error finding container acc8dd6863a23e5fb43120a04cd63235d4d985781e74e15a9a574339c25197ae: Status 404 returned error can't find the container with id acc8dd6863a23e5fb43120a04cd63235d4d985781e74e15a9a574339c25197ae
Sep 17 15:56:18.134639 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:18.134542    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5976878f85-g6lhx" event={"ID":"fc63c498-e311-4eac-9aba-166507716705","Type":"ContainerStarted","Data":"acc8dd6863a23e5fb43120a04cd63235d4d985781e74e15a9a574339c25197ae"}
Sep 17 15:56:18.135762 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:18.135735    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m6w9z" event={"ID":"7adc5269-e778-473a-a6c7-04754f5e585c","Type":"ContainerStarted","Data":"5327618cf99b415ee2d91545763d405a1b05830a45731ad05c0116784d33f15a"}
Sep 17 15:56:18.137224 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:18.137202    2585 generic.go:356] "Generic (PLEG): container finished" podID="8368d252-ce4a-4aea-94d4-7956729c3219" containerID="621e557cc7f0d29ae33201a7011edf93136f76558d7b59a493c062082d3ac2d5" exitCode=0
Sep 17 15:56:18.137328 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:18.137292    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-99ndh" event={"ID":"8368d252-ce4a-4aea-94d4-7956729c3219","Type":"ContainerDied","Data":"621e557cc7f0d29ae33201a7011edf93136f76558d7b59a493c062082d3ac2d5"}
Sep 17 15:56:18.138883 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:18.138856    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59669fb66b-sjgjs" event={"ID":"ad94a5d3-6f44-4e5d-bf10-fe69a34e67b9","Type":"ContainerStarted","Data":"e42d32d07a04b730356ece0cbd9f7e46122c87de5e2109dd826926d4a9ddd9a8"}
Sep 17 15:56:18.138995 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:18.138920    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59669fb66b-sjgjs"
Sep 17 15:56:18.141175 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:18.141154    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59669fb66b-sjgjs"
Sep 17 15:56:18.141535 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:18.141511    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-5bcr2" event={"ID":"c1fb615a-b839-4e47-b63b-47eec13d57cc","Type":"ContainerStarted","Data":"a9a595fcb145f9df3b17efd702ce5ba0b2427f110cb6de387f6b5e29c8a56c52"}
Sep 17 15:56:18.141649 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:18.141566    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-5bcr2" event={"ID":"c1fb615a-b839-4e47-b63b-47eec13d57cc","Type":"ContainerStarted","Data":"1d8dad7a3530ac870975f3c487be5ba4bdb25bf4807bca4f615551eb0e5dde79"}
Sep 17 15:56:18.141649 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:18.141597    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-5bcr2" event={"ID":"c1fb615a-b839-4e47-b63b-47eec13d57cc","Type":"ContainerStarted","Data":"313b8bc751b36f0a2299d2216d5dd7accc02ceda427796061672d73374cbffa0"}
Sep 17 15:56:18.181502 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:18.181447    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59669fb66b-sjgjs" podStartSLOduration=2.076034787 podStartE2EDuration="15.18140296s" totalImagesPullingTime="13.105368173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:56:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:56:18.180695582 +0000 UTC m=+133.951027382" watchObservedRunningTime="2026-09-17 15:56:18.18140296 +0000 UTC m=+133.951734758"
Sep 17 15:56:18.214584 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:18.214530    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-84c885658c-5bcr2" podStartSLOduration=1.473975863 podStartE2EDuration="5.214509865s" totalImagesPullingTime="3.740534002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:56:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:56:18.213973097 +0000 UTC m=+133.984304897" watchObservedRunningTime="2026-09-17 15:56:18.214509865 +0000 UTC m=+133.984841665"
Sep 17 15:56:19.148789 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:19.148751    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m6w9z" event={"ID":"7adc5269-e778-473a-a6c7-04754f5e585c","Type":"ContainerStarted","Data":"d8bf4ae3ae612c9cc5e2f1e132c5da0d9fb6f2d4a034d005567995835ce68639"}
Sep 17 15:56:19.149255 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:19.148821    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m6w9z" event={"ID":"7adc5269-e778-473a-a6c7-04754f5e585c","Type":"ContainerStarted","Data":"740233b5e4936bab22d4a727a552e16a7357ba2736c2967db602de41c32a4bee"}
Sep 17 15:56:19.151675 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:19.151633    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-99ndh" event={"ID":"8368d252-ce4a-4aea-94d4-7956729c3219","Type":"ContainerStarted","Data":"03974a663c13f543fd173518cf2cd56990a45ef933dff18da21ffbc090f57d35"}
Sep 17 15:56:19.151819 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:19.151693    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-99ndh" event={"ID":"8368d252-ce4a-4aea-94d4-7956729c3219","Type":"ContainerStarted","Data":"acc0a4552c3f60de1183c5edb8a847752423ccd39e9f8ecfb38ab20932d7d599"}
Sep 17 15:56:19.172563 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:19.172506    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-m6w9z" podStartSLOduration=131.253652675 podStartE2EDuration="2m12.172488193s" totalImagesPullingTime="918.835518ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:54:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:56:19.170253583 +0000 UTC m=+134.940585392" watchObservedRunningTime="2026-09-17 15:56:19.172488193 +0000 UTC m=+134.942819991"
Sep 17 15:56:19.209473 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:19.209373    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-99ndh" podStartSLOduration=2.770747562 podStartE2EDuration="6.209343422s" totalImagesPullingTime="3.351247478s" totalInitContainerRuntime="87.348382ms" isStatefulPod=true podCreationTimestamp="2026-09-17 15:56:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:56:19.208630803 +0000 UTC m=+134.978962625" watchObservedRunningTime="2026-09-17 15:56:19.209343422 +0000 UTC m=+134.979675313"
Sep 17 15:56:20.160489 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:20.160451    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5976878f85-g6lhx" event={"ID":"fc63c498-e311-4eac-9aba-166507716705","Type":"ContainerStarted","Data":"207e496be6550dc57f500db3528e9d9224c44ac52d3e77632974f03719d792d9"}
Sep 17 15:56:20.160891 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:20.160578    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5976878f85-g6lhx" event={"ID":"fc63c498-e311-4eac-9aba-166507716705","Type":"ContainerStarted","Data":"98d245307fe54359f59f0165fa83cde35752cac3667ba19e5da8b7072ce91229"}
Sep 17 15:56:20.160891 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:20.160623    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5976878f85-g6lhx" event={"ID":"fc63c498-e311-4eac-9aba-166507716705","Type":"ContainerStarted","Data":"6e75ea642612800d5f621dd9c98d6eaab2b642ee9b7619d5992c8afe7c43bf89"}
Sep 17 15:56:21.169750 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:21.169660    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5976878f85-g6lhx" event={"ID":"fc63c498-e311-4eac-9aba-166507716705","Type":"ContainerStarted","Data":"0fc7334f273509992cefbdaaad5d2410772489e1214b36bcfbe980a3cbf3f481"}
Sep 17 15:56:21.170138 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:21.169750    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5976878f85-g6lhx" event={"ID":"fc63c498-e311-4eac-9aba-166507716705","Type":"ContainerStarted","Data":"a48b9fa730259a73beadf3c317fb542a76203f474b2c1a99710308220bc24a9b"}
Sep 17 15:56:21.170138 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:21.169795    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5976878f85-g6lhx" event={"ID":"fc63c498-e311-4eac-9aba-166507716705","Type":"ContainerStarted","Data":"b443ed02babac7c0d00617ae0ba91b47ca7aac3b44a0bac22854f8cc1c9aa74e"}
Sep 17 15:56:21.170138 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:21.169861    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5976878f85-g6lhx"
Sep 17 15:56:21.216042 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:21.215989    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5976878f85-g6lhx" podStartSLOduration=1.373356074 podStartE2EDuration="4.215971075s" totalImagesPullingTime="2.842615001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:56:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:56:21.215348565 +0000 UTC m=+136.985680373" watchObservedRunningTime="2026-09-17 15:56:21.215971075 +0000 UTC m=+136.986302875"
Sep 17 15:56:21.244731 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:21.244692    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6594f4c75-c5p45"]
Sep 17 15:56:21.248008 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:21.247985    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-c5p45"
Sep 17 15:56:21.252497 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:21.252475    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-x4x4h\""
Sep 17 15:56:21.253513 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:21.253485    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 15:56:21.253644 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:21.253626    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 15:56:21.279101 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:21.279070    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-c5p45"]
Sep 17 15:56:21.308824 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:21.308792    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj96m\" (UniqueName: \"kubernetes.io/projected/eb2b5491-45b0-43b3-8882-e4cc21d2d52f-kube-api-access-rj96m\") pod \"downloads-6594f4c75-c5p45\" (UID: \"eb2b5491-45b0-43b3-8882-e4cc21d2d52f\") " pod="openshift-console/downloads-6594f4c75-c5p45"
Sep 17 15:56:21.410147 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:21.410104    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj96m\" (UniqueName: \"kubernetes.io/projected/eb2b5491-45b0-43b3-8882-e4cc21d2d52f-kube-api-access-rj96m\") pod \"downloads-6594f4c75-c5p45\" (UID: \"eb2b5491-45b0-43b3-8882-e4cc21d2d52f\") " pod="openshift-console/downloads-6594f4c75-c5p45"
Sep 17 15:56:21.421402 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:21.421329    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj96m\" (UniqueName: \"kubernetes.io/projected/eb2b5491-45b0-43b3-8882-e4cc21d2d52f-kube-api-access-rj96m\") pod \"downloads-6594f4c75-c5p45\" (UID: \"eb2b5491-45b0-43b3-8882-e4cc21d2d52f\") " pod="openshift-console/downloads-6594f4c75-c5p45"
Sep 17 15:56:21.557757 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:21.557708    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-c5p45"
Sep 17 15:56:21.838288 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:21.838250    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-c5p45"]
Sep 17 15:56:21.841249 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:56:21.841221    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb2b5491_45b0_43b3_8882_e4cc21d2d52f.slice/crio-bb492720e54b1759abbc2dc39496dee01eb4236b159d2c1f3ea38262131e4b94 WatchSource:0}: Error finding container bb492720e54b1759abbc2dc39496dee01eb4236b159d2c1f3ea38262131e4b94: Status 404 returned error can't find the container with id bb492720e54b1759abbc2dc39496dee01eb4236b159d2c1f3ea38262131e4b94
Sep 17 15:56:22.175080 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:22.174998    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-c5p45" event={"ID":"eb2b5491-45b0-43b3-8882-e4cc21d2d52f","Type":"ContainerStarted","Data":"bb492720e54b1759abbc2dc39496dee01eb4236b159d2c1f3ea38262131e4b94"}
Sep 17 15:56:26.086849 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:26.086820    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6ffcc66d5f-h4njt"
Sep 17 15:56:27.181946 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:27.181913    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5976878f85-g6lhx"
Sep 17 15:56:28.793582 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:28.793509    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-f65c965db-7f4nh" podUID="e0a58c2f-ea54-489b-86a9-d9c1d887b713" containerName="registry" containerID="cri-o://362e5e89ae4e22726307028bd2eb19f33f8c5739265d4669a17c427a5f6d5a69" gracePeriod=30
Sep 17 15:56:29.063001 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:29.062974    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f65c965db-7f4nh"
Sep 17 15:56:29.184106 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:29.184071    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28jtt\" (UniqueName: \"kubernetes.io/projected/e0a58c2f-ea54-489b-86a9-d9c1d887b713-kube-api-access-28jtt\") pod \"e0a58c2f-ea54-489b-86a9-d9c1d887b713\" (UID: \"e0a58c2f-ea54-489b-86a9-d9c1d887b713\") "
Sep 17 15:56:29.184288 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:29.184137    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e0a58c2f-ea54-489b-86a9-d9c1d887b713-registry-certificates\") pod \"e0a58c2f-ea54-489b-86a9-d9c1d887b713\" (UID: \"e0a58c2f-ea54-489b-86a9-d9c1d887b713\") "
Sep 17 15:56:29.184288 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:29.184184    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e0a58c2f-ea54-489b-86a9-d9c1d887b713-trusted-ca\") pod \"e0a58c2f-ea54-489b-86a9-d9c1d887b713\" (UID: \"e0a58c2f-ea54-489b-86a9-d9c1d887b713\") "
Sep 17 15:56:29.184288 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:29.184232    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e0a58c2f-ea54-489b-86a9-d9c1d887b713-installation-pull-secrets\") pod \"e0a58c2f-ea54-489b-86a9-d9c1d887b713\" (UID: \"e0a58c2f-ea54-489b-86a9-d9c1d887b713\") "
Sep 17 15:56:29.184288 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:29.184259    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e0a58c2f-ea54-489b-86a9-d9c1d887b713-image-registry-private-configuration\") pod \"e0a58c2f-ea54-489b-86a9-d9c1d887b713\" (UID: \"e0a58c2f-ea54-489b-86a9-d9c1d887b713\") "
Sep 17 15:56:29.185553 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:29.184309    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e0a58c2f-ea54-489b-86a9-d9c1d887b713-bound-sa-token\") pod \"e0a58c2f-ea54-489b-86a9-d9c1d887b713\" (UID: \"e0a58c2f-ea54-489b-86a9-d9c1d887b713\") "
Sep 17 15:56:29.185553 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:29.184348    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e0a58c2f-ea54-489b-86a9-d9c1d887b713-registry-tls\") pod \"e0a58c2f-ea54-489b-86a9-d9c1d887b713\" (UID: \"e0a58c2f-ea54-489b-86a9-d9c1d887b713\") "
Sep 17 15:56:29.185553 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:29.184387    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e0a58c2f-ea54-489b-86a9-d9c1d887b713-ca-trust-extracted\") pod \"e0a58c2f-ea54-489b-86a9-d9c1d887b713\" (UID: \"e0a58c2f-ea54-489b-86a9-d9c1d887b713\") "
Sep 17 15:56:29.185553 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:29.184612    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0a58c2f-ea54-489b-86a9-d9c1d887b713-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "e0a58c2f-ea54-489b-86a9-d9c1d887b713" (UID: "e0a58c2f-ea54-489b-86a9-d9c1d887b713"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:56:29.185553 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:29.184722    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0a58c2f-ea54-489b-86a9-d9c1d887b713-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "e0a58c2f-ea54-489b-86a9-d9c1d887b713" (UID: "e0a58c2f-ea54-489b-86a9-d9c1d887b713"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:56:29.187090 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:29.187055    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0a58c2f-ea54-489b-86a9-d9c1d887b713-kube-api-access-28jtt" (OuterVolumeSpecName: "kube-api-access-28jtt") pod "e0a58c2f-ea54-489b-86a9-d9c1d887b713" (UID: "e0a58c2f-ea54-489b-86a9-d9c1d887b713"). InnerVolumeSpecName "kube-api-access-28jtt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:56:29.187167 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:29.187111    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0a58c2f-ea54-489b-86a9-d9c1d887b713-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "e0a58c2f-ea54-489b-86a9-d9c1d887b713" (UID: "e0a58c2f-ea54-489b-86a9-d9c1d887b713"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:56:29.187240 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:29.187174    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0a58c2f-ea54-489b-86a9-d9c1d887b713-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "e0a58c2f-ea54-489b-86a9-d9c1d887b713" (UID: "e0a58c2f-ea54-489b-86a9-d9c1d887b713"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:56:29.188259 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:29.188157    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0a58c2f-ea54-489b-86a9-d9c1d887b713-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "e0a58c2f-ea54-489b-86a9-d9c1d887b713" (UID: "e0a58c2f-ea54-489b-86a9-d9c1d887b713"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:56:29.188369 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:29.188280    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0a58c2f-ea54-489b-86a9-d9c1d887b713-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "e0a58c2f-ea54-489b-86a9-d9c1d887b713" (UID: "e0a58c2f-ea54-489b-86a9-d9c1d887b713"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:56:29.196428 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:29.196393    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0a58c2f-ea54-489b-86a9-d9c1d887b713-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "e0a58c2f-ea54-489b-86a9-d9c1d887b713" (UID: "e0a58c2f-ea54-489b-86a9-d9c1d887b713"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:56:29.206771 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:29.206741    2585 generic.go:356] "Generic (PLEG): container finished" podID="e0a58c2f-ea54-489b-86a9-d9c1d887b713" containerID="362e5e89ae4e22726307028bd2eb19f33f8c5739265d4669a17c427a5f6d5a69" exitCode=0
Sep 17 15:56:29.206905 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:29.206791    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f65c965db-7f4nh"
Sep 17 15:56:29.206905 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:29.206824    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f65c965db-7f4nh" event={"ID":"e0a58c2f-ea54-489b-86a9-d9c1d887b713","Type":"ContainerDied","Data":"362e5e89ae4e22726307028bd2eb19f33f8c5739265d4669a17c427a5f6d5a69"}
Sep 17 15:56:29.206905 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:29.206891    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f65c965db-7f4nh" event={"ID":"e0a58c2f-ea54-489b-86a9-d9c1d887b713","Type":"ContainerDied","Data":"13ef5d356ff5a3c2c8091e5c9b524c087578929c49e28b79316301d9ffb98715"}
Sep 17 15:56:29.207061 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:29.206925    2585 scope.go:117] "RemoveContainer" containerID="362e5e89ae4e22726307028bd2eb19f33f8c5739265d4669a17c427a5f6d5a69"
Sep 17 15:56:29.217831 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:29.217807    2585 scope.go:117] "RemoveContainer" containerID="362e5e89ae4e22726307028bd2eb19f33f8c5739265d4669a17c427a5f6d5a69"
Sep 17 15:56:29.218328 ip-10-0-135-43 kubenswrapper[2585]: E0917 15:56:29.218305    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"362e5e89ae4e22726307028bd2eb19f33f8c5739265d4669a17c427a5f6d5a69\": container with ID starting with 362e5e89ae4e22726307028bd2eb19f33f8c5739265d4669a17c427a5f6d5a69 not found: ID does not exist" containerID="362e5e89ae4e22726307028bd2eb19f33f8c5739265d4669a17c427a5f6d5a69"
Sep 17 15:56:29.218454 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:29.218337    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"362e5e89ae4e22726307028bd2eb19f33f8c5739265d4669a17c427a5f6d5a69"} err="failed to get container status \"362e5e89ae4e22726307028bd2eb19f33f8c5739265d4669a17c427a5f6d5a69\": rpc error: code = NotFound desc = could not find container \"362e5e89ae4e22726307028bd2eb19f33f8c5739265d4669a17c427a5f6d5a69\": container with ID starting with 362e5e89ae4e22726307028bd2eb19f33f8c5739265d4669a17c427a5f6d5a69 not found: ID does not exist"
Sep 17 15:56:29.234037 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:29.233998    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-f65c965db-7f4nh"]
Sep 17 15:56:29.246645 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:29.246616    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-f65c965db-7f4nh"]
Sep 17 15:56:29.285252 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:29.285217    2585 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e0a58c2f-ea54-489b-86a9-d9c1d887b713-bound-sa-token\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 15:56:29.285252 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:29.285251    2585 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e0a58c2f-ea54-489b-86a9-d9c1d887b713-registry-tls\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 15:56:29.285589 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:29.285262    2585 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e0a58c2f-ea54-489b-86a9-d9c1d887b713-ca-trust-extracted\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 15:56:29.285589 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:29.285275    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28jtt\" (UniqueName: \"kubernetes.io/projected/e0a58c2f-ea54-489b-86a9-d9c1d887b713-kube-api-access-28jtt\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 15:56:29.285589 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:29.285290    2585 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e0a58c2f-ea54-489b-86a9-d9c1d887b713-registry-certificates\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 15:56:29.285589 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:29.285301    2585 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e0a58c2f-ea54-489b-86a9-d9c1d887b713-trusted-ca\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 15:56:29.285589 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:29.285311    2585 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e0a58c2f-ea54-489b-86a9-d9c1d887b713-installation-pull-secrets\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 15:56:29.285589 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:29.285323    2585 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e0a58c2f-ea54-489b-86a9-d9c1d887b713-image-registry-private-configuration\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 15:56:30.516157 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:30.516114    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0a58c2f-ea54-489b-86a9-d9c1d887b713" path="/var/lib/kubelet/pods/e0a58c2f-ea54-489b-86a9-d9c1d887b713/volumes"
Sep 17 15:56:33.416971 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:33.416929    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-65d5b9955c-xjdrr"]
Sep 17 15:56:33.417504 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:33.417352    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0a58c2f-ea54-489b-86a9-d9c1d887b713" containerName="registry"
Sep 17 15:56:33.417504 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:33.417374    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0a58c2f-ea54-489b-86a9-d9c1d887b713" containerName="registry"
Sep 17 15:56:33.417504 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:33.417471    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0a58c2f-ea54-489b-86a9-d9c1d887b713" containerName="registry"
Sep 17 15:56:33.422031 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:33.422003    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65d5b9955c-xjdrr"
Sep 17 15:56:33.426282 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:33.426256    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 15:56:33.426282 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:33.426270    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-824dx\""
Sep 17 15:56:33.426505 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:33.426313    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 15:56:33.426505 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:33.426323    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 15:56:33.427525 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:33.427501    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 15:56:33.427525 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:33.427525    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 15:56:33.434912 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:33.434843    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65d5b9955c-xjdrr"]
Sep 17 15:56:33.525348 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:33.525313    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9cda5c45-9f5a-4990-b618-ebab2514d750-console-config\") pod \"console-65d5b9955c-xjdrr\" (UID: \"9cda5c45-9f5a-4990-b618-ebab2514d750\") " pod="openshift-console/console-65d5b9955c-xjdrr"
Sep 17 15:56:33.525537 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:33.525373    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9cda5c45-9f5a-4990-b618-ebab2514d750-oauth-serving-cert\") pod \"console-65d5b9955c-xjdrr\" (UID: \"9cda5c45-9f5a-4990-b618-ebab2514d750\") " pod="openshift-console/console-65d5b9955c-xjdrr"
Sep 17 15:56:33.525537 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:33.525464    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9cda5c45-9f5a-4990-b618-ebab2514d750-console-oauth-config\") pod \"console-65d5b9955c-xjdrr\" (UID: \"9cda5c45-9f5a-4990-b618-ebab2514d750\") " pod="openshift-console/console-65d5b9955c-xjdrr"
Sep 17 15:56:33.525537 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:33.525490    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwjqb\" (UniqueName: \"kubernetes.io/projected/9cda5c45-9f5a-4990-b618-ebab2514d750-kube-api-access-hwjqb\") pod \"console-65d5b9955c-xjdrr\" (UID: \"9cda5c45-9f5a-4990-b618-ebab2514d750\") " pod="openshift-console/console-65d5b9955c-xjdrr"
Sep 17 15:56:33.525537 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:33.525524    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9cda5c45-9f5a-4990-b618-ebab2514d750-console-serving-cert\") pod \"console-65d5b9955c-xjdrr\" (UID: \"9cda5c45-9f5a-4990-b618-ebab2514d750\") " pod="openshift-console/console-65d5b9955c-xjdrr"
Sep 17 15:56:33.525725 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:33.525653    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9cda5c45-9f5a-4990-b618-ebab2514d750-service-ca\") pod \"console-65d5b9955c-xjdrr\" (UID: \"9cda5c45-9f5a-4990-b618-ebab2514d750\") " pod="openshift-console/console-65d5b9955c-xjdrr"
Sep 17 15:56:33.626767 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:33.626728    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9cda5c45-9f5a-4990-b618-ebab2514d750-service-ca\") pod \"console-65d5b9955c-xjdrr\" (UID: \"9cda5c45-9f5a-4990-b618-ebab2514d750\") " pod="openshift-console/console-65d5b9955c-xjdrr"
Sep 17 15:56:33.626953 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:33.626786    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9cda5c45-9f5a-4990-b618-ebab2514d750-console-config\") pod \"console-65d5b9955c-xjdrr\" (UID: \"9cda5c45-9f5a-4990-b618-ebab2514d750\") " pod="openshift-console/console-65d5b9955c-xjdrr"
Sep 17 15:56:33.626953 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:33.626827    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9cda5c45-9f5a-4990-b618-ebab2514d750-oauth-serving-cert\") pod \"console-65d5b9955c-xjdrr\" (UID: \"9cda5c45-9f5a-4990-b618-ebab2514d750\") " pod="openshift-console/console-65d5b9955c-xjdrr"
Sep 17 15:56:33.626953 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:33.626871    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9cda5c45-9f5a-4990-b618-ebab2514d750-console-oauth-config\") pod \"console-65d5b9955c-xjdrr\" (UID: \"9cda5c45-9f5a-4990-b618-ebab2514d750\") " pod="openshift-console/console-65d5b9955c-xjdrr"
Sep 17 15:56:33.626953 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:33.626891    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwjqb\" (UniqueName: \"kubernetes.io/projected/9cda5c45-9f5a-4990-b618-ebab2514d750-kube-api-access-hwjqb\") pod \"console-65d5b9955c-xjdrr\" (UID: \"9cda5c45-9f5a-4990-b618-ebab2514d750\") " pod="openshift-console/console-65d5b9955c-xjdrr"
Sep 17 15:56:33.627707 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:33.627252    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9cda5c45-9f5a-4990-b618-ebab2514d750-console-serving-cert\") pod \"console-65d5b9955c-xjdrr\" (UID: \"9cda5c45-9f5a-4990-b618-ebab2514d750\") " pod="openshift-console/console-65d5b9955c-xjdrr"
Sep 17 15:56:33.627707 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:33.627625    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9cda5c45-9f5a-4990-b618-ebab2514d750-service-ca\") pod \"console-65d5b9955c-xjdrr\" (UID: \"9cda5c45-9f5a-4990-b618-ebab2514d750\") " pod="openshift-console/console-65d5b9955c-xjdrr"
Sep 17 15:56:33.627707 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:33.627694    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9cda5c45-9f5a-4990-b618-ebab2514d750-console-config\") pod \"console-65d5b9955c-xjdrr\" (UID: \"9cda5c45-9f5a-4990-b618-ebab2514d750\") " pod="openshift-console/console-65d5b9955c-xjdrr"
Sep 17 15:56:33.627949 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:33.627844    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9cda5c45-9f5a-4990-b618-ebab2514d750-oauth-serving-cert\") pod \"console-65d5b9955c-xjdrr\" (UID: \"9cda5c45-9f5a-4990-b618-ebab2514d750\") " pod="openshift-console/console-65d5b9955c-xjdrr"
Sep 17 15:56:33.629869 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:33.629838    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9cda5c45-9f5a-4990-b618-ebab2514d750-console-oauth-config\") pod \"console-65d5b9955c-xjdrr\" (UID: \"9cda5c45-9f5a-4990-b618-ebab2514d750\") " pod="openshift-console/console-65d5b9955c-xjdrr"
Sep 17 15:56:33.630201 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:33.630179    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9cda5c45-9f5a-4990-b618-ebab2514d750-console-serving-cert\") pod \"console-65d5b9955c-xjdrr\" (UID: \"9cda5c45-9f5a-4990-b618-ebab2514d750\") " pod="openshift-console/console-65d5b9955c-xjdrr"
Sep 17 15:56:33.639452 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:33.639389    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwjqb\" (UniqueName: \"kubernetes.io/projected/9cda5c45-9f5a-4990-b618-ebab2514d750-kube-api-access-hwjqb\") pod \"console-65d5b9955c-xjdrr\" (UID: \"9cda5c45-9f5a-4990-b618-ebab2514d750\") " pod="openshift-console/console-65d5b9955c-xjdrr"
Sep 17 15:56:33.734535 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:33.734494    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65d5b9955c-xjdrr"
Sep 17 15:56:36.965387 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:36.965350    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6qpcq_1f3d6b60-625e-4015-942c-6c8150fa364f/serve-healthcheck-canary/0.log"
Sep 17 15:56:38.180650 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:38.180567    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65d5b9955c-xjdrr"]
Sep 17 15:56:38.187279 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:56:38.187242    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cda5c45_9f5a_4990_b618_ebab2514d750.slice/crio-8ff69d5b6a78dc9e2e0664daf264722f6ffca0c9c8b1dfaf6ac3702d1f3da273 WatchSource:0}: Error finding container 8ff69d5b6a78dc9e2e0664daf264722f6ffca0c9c8b1dfaf6ac3702d1f3da273: Status 404 returned error can't find the container with id 8ff69d5b6a78dc9e2e0664daf264722f6ffca0c9c8b1dfaf6ac3702d1f3da273
Sep 17 15:56:38.255661 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:38.255621    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-c5p45" event={"ID":"eb2b5491-45b0-43b3-8882-e4cc21d2d52f","Type":"ContainerStarted","Data":"6e137acce2aa5b04921f3993e7c54957645025066d5d21e96d268d3d67c7c5e0"}
Sep 17 15:56:38.255960 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:38.255937    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6594f4c75-c5p45"
Sep 17 15:56:38.257708 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:38.257633    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65d5b9955c-xjdrr" event={"ID":"9cda5c45-9f5a-4990-b618-ebab2514d750","Type":"ContainerStarted","Data":"8ff69d5b6a78dc9e2e0664daf264722f6ffca0c9c8b1dfaf6ac3702d1f3da273"}
Sep 17 15:56:38.262456 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:38.262434    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6594f4c75-c5p45"
Sep 17 15:56:38.280187 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:38.280111    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-6594f4c75-c5p45" podStartSLOduration=1.117216681 podStartE2EDuration="17.28008743s" totalImagesPullingTime="16.162870749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:56:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:56:38.279174035 +0000 UTC m=+154.049505832" watchObservedRunningTime="2026-09-17 15:56:38.28008743 +0000 UTC m=+154.050419228"
Sep 17 15:56:44.289836 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:44.289793    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65d5b9955c-xjdrr" event={"ID":"9cda5c45-9f5a-4990-b618-ebab2514d750","Type":"ContainerStarted","Data":"c63c2a8b75716effa340189fd86976d04c33666300f61766eb58e1b8f4d98c13"}
Sep 17 15:56:52.291148 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:52.291089    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-65d5b9955c-xjdrr" podStartSLOduration=13.347815839999999 podStartE2EDuration="19.291072314s" totalImagesPullingTime="5.943256474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:56:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:56:44.314879789 +0000 UTC m=+160.085211588" watchObservedRunningTime="2026-09-17 15:56:52.291072314 +0000 UTC m=+168.061404190"
Sep 17 15:56:52.291634 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:52.291547    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65d5b9955c-xjdrr"]
Sep 17 15:56:52.326126 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:52.326096    2585 generic.go:356] "Generic (PLEG): container finished" podID="84d8c186-5cbe-4c2e-9465-cc6729bf2b21" containerID="07130c1280f1861b4cd533a3778b7c1cce0296e0c791a58b793318490fca1a4b" exitCode=0
Sep 17 15:56:52.326303 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:52.326177    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-mb9mz" event={"ID":"84d8c186-5cbe-4c2e-9465-cc6729bf2b21","Type":"ContainerDied","Data":"07130c1280f1861b4cd533a3778b7c1cce0296e0c791a58b793318490fca1a4b"}
Sep 17 15:56:52.326571 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:52.326555    2585 scope.go:117] "RemoveContainer" containerID="07130c1280f1861b4cd533a3778b7c1cce0296e0c791a58b793318490fca1a4b"
Sep 17 15:56:53.331533 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:53.331495    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-mb9mz" event={"ID":"84d8c186-5cbe-4c2e-9465-cc6729bf2b21","Type":"ContainerStarted","Data":"d508fd78cc303dbaf69601e1cd1b464d86b6e1f5d92edcd8b0d3dbc0ea847ab7"}
Sep 17 15:56:53.735621 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:56:53.735586    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-65d5b9955c-xjdrr"
Sep 17 15:57:17.310602 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:57:17.310544    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-65d5b9955c-xjdrr" podUID="9cda5c45-9f5a-4990-b618-ebab2514d750" containerName="console" containerID="cri-o://c63c2a8b75716effa340189fd86976d04c33666300f61766eb58e1b8f4d98c13" gracePeriod=15
Sep 17 15:57:17.429291 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:57:17.429264    2585 generic.go:356] "Generic (PLEG): container finished" podID="9cda5c45-9f5a-4990-b618-ebab2514d750" containerID="c63c2a8b75716effa340189fd86976d04c33666300f61766eb58e1b8f4d98c13" exitCode=0
Sep 17 15:57:17.429454 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:57:17.429332    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65d5b9955c-xjdrr" event={"ID":"9cda5c45-9f5a-4990-b618-ebab2514d750","Type":"ContainerDied","Data":"c63c2a8b75716effa340189fd86976d04c33666300f61766eb58e1b8f4d98c13"}
Sep 17 15:57:17.591851 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:57:17.591794    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65d5b9955c-xjdrr"
Sep 17 15:57:17.633886 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:57:17.633837    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9cda5c45-9f5a-4990-b618-ebab2514d750-service-ca\") pod \"9cda5c45-9f5a-4990-b618-ebab2514d750\" (UID: \"9cda5c45-9f5a-4990-b618-ebab2514d750\") "
Sep 17 15:57:17.634068 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:57:17.633896    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9cda5c45-9f5a-4990-b618-ebab2514d750-console-serving-cert\") pod \"9cda5c45-9f5a-4990-b618-ebab2514d750\" (UID: \"9cda5c45-9f5a-4990-b618-ebab2514d750\") "
Sep 17 15:57:17.634068 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:57:17.633945    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9cda5c45-9f5a-4990-b618-ebab2514d750-oauth-serving-cert\") pod \"9cda5c45-9f5a-4990-b618-ebab2514d750\" (UID: \"9cda5c45-9f5a-4990-b618-ebab2514d750\") "
Sep 17 15:57:17.634068 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:57:17.634006    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9cda5c45-9f5a-4990-b618-ebab2514d750-console-config\") pod \"9cda5c45-9f5a-4990-b618-ebab2514d750\" (UID: \"9cda5c45-9f5a-4990-b618-ebab2514d750\") "
Sep 17 15:57:17.634068 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:57:17.634030    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9cda5c45-9f5a-4990-b618-ebab2514d750-console-oauth-config\") pod \"9cda5c45-9f5a-4990-b618-ebab2514d750\" (UID: \"9cda5c45-9f5a-4990-b618-ebab2514d750\") "
Sep 17 15:57:17.634068 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:57:17.634060    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwjqb\" (UniqueName: \"kubernetes.io/projected/9cda5c45-9f5a-4990-b618-ebab2514d750-kube-api-access-hwjqb\") pod \"9cda5c45-9f5a-4990-b618-ebab2514d750\" (UID: \"9cda5c45-9f5a-4990-b618-ebab2514d750\") "
Sep 17 15:57:17.634346 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:57:17.634317    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cda5c45-9f5a-4990-b618-ebab2514d750-service-ca" (OuterVolumeSpecName: "service-ca") pod "9cda5c45-9f5a-4990-b618-ebab2514d750" (UID: "9cda5c45-9f5a-4990-b618-ebab2514d750"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:57:17.634433 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:57:17.634340    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cda5c45-9f5a-4990-b618-ebab2514d750-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9cda5c45-9f5a-4990-b618-ebab2514d750" (UID: "9cda5c45-9f5a-4990-b618-ebab2514d750"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:57:17.634495 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:57:17.634472    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cda5c45-9f5a-4990-b618-ebab2514d750-console-config" (OuterVolumeSpecName: "console-config") pod "9cda5c45-9f5a-4990-b618-ebab2514d750" (UID: "9cda5c45-9f5a-4990-b618-ebab2514d750"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:57:17.636434 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:57:17.636383    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cda5c45-9f5a-4990-b618-ebab2514d750-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9cda5c45-9f5a-4990-b618-ebab2514d750" (UID: "9cda5c45-9f5a-4990-b618-ebab2514d750"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:57:17.636554 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:57:17.636476    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cda5c45-9f5a-4990-b618-ebab2514d750-kube-api-access-hwjqb" (OuterVolumeSpecName: "kube-api-access-hwjqb") pod "9cda5c45-9f5a-4990-b618-ebab2514d750" (UID: "9cda5c45-9f5a-4990-b618-ebab2514d750"). InnerVolumeSpecName "kube-api-access-hwjqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:57:17.636554 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:57:17.636477    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cda5c45-9f5a-4990-b618-ebab2514d750-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9cda5c45-9f5a-4990-b618-ebab2514d750" (UID: "9cda5c45-9f5a-4990-b618-ebab2514d750"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:57:17.734565 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:57:17.734537    2585 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9cda5c45-9f5a-4990-b618-ebab2514d750-service-ca\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 15:57:17.734565 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:57:17.734562    2585 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9cda5c45-9f5a-4990-b618-ebab2514d750-console-serving-cert\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 15:57:17.734565 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:57:17.734573    2585 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9cda5c45-9f5a-4990-b618-ebab2514d750-oauth-serving-cert\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 15:57:17.734774 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:57:17.734582    2585 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9cda5c45-9f5a-4990-b618-ebab2514d750-console-config\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 15:57:17.734774 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:57:17.734592    2585 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9cda5c45-9f5a-4990-b618-ebab2514d750-console-oauth-config\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 15:57:17.734774 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:57:17.734599    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwjqb\" (UniqueName: \"kubernetes.io/projected/9cda5c45-9f5a-4990-b618-ebab2514d750-kube-api-access-hwjqb\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 15:57:18.435147 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:57:18.435110    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65d5b9955c-xjdrr" event={"ID":"9cda5c45-9f5a-4990-b618-ebab2514d750","Type":"ContainerDied","Data":"8ff69d5b6a78dc9e2e0664daf264722f6ffca0c9c8b1dfaf6ac3702d1f3da273"}
Sep 17 15:57:18.435147 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:57:18.435128    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65d5b9955c-xjdrr"
Sep 17 15:57:18.435683 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:57:18.435172    2585 scope.go:117] "RemoveContainer" containerID="c63c2a8b75716effa340189fd86976d04c33666300f61766eb58e1b8f4d98c13"
Sep 17 15:57:18.457285 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:57:18.457256    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65d5b9955c-xjdrr"]
Sep 17 15:57:18.461033 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:57:18.461008    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-65d5b9955c-xjdrr"]
Sep 17 15:57:18.511842 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:57:18.511813    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9cda5c45-9f5a-4990-b618-ebab2514d750" path="/var/lib/kubelet/pods/9cda5c45-9f5a-4990-b618-ebab2514d750/volumes"
Sep 17 15:59:06.363712 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:59:06.363686    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2b47p_a1850a4d-4d01-4a71-8636-b5918f1426c2/ovn-acl-logging/0.log"
Sep 17 15:59:06.366323 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:59:06.366304    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2b47p_a1850a4d-4d01-4a71-8636-b5918f1426c2/ovn-acl-logging/0.log"
Sep 17 15:59:06.366549 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:59:06.366536    2585 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 15:59:21.204293 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:59:21.204258    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c88bfc66d-vl4gd"]
Sep 17 15:59:21.205729 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:59:21.204561    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cda5c45-9f5a-4990-b618-ebab2514d750" containerName="console"
Sep 17 15:59:21.205729 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:59:21.204572    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cda5c45-9f5a-4990-b618-ebab2514d750" containerName="console"
Sep 17 15:59:21.205729 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:59:21.204621    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cda5c45-9f5a-4990-b618-ebab2514d750" containerName="console"
Sep 17 15:59:21.206330 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:59:21.206315    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c88bfc66d-vl4gd"
Sep 17 15:59:21.209438 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:59:21.209400    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 15:59:21.210031 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:59:21.210015    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 15:59:21.214637 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:59:21.214616    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 15:59:21.214899 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:59:21.214880    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 15:59:21.215001 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:59:21.214881    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 15:59:21.215990 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:59:21.215973    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-824dx\""
Sep 17 15:59:21.217500 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:59:21.217459    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vdlc\" (UniqueName: \"kubernetes.io/projected/425875e1-5c2a-4d72-ae1a-b412f0b0f5c8-kube-api-access-6vdlc\") pod \"console-6c88bfc66d-vl4gd\" (UID: \"425875e1-5c2a-4d72-ae1a-b412f0b0f5c8\") " pod="openshift-console/console-6c88bfc66d-vl4gd"
Sep 17 15:59:21.217582 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:59:21.217529    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/425875e1-5c2a-4d72-ae1a-b412f0b0f5c8-service-ca\") pod \"console-6c88bfc66d-vl4gd\" (UID: \"425875e1-5c2a-4d72-ae1a-b412f0b0f5c8\") " pod="openshift-console/console-6c88bfc66d-vl4gd"
Sep 17 15:59:21.217582 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:59:21.217556    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/425875e1-5c2a-4d72-ae1a-b412f0b0f5c8-oauth-serving-cert\") pod \"console-6c88bfc66d-vl4gd\" (UID: \"425875e1-5c2a-4d72-ae1a-b412f0b0f5c8\") " pod="openshift-console/console-6c88bfc66d-vl4gd"
Sep 17 15:59:21.217684 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:59:21.217642    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/425875e1-5c2a-4d72-ae1a-b412f0b0f5c8-console-serving-cert\") pod \"console-6c88bfc66d-vl4gd\" (UID: \"425875e1-5c2a-4d72-ae1a-b412f0b0f5c8\") " pod="openshift-console/console-6c88bfc66d-vl4gd"
Sep 17 15:59:21.217726 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:59:21.217683    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/425875e1-5c2a-4d72-ae1a-b412f0b0f5c8-trusted-ca-bundle\") pod \"console-6c88bfc66d-vl4gd\" (UID: \"425875e1-5c2a-4d72-ae1a-b412f0b0f5c8\") " pod="openshift-console/console-6c88bfc66d-vl4gd"
Sep 17 15:59:21.217726 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:59:21.217718    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/425875e1-5c2a-4d72-ae1a-b412f0b0f5c8-console-oauth-config\") pod \"console-6c88bfc66d-vl4gd\" (UID: \"425875e1-5c2a-4d72-ae1a-b412f0b0f5c8\") " pod="openshift-console/console-6c88bfc66d-vl4gd"
Sep 17 15:59:21.217798 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:59:21.217781    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/425875e1-5c2a-4d72-ae1a-b412f0b0f5c8-console-config\") pod \"console-6c88bfc66d-vl4gd\" (UID: \"425875e1-5c2a-4d72-ae1a-b412f0b0f5c8\") " pod="openshift-console/console-6c88bfc66d-vl4gd"
Sep 17 15:59:21.222625 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:59:21.222605    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 15:59:21.227861 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:59:21.227841    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c88bfc66d-vl4gd"]
Sep 17 15:59:21.318440 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:59:21.318383    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vdlc\" (UniqueName: \"kubernetes.io/projected/425875e1-5c2a-4d72-ae1a-b412f0b0f5c8-kube-api-access-6vdlc\") pod \"console-6c88bfc66d-vl4gd\" (UID: \"425875e1-5c2a-4d72-ae1a-b412f0b0f5c8\") " pod="openshift-console/console-6c88bfc66d-vl4gd"
Sep 17 15:59:21.318440 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:59:21.318442    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/425875e1-5c2a-4d72-ae1a-b412f0b0f5c8-service-ca\") pod \"console-6c88bfc66d-vl4gd\" (UID: \"425875e1-5c2a-4d72-ae1a-b412f0b0f5c8\") " pod="openshift-console/console-6c88bfc66d-vl4gd"
Sep 17 15:59:21.318697 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:59:21.318458    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/425875e1-5c2a-4d72-ae1a-b412f0b0f5c8-oauth-serving-cert\") pod \"console-6c88bfc66d-vl4gd\" (UID: \"425875e1-5c2a-4d72-ae1a-b412f0b0f5c8\") " pod="openshift-console/console-6c88bfc66d-vl4gd"
Sep 17 15:59:21.318697 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:59:21.318623    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/425875e1-5c2a-4d72-ae1a-b412f0b0f5c8-console-serving-cert\") pod \"console-6c88bfc66d-vl4gd\" (UID: \"425875e1-5c2a-4d72-ae1a-b412f0b0f5c8\") " pod="openshift-console/console-6c88bfc66d-vl4gd"
Sep 17 15:59:21.318697 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:59:21.318651    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/425875e1-5c2a-4d72-ae1a-b412f0b0f5c8-trusted-ca-bundle\") pod \"console-6c88bfc66d-vl4gd\" (UID: \"425875e1-5c2a-4d72-ae1a-b412f0b0f5c8\") " pod="openshift-console/console-6c88bfc66d-vl4gd"
Sep 17 15:59:21.318697 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:59:21.318693    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/425875e1-5c2a-4d72-ae1a-b412f0b0f5c8-console-oauth-config\") pod \"console-6c88bfc66d-vl4gd\" (UID: \"425875e1-5c2a-4d72-ae1a-b412f0b0f5c8\") " pod="openshift-console/console-6c88bfc66d-vl4gd"
Sep 17 15:59:21.318901 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:59:21.318769    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/425875e1-5c2a-4d72-ae1a-b412f0b0f5c8-console-config\") pod \"console-6c88bfc66d-vl4gd\" (UID: \"425875e1-5c2a-4d72-ae1a-b412f0b0f5c8\") " pod="openshift-console/console-6c88bfc66d-vl4gd"
Sep 17 15:59:21.319265 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:59:21.319230    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/425875e1-5c2a-4d72-ae1a-b412f0b0f5c8-service-ca\") pod \"console-6c88bfc66d-vl4gd\" (UID: \"425875e1-5c2a-4d72-ae1a-b412f0b0f5c8\") " pod="openshift-console/console-6c88bfc66d-vl4gd"
Sep 17 15:59:21.319407 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:59:21.319233    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/425875e1-5c2a-4d72-ae1a-b412f0b0f5c8-oauth-serving-cert\") pod \"console-6c88bfc66d-vl4gd\" (UID: \"425875e1-5c2a-4d72-ae1a-b412f0b0f5c8\") " pod="openshift-console/console-6c88bfc66d-vl4gd"
Sep 17 15:59:21.319407 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:59:21.319368    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/425875e1-5c2a-4d72-ae1a-b412f0b0f5c8-console-config\") pod \"console-6c88bfc66d-vl4gd\" (UID: \"425875e1-5c2a-4d72-ae1a-b412f0b0f5c8\") " pod="openshift-console/console-6c88bfc66d-vl4gd"
Sep 17 15:59:21.319603 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:59:21.319583    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/425875e1-5c2a-4d72-ae1a-b412f0b0f5c8-trusted-ca-bundle\") pod \"console-6c88bfc66d-vl4gd\" (UID: \"425875e1-5c2a-4d72-ae1a-b412f0b0f5c8\") " pod="openshift-console/console-6c88bfc66d-vl4gd"
Sep 17 15:59:21.321060 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:59:21.321030    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/425875e1-5c2a-4d72-ae1a-b412f0b0f5c8-console-serving-cert\") pod \"console-6c88bfc66d-vl4gd\" (UID: \"425875e1-5c2a-4d72-ae1a-b412f0b0f5c8\") " pod="openshift-console/console-6c88bfc66d-vl4gd"
Sep 17 15:59:21.321151 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:59:21.321060    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/425875e1-5c2a-4d72-ae1a-b412f0b0f5c8-console-oauth-config\") pod \"console-6c88bfc66d-vl4gd\" (UID: \"425875e1-5c2a-4d72-ae1a-b412f0b0f5c8\") " pod="openshift-console/console-6c88bfc66d-vl4gd"
Sep 17 15:59:21.326411 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:59:21.326387    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vdlc\" (UniqueName: \"kubernetes.io/projected/425875e1-5c2a-4d72-ae1a-b412f0b0f5c8-kube-api-access-6vdlc\") pod \"console-6c88bfc66d-vl4gd\" (UID: \"425875e1-5c2a-4d72-ae1a-b412f0b0f5c8\") " pod="openshift-console/console-6c88bfc66d-vl4gd"
Sep 17 15:59:21.516465 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:59:21.516350    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c88bfc66d-vl4gd"
Sep 17 15:59:21.791826 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:59:21.791748    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c88bfc66d-vl4gd"]
Sep 17 15:59:21.794774 ip-10-0-135-43 kubenswrapper[2585]: W0917 15:59:21.794747    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod425875e1_5c2a_4d72_ae1a_b412f0b0f5c8.slice/crio-4c6389dc04f8da8f12c5be42aab0f30eb67fa14b5f2938aea64501821081b980 WatchSource:0}: Error finding container 4c6389dc04f8da8f12c5be42aab0f30eb67fa14b5f2938aea64501821081b980: Status 404 returned error can't find the container with id 4c6389dc04f8da8f12c5be42aab0f30eb67fa14b5f2938aea64501821081b980
Sep 17 15:59:21.893979 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:59:21.893936    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c88bfc66d-vl4gd" event={"ID":"425875e1-5c2a-4d72-ae1a-b412f0b0f5c8","Type":"ContainerStarted","Data":"f64e0cd2493ccf2ea1b67ed081d06a1991328fd3ab89db66b68879cd05062d6e"}
Sep 17 15:59:21.894127 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:59:21.893994    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c88bfc66d-vl4gd" event={"ID":"425875e1-5c2a-4d72-ae1a-b412f0b0f5c8","Type":"ContainerStarted","Data":"4c6389dc04f8da8f12c5be42aab0f30eb67fa14b5f2938aea64501821081b980"}
Sep 17 15:59:21.915732 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:59:21.915683    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6c88bfc66d-vl4gd" podStartSLOduration=0.915666162 podStartE2EDuration="915.666162ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:59:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:59:21.914568836 +0000 UTC m=+317.684900649" watchObservedRunningTime="2026-09-17 15:59:21.915666162 +0000 UTC m=+317.685997960"
Sep 17 15:59:31.517020 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:59:31.516973    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c88bfc66d-vl4gd"
Sep 17 15:59:31.517408 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:59:31.517064    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6c88bfc66d-vl4gd"
Sep 17 15:59:31.521983 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:59:31.521954    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6c88bfc66d-vl4gd"
Sep 17 15:59:31.935073 ip-10-0-135-43 kubenswrapper[2585]: I0917 15:59:31.934988    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c88bfc66d-vl4gd"
Sep 17 16:00:42.789993 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:00:42.789952    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awskwb"]
Sep 17 16:00:42.792623 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:00:42.792604    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awskwb"
Sep 17 16:00:42.795173 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:00:42.795155    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 16:00:42.796263 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:00:42.796242    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 16:00:42.796340 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:00:42.796252    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-tdljp\""
Sep 17 16:00:42.804221 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:00:42.804194    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awskwb"]
Sep 17 16:00:42.833799 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:00:42.833761    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8becaa43-d8d8-44e4-b395-2e5a251fd230-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awskwb\" (UID: \"8becaa43-d8d8-44e4-b395-2e5a251fd230\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awskwb"
Sep 17 16:00:42.833971 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:00:42.833820    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9xnm\" (UniqueName: \"kubernetes.io/projected/8becaa43-d8d8-44e4-b395-2e5a251fd230-kube-api-access-g9xnm\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awskwb\" (UID: \"8becaa43-d8d8-44e4-b395-2e5a251fd230\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awskwb"
Sep 17 16:00:42.833971 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:00:42.833856    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8becaa43-d8d8-44e4-b395-2e5a251fd230-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awskwb\" (UID: \"8becaa43-d8d8-44e4-b395-2e5a251fd230\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awskwb"
Sep 17 16:00:42.934174 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:00:42.934136    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9xnm\" (UniqueName: \"kubernetes.io/projected/8becaa43-d8d8-44e4-b395-2e5a251fd230-kube-api-access-g9xnm\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awskwb\" (UID: \"8becaa43-d8d8-44e4-b395-2e5a251fd230\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awskwb"
Sep 17 16:00:42.934174 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:00:42.934179    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8becaa43-d8d8-44e4-b395-2e5a251fd230-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awskwb\" (UID: \"8becaa43-d8d8-44e4-b395-2e5a251fd230\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awskwb"
Sep 17 16:00:42.934370 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:00:42.934208    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8becaa43-d8d8-44e4-b395-2e5a251fd230-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awskwb\" (UID: \"8becaa43-d8d8-44e4-b395-2e5a251fd230\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awskwb"
Sep 17 16:00:42.934607 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:00:42.934588    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8becaa43-d8d8-44e4-b395-2e5a251fd230-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awskwb\" (UID: \"8becaa43-d8d8-44e4-b395-2e5a251fd230\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awskwb"
Sep 17 16:00:42.934656 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:00:42.934627    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8becaa43-d8d8-44e4-b395-2e5a251fd230-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awskwb\" (UID: \"8becaa43-d8d8-44e4-b395-2e5a251fd230\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awskwb"
Sep 17 16:00:42.942825 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:00:42.942794    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9xnm\" (UniqueName: \"kubernetes.io/projected/8becaa43-d8d8-44e4-b395-2e5a251fd230-kube-api-access-g9xnm\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awskwb\" (UID: \"8becaa43-d8d8-44e4-b395-2e5a251fd230\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awskwb"
Sep 17 16:00:43.101390 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:00:43.101305    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awskwb"
Sep 17 16:00:43.375133 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:00:43.375011    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awskwb"]
Sep 17 16:00:43.377991 ip-10-0-135-43 kubenswrapper[2585]: W0917 16:00:43.377962    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8becaa43_d8d8_44e4_b395_2e5a251fd230.slice/crio-f9bcaad5b8de5e9b21d2ccf3ca007f07b8087843691d9e2ae5b07aaede77b383 WatchSource:0}: Error finding container f9bcaad5b8de5e9b21d2ccf3ca007f07b8087843691d9e2ae5b07aaede77b383: Status 404 returned error can't find the container with id f9bcaad5b8de5e9b21d2ccf3ca007f07b8087843691d9e2ae5b07aaede77b383
Sep 17 16:00:43.379981 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:00:43.379963    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 16:00:44.201283 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:00:44.201241    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awskwb" event={"ID":"8becaa43-d8d8-44e4-b395-2e5a251fd230","Type":"ContainerStarted","Data":"f9bcaad5b8de5e9b21d2ccf3ca007f07b8087843691d9e2ae5b07aaede77b383"}
Sep 17 16:00:50.226309 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:00:50.226272    2585 generic.go:356] "Generic (PLEG): container finished" podID="8becaa43-d8d8-44e4-b395-2e5a251fd230" containerID="c5af14b1f257f4198d6b6a8dd6f3b7edeb39d79f3ac36dc12299eb38b8454a29" exitCode=0
Sep 17 16:00:50.226820 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:00:50.226328    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awskwb" event={"ID":"8becaa43-d8d8-44e4-b395-2e5a251fd230","Type":"ContainerDied","Data":"c5af14b1f257f4198d6b6a8dd6f3b7edeb39d79f3ac36dc12299eb38b8454a29"}
Sep 17 16:01:01.267757 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:01.267725    2585 generic.go:356] "Generic (PLEG): container finished" podID="8becaa43-d8d8-44e4-b395-2e5a251fd230" containerID="9f8ce55cf6cbc1f4cf70fdeb5d12035e10f512260bf54493c927d27cd85021d7" exitCode=0
Sep 17 16:01:01.268120 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:01.267814    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awskwb" event={"ID":"8becaa43-d8d8-44e4-b395-2e5a251fd230","Type":"ContainerDied","Data":"9f8ce55cf6cbc1f4cf70fdeb5d12035e10f512260bf54493c927d27cd85021d7"}
Sep 17 16:01:08.305741 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:08.305701    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awskwb" event={"ID":"8becaa43-d8d8-44e4-b395-2e5a251fd230","Type":"ContainerDied","Data":"deb4f7d91a25a2fd999a01e422239d6f8ba9775ec1bfb03f3280e8f6bbd27620"}
Sep 17 16:01:08.306160 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:08.305757    2585 generic.go:356] "Generic (PLEG): container finished" podID="8becaa43-d8d8-44e4-b395-2e5a251fd230" containerID="deb4f7d91a25a2fd999a01e422239d6f8ba9775ec1bfb03f3280e8f6bbd27620" exitCode=0
Sep 17 16:01:09.427090 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:09.427063    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awskwb"
Sep 17 16:01:09.540786 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:09.540753    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8becaa43-d8d8-44e4-b395-2e5a251fd230-util\") pod \"8becaa43-d8d8-44e4-b395-2e5a251fd230\" (UID: \"8becaa43-d8d8-44e4-b395-2e5a251fd230\") "
Sep 17 16:01:09.540786 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:09.540793    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9xnm\" (UniqueName: \"kubernetes.io/projected/8becaa43-d8d8-44e4-b395-2e5a251fd230-kube-api-access-g9xnm\") pod \"8becaa43-d8d8-44e4-b395-2e5a251fd230\" (UID: \"8becaa43-d8d8-44e4-b395-2e5a251fd230\") "
Sep 17 16:01:09.541047 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:09.540825    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8becaa43-d8d8-44e4-b395-2e5a251fd230-bundle\") pod \"8becaa43-d8d8-44e4-b395-2e5a251fd230\" (UID: \"8becaa43-d8d8-44e4-b395-2e5a251fd230\") "
Sep 17 16:01:09.541516 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:09.541486    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8becaa43-d8d8-44e4-b395-2e5a251fd230-bundle" (OuterVolumeSpecName: "bundle") pod "8becaa43-d8d8-44e4-b395-2e5a251fd230" (UID: "8becaa43-d8d8-44e4-b395-2e5a251fd230"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:01:09.543075 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:09.543050    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8becaa43-d8d8-44e4-b395-2e5a251fd230-kube-api-access-g9xnm" (OuterVolumeSpecName: "kube-api-access-g9xnm") pod "8becaa43-d8d8-44e4-b395-2e5a251fd230" (UID: "8becaa43-d8d8-44e4-b395-2e5a251fd230"). InnerVolumeSpecName "kube-api-access-g9xnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:01:09.545857 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:09.545837    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8becaa43-d8d8-44e4-b395-2e5a251fd230-util" (OuterVolumeSpecName: "util") pod "8becaa43-d8d8-44e4-b395-2e5a251fd230" (UID: "8becaa43-d8d8-44e4-b395-2e5a251fd230"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:01:09.642267 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:09.642186    2585 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8becaa43-d8d8-44e4-b395-2e5a251fd230-bundle\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:01:09.642267 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:09.642210    2585 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8becaa43-d8d8-44e4-b395-2e5a251fd230-util\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:01:09.642267 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:09.642219    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9xnm\" (UniqueName: \"kubernetes.io/projected/8becaa43-d8d8-44e4-b395-2e5a251fd230-kube-api-access-g9xnm\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:01:10.314541 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:10.314507    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awskwb" event={"ID":"8becaa43-d8d8-44e4-b395-2e5a251fd230","Type":"ContainerDied","Data":"f9bcaad5b8de5e9b21d2ccf3ca007f07b8087843691d9e2ae5b07aaede77b383"}
Sep 17 16:01:10.314732 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:10.314545    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1awskwb"
Sep 17 16:01:10.314732 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:10.314554    2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9bcaad5b8de5e9b21d2ccf3ca007f07b8087843691d9e2ae5b07aaede77b383"
Sep 17 16:01:16.042233 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:16.042198    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-l9jnt"]
Sep 17 16:01:16.042739 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:16.042522    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8becaa43-d8d8-44e4-b395-2e5a251fd230" containerName="pull"
Sep 17 16:01:16.042739 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:16.042534    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="8becaa43-d8d8-44e4-b395-2e5a251fd230" containerName="pull"
Sep 17 16:01:16.042739 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:16.042544    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8becaa43-d8d8-44e4-b395-2e5a251fd230" containerName="extract"
Sep 17 16:01:16.042739 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:16.042549    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="8becaa43-d8d8-44e4-b395-2e5a251fd230" containerName="extract"
Sep 17 16:01:16.042739 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:16.042562    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8becaa43-d8d8-44e4-b395-2e5a251fd230" containerName="util"
Sep 17 16:01:16.042739 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:16.042568    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="8becaa43-d8d8-44e4-b395-2e5a251fd230" containerName="util"
Sep 17 16:01:16.042739 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:16.042621    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="8becaa43-d8d8-44e4-b395-2e5a251fd230" containerName="extract"
Sep 17 16:01:16.044583 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:16.044565    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-l9jnt"
Sep 17 16:01:16.047530 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:16.047501    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 17 16:01:16.047679 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:16.047543    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 17 16:01:16.047679 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:16.047647    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 17 16:01:16.047801 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:16.047771    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-d49x5\""
Sep 17 16:01:16.062199 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:16.062172    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-l9jnt"]
Sep 17 16:01:16.098068 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:16.098032    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/b5469287-4b5d-4d02-a4e5-66b7d12c0302-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-l9jnt\" (UID: \"b5469287-4b5d-4d02-a4e5-66b7d12c0302\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-l9jnt"
Sep 17 16:01:16.098233 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:16.098094    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6ksq\" (UniqueName: \"kubernetes.io/projected/b5469287-4b5d-4d02-a4e5-66b7d12c0302-kube-api-access-x6ksq\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-l9jnt\" (UID: \"b5469287-4b5d-4d02-a4e5-66b7d12c0302\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-l9jnt"
Sep 17 16:01:16.198985 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:16.198951    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/b5469287-4b5d-4d02-a4e5-66b7d12c0302-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-l9jnt\" (UID: \"b5469287-4b5d-4d02-a4e5-66b7d12c0302\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-l9jnt"
Sep 17 16:01:16.199162 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:16.199006    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6ksq\" (UniqueName: \"kubernetes.io/projected/b5469287-4b5d-4d02-a4e5-66b7d12c0302-kube-api-access-x6ksq\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-l9jnt\" (UID: \"b5469287-4b5d-4d02-a4e5-66b7d12c0302\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-l9jnt"
Sep 17 16:01:16.201708 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:16.201681    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/b5469287-4b5d-4d02-a4e5-66b7d12c0302-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-l9jnt\" (UID: \"b5469287-4b5d-4d02-a4e5-66b7d12c0302\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-l9jnt"
Sep 17 16:01:16.223739 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:16.223709    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6ksq\" (UniqueName: \"kubernetes.io/projected/b5469287-4b5d-4d02-a4e5-66b7d12c0302-kube-api-access-x6ksq\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-l9jnt\" (UID: \"b5469287-4b5d-4d02-a4e5-66b7d12c0302\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-l9jnt"
Sep 17 16:01:16.355009 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:16.354925    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-l9jnt"
Sep 17 16:01:16.632565 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:16.632539    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-l9jnt"]
Sep 17 16:01:16.635381 ip-10-0-135-43 kubenswrapper[2585]: W0917 16:01:16.635351    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5469287_4b5d_4d02_a4e5_66b7d12c0302.slice/crio-4907a8b3a050248d810daa07223021775bc53cf4675a882b6d71f58c48e17088 WatchSource:0}: Error finding container 4907a8b3a050248d810daa07223021775bc53cf4675a882b6d71f58c48e17088: Status 404 returned error can't find the container with id 4907a8b3a050248d810daa07223021775bc53cf4675a882b6d71f58c48e17088
Sep 17 16:01:17.342416 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:17.342380    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-l9jnt" event={"ID":"b5469287-4b5d-4d02-a4e5-66b7d12c0302","Type":"ContainerStarted","Data":"4907a8b3a050248d810daa07223021775bc53cf4675a882b6d71f58c48e17088"}
Sep 17 16:01:32.411343 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:32.411249    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-l9jnt" event={"ID":"b5469287-4b5d-4d02-a4e5-66b7d12c0302","Type":"ContainerStarted","Data":"193af8665bc7edcf926a6a32066360c62f669f4d01d035de37a296c4395a4e47"}
Sep 17 16:01:32.411343 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:32.411328    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-l9jnt"
Sep 17 16:01:32.441042 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:32.437981    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-l9jnt" podStartSLOduration=0.978351993 podStartE2EDuration="16.437962171s" totalImagesPullingTime="15.459610178s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:01:32.436126568 +0000 UTC m=+448.206458367" watchObservedRunningTime="2026-09-17 16:01:32.437962171 +0000 UTC m=+448.208293971"
Sep 17 16:01:33.778084 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:33.778049    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-5n24p"]
Sep 17 16:01:33.782577 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:33.782554    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-5n24p"
Sep 17 16:01:33.786305 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:33.786275    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-qsmg9\""
Sep 17 16:01:33.786516 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:33.786302    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 17 16:01:33.786516 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:33.786283    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 17 16:01:33.791559 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:33.791538    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-5n24p"]
Sep 17 16:01:33.960589 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:33.960548    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4sgr\" (UniqueName: \"kubernetes.io/projected/8ecd76d5-070a-4625-ad28-fdf53b6d1c99-kube-api-access-w4sgr\") pod \"keda-admission-6c4d864b86-5n24p\" (UID: \"8ecd76d5-070a-4625-ad28-fdf53b6d1c99\") " pod="openshift-keda/keda-admission-6c4d864b86-5n24p"
Sep 17 16:01:33.960589 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:33.960588    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8ecd76d5-070a-4625-ad28-fdf53b6d1c99-certificates\") pod \"keda-admission-6c4d864b86-5n24p\" (UID: \"8ecd76d5-070a-4625-ad28-fdf53b6d1c99\") " pod="openshift-keda/keda-admission-6c4d864b86-5n24p"
Sep 17 16:01:34.061587 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:34.061503    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4sgr\" (UniqueName: \"kubernetes.io/projected/8ecd76d5-070a-4625-ad28-fdf53b6d1c99-kube-api-access-w4sgr\") pod \"keda-admission-6c4d864b86-5n24p\" (UID: \"8ecd76d5-070a-4625-ad28-fdf53b6d1c99\") " pod="openshift-keda/keda-admission-6c4d864b86-5n24p"
Sep 17 16:01:34.061587 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:34.061544    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8ecd76d5-070a-4625-ad28-fdf53b6d1c99-certificates\") pod \"keda-admission-6c4d864b86-5n24p\" (UID: \"8ecd76d5-070a-4625-ad28-fdf53b6d1c99\") " pod="openshift-keda/keda-admission-6c4d864b86-5n24p"
Sep 17 16:01:34.064229 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:34.064196    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8ecd76d5-070a-4625-ad28-fdf53b6d1c99-certificates\") pod \"keda-admission-6c4d864b86-5n24p\" (UID: \"8ecd76d5-070a-4625-ad28-fdf53b6d1c99\") " pod="openshift-keda/keda-admission-6c4d864b86-5n24p"
Sep 17 16:01:34.070066 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:34.070030    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4sgr\" (UniqueName: \"kubernetes.io/projected/8ecd76d5-070a-4625-ad28-fdf53b6d1c99-kube-api-access-w4sgr\") pod \"keda-admission-6c4d864b86-5n24p\" (UID: \"8ecd76d5-070a-4625-ad28-fdf53b6d1c99\") " pod="openshift-keda/keda-admission-6c4d864b86-5n24p"
Sep 17 16:01:34.093925 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:34.093890    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-5n24p"
Sep 17 16:01:34.377005 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:34.376977    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-5n24p"]
Sep 17 16:01:34.379562 ip-10-0-135-43 kubenswrapper[2585]: W0917 16:01:34.379531    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ecd76d5_070a_4625_ad28_fdf53b6d1c99.slice/crio-f0778816ddb95fbf32839cf110c5961b52212419d9faaba72caf5f9a0b87bdc4 WatchSource:0}: Error finding container f0778816ddb95fbf32839cf110c5961b52212419d9faaba72caf5f9a0b87bdc4: Status 404 returned error can't find the container with id f0778816ddb95fbf32839cf110c5961b52212419d9faaba72caf5f9a0b87bdc4
Sep 17 16:01:34.422161 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:34.422120    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-5n24p" event={"ID":"8ecd76d5-070a-4625-ad28-fdf53b6d1c99","Type":"ContainerStarted","Data":"f0778816ddb95fbf32839cf110c5961b52212419d9faaba72caf5f9a0b87bdc4"}
Sep 17 16:01:36.431450 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:36.431403    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-5n24p" event={"ID":"8ecd76d5-070a-4625-ad28-fdf53b6d1c99","Type":"ContainerStarted","Data":"ee3806b7d43228c5414562d505cdcd52cb1a78903e26ba124da9ce0b667184f3"}
Sep 17 16:01:36.431779 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:36.431555    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-5n24p"
Sep 17 16:01:36.450702 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:36.450647    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-5n24p" podStartSLOduration=1.463473838 podStartE2EDuration="3.450625826s" totalImagesPullingTime="1.987151988s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:01:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:01:36.450187537 +0000 UTC m=+452.220519336" watchObservedRunningTime="2026-09-17 16:01:36.450625826 +0000 UTC m=+452.220957624"
Sep 17 16:01:45.240641 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:45.240596    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c88bfc66d-vl4gd"]
Sep 17 16:01:53.418534 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:53.418499    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-l9jnt"
Sep 17 16:01:57.438614 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:01:57.438577    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-5n24p"
Sep 17 16:02:10.259617 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:10.259562    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6c88bfc66d-vl4gd" podUID="425875e1-5c2a-4d72-ae1a-b412f0b0f5c8" containerName="console" containerID="cri-o://f64e0cd2493ccf2ea1b67ed081d06a1991328fd3ab89db66b68879cd05062d6e" gracePeriod=15
Sep 17 16:02:10.493856 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:10.493832    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c88bfc66d-vl4gd"
Sep 17 16:02:10.562299 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:10.562207    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/425875e1-5c2a-4d72-ae1a-b412f0b0f5c8-console-serving-cert\") pod \"425875e1-5c2a-4d72-ae1a-b412f0b0f5c8\" (UID: \"425875e1-5c2a-4d72-ae1a-b412f0b0f5c8\") "
Sep 17 16:02:10.562299 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:10.562256    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/425875e1-5c2a-4d72-ae1a-b412f0b0f5c8-trusted-ca-bundle\") pod \"425875e1-5c2a-4d72-ae1a-b412f0b0f5c8\" (UID: \"425875e1-5c2a-4d72-ae1a-b412f0b0f5c8\") "
Sep 17 16:02:10.562299 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:10.562298    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/425875e1-5c2a-4d72-ae1a-b412f0b0f5c8-console-oauth-config\") pod \"425875e1-5c2a-4d72-ae1a-b412f0b0f5c8\" (UID: \"425875e1-5c2a-4d72-ae1a-b412f0b0f5c8\") "
Sep 17 16:02:10.562604 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:10.562320    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/425875e1-5c2a-4d72-ae1a-b412f0b0f5c8-service-ca\") pod \"425875e1-5c2a-4d72-ae1a-b412f0b0f5c8\" (UID: \"425875e1-5c2a-4d72-ae1a-b412f0b0f5c8\") "
Sep 17 16:02:10.562604 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:10.562344    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/425875e1-5c2a-4d72-ae1a-b412f0b0f5c8-console-config\") pod \"425875e1-5c2a-4d72-ae1a-b412f0b0f5c8\" (UID: \"425875e1-5c2a-4d72-ae1a-b412f0b0f5c8\") "
Sep 17 16:02:10.562604 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:10.562374    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/425875e1-5c2a-4d72-ae1a-b412f0b0f5c8-oauth-serving-cert\") pod \"425875e1-5c2a-4d72-ae1a-b412f0b0f5c8\" (UID: \"425875e1-5c2a-4d72-ae1a-b412f0b0f5c8\") "
Sep 17 16:02:10.562604 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:10.562501    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vdlc\" (UniqueName: \"kubernetes.io/projected/425875e1-5c2a-4d72-ae1a-b412f0b0f5c8-kube-api-access-6vdlc\") pod \"425875e1-5c2a-4d72-ae1a-b412f0b0f5c8\" (UID: \"425875e1-5c2a-4d72-ae1a-b412f0b0f5c8\") "
Sep 17 16:02:10.562858 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:10.562834    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/425875e1-5c2a-4d72-ae1a-b412f0b0f5c8-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "425875e1-5c2a-4d72-ae1a-b412f0b0f5c8" (UID: "425875e1-5c2a-4d72-ae1a-b412f0b0f5c8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:02:10.562858 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:10.562844    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/425875e1-5c2a-4d72-ae1a-b412f0b0f5c8-service-ca" (OuterVolumeSpecName: "service-ca") pod "425875e1-5c2a-4d72-ae1a-b412f0b0f5c8" (UID: "425875e1-5c2a-4d72-ae1a-b412f0b0f5c8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:02:10.562978 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:10.562845    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/425875e1-5c2a-4d72-ae1a-b412f0b0f5c8-console-config" (OuterVolumeSpecName: "console-config") pod "425875e1-5c2a-4d72-ae1a-b412f0b0f5c8" (UID: "425875e1-5c2a-4d72-ae1a-b412f0b0f5c8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:02:10.562978 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:10.562922    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/425875e1-5c2a-4d72-ae1a-b412f0b0f5c8-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "425875e1-5c2a-4d72-ae1a-b412f0b0f5c8" (UID: "425875e1-5c2a-4d72-ae1a-b412f0b0f5c8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:02:10.564570 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:10.564540    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/425875e1-5c2a-4d72-ae1a-b412f0b0f5c8-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "425875e1-5c2a-4d72-ae1a-b412f0b0f5c8" (UID: "425875e1-5c2a-4d72-ae1a-b412f0b0f5c8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:02:10.564570 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:10.564552    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/425875e1-5c2a-4d72-ae1a-b412f0b0f5c8-kube-api-access-6vdlc" (OuterVolumeSpecName: "kube-api-access-6vdlc") pod "425875e1-5c2a-4d72-ae1a-b412f0b0f5c8" (UID: "425875e1-5c2a-4d72-ae1a-b412f0b0f5c8"). InnerVolumeSpecName "kube-api-access-6vdlc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:02:10.564570 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:10.564552    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/425875e1-5c2a-4d72-ae1a-b412f0b0f5c8-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "425875e1-5c2a-4d72-ae1a-b412f0b0f5c8" (UID: "425875e1-5c2a-4d72-ae1a-b412f0b0f5c8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:02:10.569828 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:10.569802    2585 generic.go:356] "Generic (PLEG): container finished" podID="425875e1-5c2a-4d72-ae1a-b412f0b0f5c8" containerID="f64e0cd2493ccf2ea1b67ed081d06a1991328fd3ab89db66b68879cd05062d6e" exitCode=0
Sep 17 16:02:10.569922 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:10.569856    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c88bfc66d-vl4gd"
Sep 17 16:02:10.569922 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:10.569875    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c88bfc66d-vl4gd" event={"ID":"425875e1-5c2a-4d72-ae1a-b412f0b0f5c8","Type":"ContainerDied","Data":"f64e0cd2493ccf2ea1b67ed081d06a1991328fd3ab89db66b68879cd05062d6e"}
Sep 17 16:02:10.569989 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:10.569929    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c88bfc66d-vl4gd" event={"ID":"425875e1-5c2a-4d72-ae1a-b412f0b0f5c8","Type":"ContainerDied","Data":"4c6389dc04f8da8f12c5be42aab0f30eb67fa14b5f2938aea64501821081b980"}
Sep 17 16:02:10.569989 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:10.569953    2585 scope.go:117] "RemoveContainer" containerID="f64e0cd2493ccf2ea1b67ed081d06a1991328fd3ab89db66b68879cd05062d6e"
Sep 17 16:02:10.579920 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:10.579905    2585 scope.go:117] "RemoveContainer" containerID="f64e0cd2493ccf2ea1b67ed081d06a1991328fd3ab89db66b68879cd05062d6e"
Sep 17 16:02:10.580227 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:02:10.580193    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f64e0cd2493ccf2ea1b67ed081d06a1991328fd3ab89db66b68879cd05062d6e\": container with ID starting with f64e0cd2493ccf2ea1b67ed081d06a1991328fd3ab89db66b68879cd05062d6e not found: ID does not exist" containerID="f64e0cd2493ccf2ea1b67ed081d06a1991328fd3ab89db66b68879cd05062d6e"
Sep 17 16:02:10.580306 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:10.580233    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f64e0cd2493ccf2ea1b67ed081d06a1991328fd3ab89db66b68879cd05062d6e"} err="failed to get container status \"f64e0cd2493ccf2ea1b67ed081d06a1991328fd3ab89db66b68879cd05062d6e\": rpc error: code = NotFound desc = could not find container \"f64e0cd2493ccf2ea1b67ed081d06a1991328fd3ab89db66b68879cd05062d6e\": container with ID starting with f64e0cd2493ccf2ea1b67ed081d06a1991328fd3ab89db66b68879cd05062d6e not found: ID does not exist"
Sep 17 16:02:10.589146 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:10.589124    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c88bfc66d-vl4gd"]
Sep 17 16:02:10.592142 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:10.592120    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6c88bfc66d-vl4gd"]
Sep 17 16:02:10.663213 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:10.663179    2585 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/425875e1-5c2a-4d72-ae1a-b412f0b0f5c8-trusted-ca-bundle\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:02:10.663213 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:10.663208    2585 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/425875e1-5c2a-4d72-ae1a-b412f0b0f5c8-console-oauth-config\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:02:10.663213 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:10.663219    2585 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/425875e1-5c2a-4d72-ae1a-b412f0b0f5c8-service-ca\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:02:10.663515 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:10.663228    2585 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/425875e1-5c2a-4d72-ae1a-b412f0b0f5c8-console-config\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:02:10.663515 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:10.663236    2585 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/425875e1-5c2a-4d72-ae1a-b412f0b0f5c8-oauth-serving-cert\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:02:10.663515 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:10.663244    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vdlc\" (UniqueName: \"kubernetes.io/projected/425875e1-5c2a-4d72-ae1a-b412f0b0f5c8-kube-api-access-6vdlc\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:02:10.663515 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:10.663252    2585 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/425875e1-5c2a-4d72-ae1a-b412f0b0f5c8-console-serving-cert\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:02:12.511795 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:12.511756    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="425875e1-5c2a-4d72-ae1a-b412f0b0f5c8" path="/var/lib/kubelet/pods/425875e1-5c2a-4d72-ae1a-b412f0b0f5c8/volumes"
Sep 17 16:02:52.173337 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:52.173265    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-6fdd8844c6-wfxds"]
Sep 17 16:02:52.173795 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:52.173613    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="425875e1-5c2a-4d72-ae1a-b412f0b0f5c8" containerName="console"
Sep 17 16:02:52.173795 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:52.173625    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="425875e1-5c2a-4d72-ae1a-b412f0b0f5c8" containerName="console"
Sep 17 16:02:52.173795 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:52.173695    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="425875e1-5c2a-4d72-ae1a-b412f0b0f5c8" containerName="console"
Sep 17 16:02:52.175610 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:52.175595    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6fdd8844c6-wfxds"
Sep 17 16:02:52.180571 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:52.180549    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 17 16:02:52.181435 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:52.181390    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 16:02:52.181544 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:52.181450    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 16:02:52.181544 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:52.181460    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-km5w2\""
Sep 17 16:02:52.188794 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:52.188771    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6fdd8844c6-wfxds"]
Sep 17 16:02:52.195410 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:52.195374    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/efca2d86-aa4f-4711-873a-d9f71214fcaa-cert\") pod \"kserve-controller-manager-6fdd8844c6-wfxds\" (UID: \"efca2d86-aa4f-4711-873a-d9f71214fcaa\") " pod="kserve/kserve-controller-manager-6fdd8844c6-wfxds"
Sep 17 16:02:52.195549 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:52.195472    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjt4n\" (UniqueName: \"kubernetes.io/projected/efca2d86-aa4f-4711-873a-d9f71214fcaa-kube-api-access-vjt4n\") pod \"kserve-controller-manager-6fdd8844c6-wfxds\" (UID: \"efca2d86-aa4f-4711-873a-d9f71214fcaa\") " pod="kserve/kserve-controller-manager-6fdd8844c6-wfxds"
Sep 17 16:02:52.213336 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:52.213304    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-7x29p"]
Sep 17 16:02:52.215381 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:52.215363    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-7x29p"
Sep 17 16:02:52.218832 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:52.218812    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-9gvx7\""
Sep 17 16:02:52.218917 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:52.218812    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 17 16:02:52.225739 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:52.225718    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-7x29p"]
Sep 17 16:02:52.296472 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:52.296437    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnfrh\" (UniqueName: \"kubernetes.io/projected/e27478c9-a7f6-491a-bd3f-5e37c0b3808a-kube-api-access-fnfrh\") pod \"seaweedfs-64568bcd49-7x29p\" (UID: \"e27478c9-a7f6-491a-bd3f-5e37c0b3808a\") " pod="kserve/seaweedfs-64568bcd49-7x29p"
Sep 17 16:02:52.296642 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:52.296506    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/efca2d86-aa4f-4711-873a-d9f71214fcaa-cert\") pod \"kserve-controller-manager-6fdd8844c6-wfxds\" (UID: \"efca2d86-aa4f-4711-873a-d9f71214fcaa\") " pod="kserve/kserve-controller-manager-6fdd8844c6-wfxds"
Sep 17 16:02:52.296642 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:52.296534    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e27478c9-a7f6-491a-bd3f-5e37c0b3808a-data\") pod \"seaweedfs-64568bcd49-7x29p\" (UID: \"e27478c9-a7f6-491a-bd3f-5e37c0b3808a\") " pod="kserve/seaweedfs-64568bcd49-7x29p"
Sep 17 16:02:52.296642 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:52.296569    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjt4n\" (UniqueName: \"kubernetes.io/projected/efca2d86-aa4f-4711-873a-d9f71214fcaa-kube-api-access-vjt4n\") pod \"kserve-controller-manager-6fdd8844c6-wfxds\" (UID: \"efca2d86-aa4f-4711-873a-d9f71214fcaa\") " pod="kserve/kserve-controller-manager-6fdd8844c6-wfxds"
Sep 17 16:02:52.296771 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:02:52.296640    2585 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 17 16:02:52.296771 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:02:52.296707    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efca2d86-aa4f-4711-873a-d9f71214fcaa-cert podName:efca2d86-aa4f-4711-873a-d9f71214fcaa nodeName:}" failed. No retries permitted until 2026-09-17 16:02:52.7966901 +0000 UTC m=+528.567021876 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/efca2d86-aa4f-4711-873a-d9f71214fcaa-cert") pod "kserve-controller-manager-6fdd8844c6-wfxds" (UID: "efca2d86-aa4f-4711-873a-d9f71214fcaa") : secret "kserve-webhook-server-cert" not found
Sep 17 16:02:52.308212 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:52.308185    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjt4n\" (UniqueName: \"kubernetes.io/projected/efca2d86-aa4f-4711-873a-d9f71214fcaa-kube-api-access-vjt4n\") pod \"kserve-controller-manager-6fdd8844c6-wfxds\" (UID: \"efca2d86-aa4f-4711-873a-d9f71214fcaa\") " pod="kserve/kserve-controller-manager-6fdd8844c6-wfxds"
Sep 17 16:02:52.397467 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:52.397407    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnfrh\" (UniqueName: \"kubernetes.io/projected/e27478c9-a7f6-491a-bd3f-5e37c0b3808a-kube-api-access-fnfrh\") pod \"seaweedfs-64568bcd49-7x29p\" (UID: \"e27478c9-a7f6-491a-bd3f-5e37c0b3808a\") " pod="kserve/seaweedfs-64568bcd49-7x29p"
Sep 17 16:02:52.397671 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:52.397503    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e27478c9-a7f6-491a-bd3f-5e37c0b3808a-data\") pod \"seaweedfs-64568bcd49-7x29p\" (UID: \"e27478c9-a7f6-491a-bd3f-5e37c0b3808a\") " pod="kserve/seaweedfs-64568bcd49-7x29p"
Sep 17 16:02:52.397849 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:52.397833    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e27478c9-a7f6-491a-bd3f-5e37c0b3808a-data\") pod \"seaweedfs-64568bcd49-7x29p\" (UID: \"e27478c9-a7f6-491a-bd3f-5e37c0b3808a\") " pod="kserve/seaweedfs-64568bcd49-7x29p"
Sep 17 16:02:52.405854 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:52.405832    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnfrh\" (UniqueName: \"kubernetes.io/projected/e27478c9-a7f6-491a-bd3f-5e37c0b3808a-kube-api-access-fnfrh\") pod \"seaweedfs-64568bcd49-7x29p\" (UID: \"e27478c9-a7f6-491a-bd3f-5e37c0b3808a\") " pod="kserve/seaweedfs-64568bcd49-7x29p"
Sep 17 16:02:52.524613 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:52.524584    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-7x29p"
Sep 17 16:02:52.798657 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:52.798634    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-7x29p"]
Sep 17 16:02:52.800737 ip-10-0-135-43 kubenswrapper[2585]: W0917 16:02:52.800709    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode27478c9_a7f6_491a_bd3f_5e37c0b3808a.slice/crio-544224bcd7eb5efde153b0fac9a50808c80968a8844fb8b3b3d54f7cbf72e9e5 WatchSource:0}: Error finding container 544224bcd7eb5efde153b0fac9a50808c80968a8844fb8b3b3d54f7cbf72e9e5: Status 404 returned error can't find the container with id 544224bcd7eb5efde153b0fac9a50808c80968a8844fb8b3b3d54f7cbf72e9e5
Sep 17 16:02:52.806546 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:52.806521    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/efca2d86-aa4f-4711-873a-d9f71214fcaa-cert\") pod \"kserve-controller-manager-6fdd8844c6-wfxds\" (UID: \"efca2d86-aa4f-4711-873a-d9f71214fcaa\") " pod="kserve/kserve-controller-manager-6fdd8844c6-wfxds"
Sep 17 16:02:52.808728 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:52.808702    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/efca2d86-aa4f-4711-873a-d9f71214fcaa-cert\") pod \"kserve-controller-manager-6fdd8844c6-wfxds\" (UID: \"efca2d86-aa4f-4711-873a-d9f71214fcaa\") " pod="kserve/kserve-controller-manager-6fdd8844c6-wfxds"
Sep 17 16:02:53.086673 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:53.086592    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6fdd8844c6-wfxds"
Sep 17 16:02:53.358629 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:53.358599    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6fdd8844c6-wfxds"]
Sep 17 16:02:53.360962 ip-10-0-135-43 kubenswrapper[2585]: W0917 16:02:53.360936    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefca2d86_aa4f_4711_873a_d9f71214fcaa.slice/crio-d269d35cf13c35902adc76f8c84c6aab912e0aa903cd129302d7c131af30c483 WatchSource:0}: Error finding container d269d35cf13c35902adc76f8c84c6aab912e0aa903cd129302d7c131af30c483: Status 404 returned error can't find the container with id d269d35cf13c35902adc76f8c84c6aab912e0aa903cd129302d7c131af30c483
Sep 17 16:02:53.764065 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:53.764023    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6fdd8844c6-wfxds" event={"ID":"efca2d86-aa4f-4711-873a-d9f71214fcaa","Type":"ContainerStarted","Data":"d269d35cf13c35902adc76f8c84c6aab912e0aa903cd129302d7c131af30c483"}
Sep 17 16:02:53.765331 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:53.765291    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-7x29p" event={"ID":"e27478c9-a7f6-491a-bd3f-5e37c0b3808a","Type":"ContainerStarted","Data":"544224bcd7eb5efde153b0fac9a50808c80968a8844fb8b3b3d54f7cbf72e9e5"}
Sep 17 16:02:57.790975 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:57.790936    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-7x29p" event={"ID":"e27478c9-a7f6-491a-bd3f-5e37c0b3808a","Type":"ContainerStarted","Data":"8d51eb4a86e336745ab8e9a5421ea9da2f96cda2c52b840f1f23e29ced1a344f"}
Sep 17 16:02:57.791546 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:57.791049    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-7x29p"
Sep 17 16:02:57.792532 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:57.792508    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6fdd8844c6-wfxds" event={"ID":"efca2d86-aa4f-4711-873a-d9f71214fcaa","Type":"ContainerStarted","Data":"3f1e8bafce5eb625f58cb6a65ec455c6d2c0754115716621ca4cd8716bcc4f65"}
Sep 17 16:02:57.792658 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:57.792637    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-6fdd8844c6-wfxds"
Sep 17 16:02:57.809598 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:57.809551    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-7x29p" podStartSLOduration=1.7829773439999999 podStartE2EDuration="5.809534866s" totalImagesPullingTime="4.026557522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:02:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:02:57.808048157 +0000 UTC m=+533.578379954" watchObservedRunningTime="2026-09-17 16:02:57.809534866 +0000 UTC m=+533.579866664"
Sep 17 16:02:57.832814 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:02:57.832751    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-6fdd8844c6-wfxds" podStartSLOduration=2.403973096 podStartE2EDuration="5.832730224s" totalImagesPullingTime="3.428757128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:02:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:02:57.831400502 +0000 UTC m=+533.601732313" watchObservedRunningTime="2026-09-17 16:02:57.832730224 +0000 UTC m=+533.603062021"
Sep 17 16:03:03.799730 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:03:03.799699    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-7x29p"
Sep 17 16:03:28.802071 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:03:28.802037    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-6fdd8844c6-wfxds"
Sep 17 16:03:34.803992 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:03:34.803959    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-6fdd8844c6-wfxds"]
Sep 17 16:03:34.804568 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:03:34.804141    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-6fdd8844c6-wfxds" podUID="efca2d86-aa4f-4711-873a-d9f71214fcaa" containerName="manager" containerID="cri-o://3f1e8bafce5eb625f58cb6a65ec455c6d2c0754115716621ca4cd8716bcc4f65" gracePeriod=10
Sep 17 16:03:34.835722 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:03:34.835689    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-6fdd8844c6-x5r6k"]
Sep 17 16:03:34.837832 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:03:34.837813    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6fdd8844c6-x5r6k"
Sep 17 16:03:34.857128 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:03:34.854240    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6fdd8844c6-x5r6k"]
Sep 17 16:03:34.956242 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:03:34.956211    2585 generic.go:356] "Generic (PLEG): container finished" podID="efca2d86-aa4f-4711-873a-d9f71214fcaa" containerID="3f1e8bafce5eb625f58cb6a65ec455c6d2c0754115716621ca4cd8716bcc4f65" exitCode=0
Sep 17 16:03:34.956398 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:03:34.956250    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6fdd8844c6-wfxds" event={"ID":"efca2d86-aa4f-4711-873a-d9f71214fcaa","Type":"ContainerDied","Data":"3f1e8bafce5eb625f58cb6a65ec455c6d2c0754115716621ca4cd8716bcc4f65"}
Sep 17 16:03:34.960563 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:03:34.960542    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p727c\" (UniqueName: \"kubernetes.io/projected/94a399c0-cbd4-49d6-995f-b8e121a88af2-kube-api-access-p727c\") pod \"kserve-controller-manager-6fdd8844c6-x5r6k\" (UID: \"94a399c0-cbd4-49d6-995f-b8e121a88af2\") " pod="kserve/kserve-controller-manager-6fdd8844c6-x5r6k"
Sep 17 16:03:34.960616 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:03:34.960573    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/94a399c0-cbd4-49d6-995f-b8e121a88af2-cert\") pod \"kserve-controller-manager-6fdd8844c6-x5r6k\" (UID: \"94a399c0-cbd4-49d6-995f-b8e121a88af2\") " pod="kserve/kserve-controller-manager-6fdd8844c6-x5r6k"
Sep 17 16:03:35.042301 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:03:35.042275    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6fdd8844c6-wfxds"
Sep 17 16:03:35.061358 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:03:35.061262    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/94a399c0-cbd4-49d6-995f-b8e121a88af2-cert\") pod \"kserve-controller-manager-6fdd8844c6-x5r6k\" (UID: \"94a399c0-cbd4-49d6-995f-b8e121a88af2\") " pod="kserve/kserve-controller-manager-6fdd8844c6-x5r6k"
Sep 17 16:03:35.061594 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:03:35.061571    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p727c\" (UniqueName: \"kubernetes.io/projected/94a399c0-cbd4-49d6-995f-b8e121a88af2-kube-api-access-p727c\") pod \"kserve-controller-manager-6fdd8844c6-x5r6k\" (UID: \"94a399c0-cbd4-49d6-995f-b8e121a88af2\") " pod="kserve/kserve-controller-manager-6fdd8844c6-x5r6k"
Sep 17 16:03:35.063909 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:03:35.063888    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/94a399c0-cbd4-49d6-995f-b8e121a88af2-cert\") pod \"kserve-controller-manager-6fdd8844c6-x5r6k\" (UID: \"94a399c0-cbd4-49d6-995f-b8e121a88af2\") " pod="kserve/kserve-controller-manager-6fdd8844c6-x5r6k"
Sep 17 16:03:35.073208 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:03:35.073181    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p727c\" (UniqueName: \"kubernetes.io/projected/94a399c0-cbd4-49d6-995f-b8e121a88af2-kube-api-access-p727c\") pod \"kserve-controller-manager-6fdd8844c6-x5r6k\" (UID: \"94a399c0-cbd4-49d6-995f-b8e121a88af2\") " pod="kserve/kserve-controller-manager-6fdd8844c6-x5r6k"
Sep 17 16:03:35.162222 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:03:35.162178    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/efca2d86-aa4f-4711-873a-d9f71214fcaa-cert\") pod \"efca2d86-aa4f-4711-873a-d9f71214fcaa\" (UID: \"efca2d86-aa4f-4711-873a-d9f71214fcaa\") "
Sep 17 16:03:35.162389 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:03:35.162238    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vjt4n\" (UniqueName: \"kubernetes.io/projected/efca2d86-aa4f-4711-873a-d9f71214fcaa-kube-api-access-vjt4n\") pod \"efca2d86-aa4f-4711-873a-d9f71214fcaa\" (UID: \"efca2d86-aa4f-4711-873a-d9f71214fcaa\") "
Sep 17 16:03:35.164568 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:03:35.164541    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efca2d86-aa4f-4711-873a-d9f71214fcaa-cert" (OuterVolumeSpecName: "cert") pod "efca2d86-aa4f-4711-873a-d9f71214fcaa" (UID: "efca2d86-aa4f-4711-873a-d9f71214fcaa"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:03:35.164660 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:03:35.164545    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efca2d86-aa4f-4711-873a-d9f71214fcaa-kube-api-access-vjt4n" (OuterVolumeSpecName: "kube-api-access-vjt4n") pod "efca2d86-aa4f-4711-873a-d9f71214fcaa" (UID: "efca2d86-aa4f-4711-873a-d9f71214fcaa"). InnerVolumeSpecName "kube-api-access-vjt4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:03:35.197884 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:03:35.197853    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6fdd8844c6-x5r6k"
Sep 17 16:03:35.262949 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:03:35.262918    2585 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/efca2d86-aa4f-4711-873a-d9f71214fcaa-cert\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:03:35.262949 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:03:35.262948    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjt4n\" (UniqueName: \"kubernetes.io/projected/efca2d86-aa4f-4711-873a-d9f71214fcaa-kube-api-access-vjt4n\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:03:35.468379 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:03:35.468348    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6fdd8844c6-x5r6k"]
Sep 17 16:03:35.471771 ip-10-0-135-43 kubenswrapper[2585]: W0917 16:03:35.471741    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94a399c0_cbd4_49d6_995f_b8e121a88af2.slice/crio-1c109a46df69e743ab674f9906311b5a2ebb277f2afd31e6b952c1ffc4b88d5a WatchSource:0}: Error finding container 1c109a46df69e743ab674f9906311b5a2ebb277f2afd31e6b952c1ffc4b88d5a: Status 404 returned error can't find the container with id 1c109a46df69e743ab674f9906311b5a2ebb277f2afd31e6b952c1ffc4b88d5a
Sep 17 16:03:35.962054 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:03:35.962023    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6fdd8844c6-wfxds"
Sep 17 16:03:35.962525 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:03:35.962021    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6fdd8844c6-wfxds" event={"ID":"efca2d86-aa4f-4711-873a-d9f71214fcaa","Type":"ContainerDied","Data":"d269d35cf13c35902adc76f8c84c6aab912e0aa903cd129302d7c131af30c483"}
Sep 17 16:03:35.962525 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:03:35.962159    2585 scope.go:117] "RemoveContainer" containerID="3f1e8bafce5eb625f58cb6a65ec455c6d2c0754115716621ca4cd8716bcc4f65"
Sep 17 16:03:35.963914 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:03:35.963891    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6fdd8844c6-x5r6k" event={"ID":"94a399c0-cbd4-49d6-995f-b8e121a88af2","Type":"ContainerStarted","Data":"6dde0f8b6e463c760974c579832053aa46162ab8e85b0d53b9311592a7e33ee3"}
Sep 17 16:03:35.964007 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:03:35.963941    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6fdd8844c6-x5r6k" event={"ID":"94a399c0-cbd4-49d6-995f-b8e121a88af2","Type":"ContainerStarted","Data":"1c109a46df69e743ab674f9906311b5a2ebb277f2afd31e6b952c1ffc4b88d5a"}
Sep 17 16:03:35.964007 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:03:35.963986    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-6fdd8844c6-x5r6k"
Sep 17 16:03:35.985625 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:03:35.985576    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-6fdd8844c6-x5r6k" podStartSLOduration=1.567565729 podStartE2EDuration="1.985559943s" totalImagesPullingTime="417.994214ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:03:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:03:35.98505987 +0000 UTC m=+571.755391668" watchObservedRunningTime="2026-09-17 16:03:35.985559943 +0000 UTC m=+571.755891742"
Sep 17 16:03:35.999350 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:03:35.999309    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-6fdd8844c6-wfxds"]
Sep 17 16:03:36.003504 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:03:36.003473    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-6fdd8844c6-wfxds"]
Sep 17 16:03:36.512640 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:03:36.512610    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efca2d86-aa4f-4711-873a-d9f71214fcaa" path="/var/lib/kubelet/pods/efca2d86-aa4f-4711-873a-d9f71214fcaa/volumes"
Sep 17 16:04:06.391709 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:04:06.391682    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2b47p_a1850a4d-4d01-4a71-8636-b5918f1426c2/ovn-acl-logging/0.log"
Sep 17 16:04:06.392630 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:04:06.392605    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2b47p_a1850a4d-4d01-4a71-8636-b5918f1426c2/ovn-acl-logging/0.log"
Sep 17 16:04:06.972544 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:04:06.972507    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-6fdd8844c6-x5r6k"
Sep 17 16:04:08.343458 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:04:08.343411    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-w46wj"]
Sep 17 16:04:08.343848 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:04:08.343800    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efca2d86-aa4f-4711-873a-d9f71214fcaa" containerName="manager"
Sep 17 16:04:08.343848 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:04:08.343818    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="efca2d86-aa4f-4711-873a-d9f71214fcaa" containerName="manager"
Sep 17 16:04:08.343946 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:04:08.343890    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="efca2d86-aa4f-4711-873a-d9f71214fcaa" containerName="manager"
Sep 17 16:04:08.348197 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:04:08.348176    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-w46wj"
Sep 17 16:04:08.351035 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:04:08.351011    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-zbdsr\""
Sep 17 16:04:08.351168 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:04:08.351011    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 17 16:04:08.358725 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:04:08.358700    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-w46wj"]
Sep 17 16:04:08.362208 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:04:08.362185    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-nvjlp"]
Sep 17 16:04:08.365563 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:04:08.365541    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-nvjlp"
Sep 17 16:04:08.367997 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:04:08.367973    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 17 16:04:08.368105 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:04:08.368090    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 17 16:04:08.368171 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:04:08.368153    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-9q9nd\""
Sep 17 16:04:08.375577 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:04:08.375552    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-nvjlp"]
Sep 17 16:04:08.418439 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:04:08.418396    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwdkd\" (UniqueName: \"kubernetes.io/projected/033170f4-5975-4cf6-91d6-6abcd2bf9251-kube-api-access-wwdkd\") pod \"odh-model-controller-7b966d7984-nvjlp\" (UID: \"033170f4-5975-4cf6-91d6-6abcd2bf9251\") " pod="kserve/odh-model-controller-7b966d7984-nvjlp"
Sep 17 16:04:08.418583 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:04:08.418442    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6accedc-04f7-466d-86ea-c8215e9b416e-tls-certs\") pod \"model-serving-api-7b954ffc95-w46wj\" (UID: \"a6accedc-04f7-466d-86ea-c8215e9b416e\") " pod="kserve/model-serving-api-7b954ffc95-w46wj"
Sep 17 16:04:08.418583 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:04:08.418479    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/033170f4-5975-4cf6-91d6-6abcd2bf9251-metrics-certs\") pod \"odh-model-controller-7b966d7984-nvjlp\" (UID: \"033170f4-5975-4cf6-91d6-6abcd2bf9251\") " pod="kserve/odh-model-controller-7b966d7984-nvjlp"
Sep 17 16:04:08.418583 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:04:08.418518    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qk6f\" (UniqueName: \"kubernetes.io/projected/a6accedc-04f7-466d-86ea-c8215e9b416e-kube-api-access-7qk6f\") pod \"model-serving-api-7b954ffc95-w46wj\" (UID: \"a6accedc-04f7-466d-86ea-c8215e9b416e\") " pod="kserve/model-serving-api-7b954ffc95-w46wj"
Sep 17 16:04:08.418583 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:04:08.418544    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/033170f4-5975-4cf6-91d6-6abcd2bf9251-cert\") pod \"odh-model-controller-7b966d7984-nvjlp\" (UID: \"033170f4-5975-4cf6-91d6-6abcd2bf9251\") " pod="kserve/odh-model-controller-7b966d7984-nvjlp"
Sep 17 16:04:08.519173 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:04:08.519139    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qk6f\" (UniqueName: \"kubernetes.io/projected/a6accedc-04f7-466d-86ea-c8215e9b416e-kube-api-access-7qk6f\") pod \"model-serving-api-7b954ffc95-w46wj\" (UID: \"a6accedc-04f7-466d-86ea-c8215e9b416e\") " pod="kserve/model-serving-api-7b954ffc95-w46wj"
Sep 17 16:04:08.519173 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:04:08.519177    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/033170f4-5975-4cf6-91d6-6abcd2bf9251-cert\") pod \"odh-model-controller-7b966d7984-nvjlp\" (UID: \"033170f4-5975-4cf6-91d6-6abcd2bf9251\") " pod="kserve/odh-model-controller-7b966d7984-nvjlp"
Sep 17 16:04:08.519388 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:04:08.519229    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwdkd\" (UniqueName: \"kubernetes.io/projected/033170f4-5975-4cf6-91d6-6abcd2bf9251-kube-api-access-wwdkd\") pod \"odh-model-controller-7b966d7984-nvjlp\" (UID: \"033170f4-5975-4cf6-91d6-6abcd2bf9251\") " pod="kserve/odh-model-controller-7b966d7984-nvjlp"
Sep 17 16:04:08.519388 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:04:08.519264    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6accedc-04f7-466d-86ea-c8215e9b416e-tls-certs\") pod \"model-serving-api-7b954ffc95-w46wj\" (UID: \"a6accedc-04f7-466d-86ea-c8215e9b416e\") " pod="kserve/model-serving-api-7b954ffc95-w46wj"
Sep 17 16:04:08.519388 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:04:08.519294    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/033170f4-5975-4cf6-91d6-6abcd2bf9251-metrics-certs\") pod \"odh-model-controller-7b966d7984-nvjlp\" (UID: \"033170f4-5975-4cf6-91d6-6abcd2bf9251\") " pod="kserve/odh-model-controller-7b966d7984-nvjlp"
Sep 17 16:04:08.519388 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:04:08.519367    2585 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 17 16:04:08.519561 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:04:08.519377    2585 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 17 16:04:08.519561 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:04:08.519453    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/033170f4-5975-4cf6-91d6-6abcd2bf9251-cert podName:033170f4-5975-4cf6-91d6-6abcd2bf9251 nodeName:}" failed. No retries permitted until 2026-09-17 16:04:09.019432638 +0000 UTC m=+604.789764414 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/033170f4-5975-4cf6-91d6-6abcd2bf9251-cert") pod "odh-model-controller-7b966d7984-nvjlp" (UID: "033170f4-5975-4cf6-91d6-6abcd2bf9251") : secret "odh-model-controller-webhook-cert" not found
Sep 17 16:04:08.519561 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:04:08.519492    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6accedc-04f7-466d-86ea-c8215e9b416e-tls-certs podName:a6accedc-04f7-466d-86ea-c8215e9b416e nodeName:}" failed. No retries permitted until 2026-09-17 16:04:09.019475877 +0000 UTC m=+604.789807654 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/a6accedc-04f7-466d-86ea-c8215e9b416e-tls-certs") pod "model-serving-api-7b954ffc95-w46wj" (UID: "a6accedc-04f7-466d-86ea-c8215e9b416e") : secret "model-serving-api-tls" not found
Sep 17 16:04:08.521867 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:04:08.521846    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/033170f4-5975-4cf6-91d6-6abcd2bf9251-metrics-certs\") pod \"odh-model-controller-7b966d7984-nvjlp\" (UID: \"033170f4-5975-4cf6-91d6-6abcd2bf9251\") " pod="kserve/odh-model-controller-7b966d7984-nvjlp"
Sep 17 16:04:08.528293 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:04:08.528272    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qk6f\" (UniqueName: \"kubernetes.io/projected/a6accedc-04f7-466d-86ea-c8215e9b416e-kube-api-access-7qk6f\") pod \"model-serving-api-7b954ffc95-w46wj\" (UID: \"a6accedc-04f7-466d-86ea-c8215e9b416e\") " pod="kserve/model-serving-api-7b954ffc95-w46wj"
Sep 17 16:04:08.528436 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:04:08.528401    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwdkd\" (UniqueName: \"kubernetes.io/projected/033170f4-5975-4cf6-91d6-6abcd2bf9251-kube-api-access-wwdkd\") pod \"odh-model-controller-7b966d7984-nvjlp\" (UID: \"033170f4-5975-4cf6-91d6-6abcd2bf9251\") " pod="kserve/odh-model-controller-7b966d7984-nvjlp"
Sep 17 16:04:09.022669 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:04:09.022631    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6accedc-04f7-466d-86ea-c8215e9b416e-tls-certs\") pod \"model-serving-api-7b954ffc95-w46wj\" (UID: \"a6accedc-04f7-466d-86ea-c8215e9b416e\") " pod="kserve/model-serving-api-7b954ffc95-w46wj"
Sep 17 16:04:09.022864 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:04:09.022699    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/033170f4-5975-4cf6-91d6-6abcd2bf9251-cert\") pod \"odh-model-controller-7b966d7984-nvjlp\" (UID: \"033170f4-5975-4cf6-91d6-6abcd2bf9251\") " pod="kserve/odh-model-controller-7b966d7984-nvjlp"
Sep 17 16:04:09.025140 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:04:09.025103    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6accedc-04f7-466d-86ea-c8215e9b416e-tls-certs\") pod \"model-serving-api-7b954ffc95-w46wj\" (UID: \"a6accedc-04f7-466d-86ea-c8215e9b416e\") " pod="kserve/model-serving-api-7b954ffc95-w46wj"
Sep 17 16:04:09.025263 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:04:09.025156    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/033170f4-5975-4cf6-91d6-6abcd2bf9251-cert\") pod \"odh-model-controller-7b966d7984-nvjlp\" (UID: \"033170f4-5975-4cf6-91d6-6abcd2bf9251\") " pod="kserve/odh-model-controller-7b966d7984-nvjlp"
Sep 17 16:04:09.260070 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:04:09.260033    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-w46wj"
Sep 17 16:04:09.276324 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:04:09.276265    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-nvjlp"
Sep 17 16:04:09.534080 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:04:09.534056    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-w46wj"]
Sep 17 16:04:09.536469 ip-10-0-135-43 kubenswrapper[2585]: W0917 16:04:09.536438    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6accedc_04f7_466d_86ea_c8215e9b416e.slice/crio-65de8e33f749d7bb5efa7b3ee1ce35a3ffe9d01b055eaa574cbaad9275d3ecb6 WatchSource:0}: Error finding container 65de8e33f749d7bb5efa7b3ee1ce35a3ffe9d01b055eaa574cbaad9275d3ecb6: Status 404 returned error can't find the container with id 65de8e33f749d7bb5efa7b3ee1ce35a3ffe9d01b055eaa574cbaad9275d3ecb6
Sep 17 16:04:09.564657 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:04:09.564631    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-nvjlp"]
Sep 17 16:04:09.567481 ip-10-0-135-43 kubenswrapper[2585]: W0917 16:04:09.567457    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod033170f4_5975_4cf6_91d6_6abcd2bf9251.slice/crio-19c6d5c21c37d3fd24539abbe7654f8849959889ace29f2629b7c7a5c3a72ba5 WatchSource:0}: Error finding container 19c6d5c21c37d3fd24539abbe7654f8849959889ace29f2629b7c7a5c3a72ba5: Status 404 returned error can't find the container with id 19c6d5c21c37d3fd24539abbe7654f8849959889ace29f2629b7c7a5c3a72ba5
Sep 17 16:04:10.113251 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:04:10.113213    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-w46wj" event={"ID":"a6accedc-04f7-466d-86ea-c8215e9b416e","Type":"ContainerStarted","Data":"65de8e33f749d7bb5efa7b3ee1ce35a3ffe9d01b055eaa574cbaad9275d3ecb6"}
Sep 17 16:04:10.114355 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:04:10.114331    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-nvjlp" event={"ID":"033170f4-5975-4cf6-91d6-6abcd2bf9251","Type":"ContainerStarted","Data":"19c6d5c21c37d3fd24539abbe7654f8849959889ace29f2629b7c7a5c3a72ba5"}
Sep 17 16:04:14.138683 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:04:14.138638    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-w46wj" event={"ID":"a6accedc-04f7-466d-86ea-c8215e9b416e","Type":"ContainerStarted","Data":"c95da0075c30d679c24caaa74c4623c9535e044d7df8b2af2b38454968f4d484"}
Sep 17 16:04:14.139133 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:04:14.138735    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-w46wj"
Sep 17 16:04:14.140281 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:04:14.140253    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-nvjlp" event={"ID":"033170f4-5975-4cf6-91d6-6abcd2bf9251","Type":"ContainerStarted","Data":"0d8b53d2cb3e21ed7d2af17b88019f0e1e23c6eacb35c8eba96882e87a8601cb"}
Sep 17 16:04:14.140386 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:04:14.140321    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-nvjlp"
Sep 17 16:04:14.159648 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:04:14.159609    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-w46wj" podStartSLOduration=2.165415903 podStartE2EDuration="6.159596908s" totalImagesPullingTime="3.994181005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:04:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:04:14.157391718 +0000 UTC m=+609.927723516" watchObservedRunningTime="2026-09-17 16:04:14.159596908 +0000 UTC m=+609.929928707"
Sep 17 16:04:14.176179 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:04:14.176133    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-nvjlp" podStartSLOduration=2.167141052 podStartE2EDuration="6.176119467s" totalImagesPullingTime="4.008978415s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:04:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:04:14.174854431 +0000 UTC m=+609.945186230" watchObservedRunningTime="2026-09-17 16:04:14.176119467 +0000 UTC m=+609.946451264"
Sep 17 16:04:25.147894 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:04:25.147859    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-nvjlp"
Sep 17 16:04:25.149853 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:04:25.149478    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-w46wj"
Sep 17 16:05:46.750967 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:05:46.748289    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-p2mcb"]
Sep 17 16:05:46.752564 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:05:46.752537    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-p2mcb"
Sep 17 16:05:46.755919 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:05:46.755884    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 17 16:05:46.760309 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:05:46.760278    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-p2mcb"]
Sep 17 16:05:46.856005 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:05:46.855965    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8ea26ad2-9493-4b7b-a536-de88a1b39830-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-p2mcb\" (UID: \"8ea26ad2-9493-4b7b-a536-de88a1b39830\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-p2mcb"
Sep 17 16:05:46.856190 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:05:46.856012    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg7kd\" (UniqueName: \"kubernetes.io/projected/8ea26ad2-9493-4b7b-a536-de88a1b39830-kube-api-access-xg7kd\") pod \"seaweedfs-tls-custom-6644bf5d6c-p2mcb\" (UID: \"8ea26ad2-9493-4b7b-a536-de88a1b39830\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-p2mcb"
Sep 17 16:05:46.957439 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:05:46.957370    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8ea26ad2-9493-4b7b-a536-de88a1b39830-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-p2mcb\" (UID: \"8ea26ad2-9493-4b7b-a536-de88a1b39830\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-p2mcb"
Sep 17 16:05:46.957614 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:05:46.957467    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg7kd\" (UniqueName: \"kubernetes.io/projected/8ea26ad2-9493-4b7b-a536-de88a1b39830-kube-api-access-xg7kd\") pod \"seaweedfs-tls-custom-6644bf5d6c-p2mcb\" (UID: \"8ea26ad2-9493-4b7b-a536-de88a1b39830\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-p2mcb"
Sep 17 16:05:46.957839 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:05:46.957817    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8ea26ad2-9493-4b7b-a536-de88a1b39830-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-p2mcb\" (UID: \"8ea26ad2-9493-4b7b-a536-de88a1b39830\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-p2mcb"
Sep 17 16:05:46.966010 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:05:46.965982    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg7kd\" (UniqueName: \"kubernetes.io/projected/8ea26ad2-9493-4b7b-a536-de88a1b39830-kube-api-access-xg7kd\") pod \"seaweedfs-tls-custom-6644bf5d6c-p2mcb\" (UID: \"8ea26ad2-9493-4b7b-a536-de88a1b39830\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-p2mcb"
Sep 17 16:05:47.063747 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:05:47.063656    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-p2mcb"
Sep 17 16:05:47.340517 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:05:47.340483    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-p2mcb"]
Sep 17 16:05:47.342984 ip-10-0-135-43 kubenswrapper[2585]: W0917 16:05:47.342953    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ea26ad2_9493_4b7b_a536_de88a1b39830.slice/crio-373474a9ebc8ed16abe10cf42e2949adfdbe6ab84e9541b22396ec0835e0673c WatchSource:0}: Error finding container 373474a9ebc8ed16abe10cf42e2949adfdbe6ab84e9541b22396ec0835e0673c: Status 404 returned error can't find the container with id 373474a9ebc8ed16abe10cf42e2949adfdbe6ab84e9541b22396ec0835e0673c
Sep 17 16:05:47.344284 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:05:47.344269    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 16:05:47.543774 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:05:47.543737    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-p2mcb" event={"ID":"8ea26ad2-9493-4b7b-a536-de88a1b39830","Type":"ContainerStarted","Data":"373474a9ebc8ed16abe10cf42e2949adfdbe6ab84e9541b22396ec0835e0673c"}
Sep 17 16:05:48.549381 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:05:48.549343    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-p2mcb" event={"ID":"8ea26ad2-9493-4b7b-a536-de88a1b39830","Type":"ContainerStarted","Data":"a866e1d403794acf26a5de0df157f4e5d84211871a1b37e6a0ec467373f8d033"}
Sep 17 16:05:48.566363 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:05:48.566305    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-p2mcb" podStartSLOduration=2.296251051 podStartE2EDuration="2.566283813s" totalImagesPullingTime="270.032762ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:05:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:05:48.566097179 +0000 UTC m=+704.336428977" watchObservedRunningTime="2026-09-17 16:05:48.566283813 +0000 UTC m=+704.336615610"
Sep 17 16:05:49.471249 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:05:49.471210    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-p2mcb"]
Sep 17 16:05:50.557710 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:05:50.557649    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-p2mcb" podUID="8ea26ad2-9493-4b7b-a536-de88a1b39830" containerName="seaweedfs-tls-custom" containerID="cri-o://a866e1d403794acf26a5de0df157f4e5d84211871a1b37e6a0ec467373f8d033" gracePeriod=30
Sep 17 16:05:51.794227 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:05:51.794202    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-p2mcb"
Sep 17 16:05:51.902576 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:05:51.902479    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xg7kd\" (UniqueName: \"kubernetes.io/projected/8ea26ad2-9493-4b7b-a536-de88a1b39830-kube-api-access-xg7kd\") pod \"8ea26ad2-9493-4b7b-a536-de88a1b39830\" (UID: \"8ea26ad2-9493-4b7b-a536-de88a1b39830\") "
Sep 17 16:05:51.902576 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:05:51.902553    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8ea26ad2-9493-4b7b-a536-de88a1b39830-data\") pod \"8ea26ad2-9493-4b7b-a536-de88a1b39830\" (UID: \"8ea26ad2-9493-4b7b-a536-de88a1b39830\") "
Sep 17 16:05:51.903871 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:05:51.903841    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ea26ad2-9493-4b7b-a536-de88a1b39830-data" (OuterVolumeSpecName: "data") pod "8ea26ad2-9493-4b7b-a536-de88a1b39830" (UID: "8ea26ad2-9493-4b7b-a536-de88a1b39830"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:05:51.904608 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:05:51.904574    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ea26ad2-9493-4b7b-a536-de88a1b39830-kube-api-access-xg7kd" (OuterVolumeSpecName: "kube-api-access-xg7kd") pod "8ea26ad2-9493-4b7b-a536-de88a1b39830" (UID: "8ea26ad2-9493-4b7b-a536-de88a1b39830"). InnerVolumeSpecName "kube-api-access-xg7kd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:05:52.009015 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:05:52.004533    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg7kd\" (UniqueName: \"kubernetes.io/projected/8ea26ad2-9493-4b7b-a536-de88a1b39830-kube-api-access-xg7kd\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:05:52.009015 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:05:52.004574    2585 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8ea26ad2-9493-4b7b-a536-de88a1b39830-data\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:05:52.567380 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:05:52.567342    2585 generic.go:356] "Generic (PLEG): container finished" podID="8ea26ad2-9493-4b7b-a536-de88a1b39830" containerID="a866e1d403794acf26a5de0df157f4e5d84211871a1b37e6a0ec467373f8d033" exitCode=0
Sep 17 16:05:52.567380 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:05:52.567376    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-p2mcb" event={"ID":"8ea26ad2-9493-4b7b-a536-de88a1b39830","Type":"ContainerDied","Data":"a866e1d403794acf26a5de0df157f4e5d84211871a1b37e6a0ec467373f8d033"}
Sep 17 16:05:52.567649 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:05:52.567392    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-p2mcb"
Sep 17 16:05:52.567649 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:05:52.567453    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-p2mcb" event={"ID":"8ea26ad2-9493-4b7b-a536-de88a1b39830","Type":"ContainerDied","Data":"373474a9ebc8ed16abe10cf42e2949adfdbe6ab84e9541b22396ec0835e0673c"}
Sep 17 16:05:52.567649 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:05:52.567482    2585 scope.go:117] "RemoveContainer" containerID="a866e1d403794acf26a5de0df157f4e5d84211871a1b37e6a0ec467373f8d033"
Sep 17 16:05:52.577337 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:05:52.577312    2585 scope.go:117] "RemoveContainer" containerID="a866e1d403794acf26a5de0df157f4e5d84211871a1b37e6a0ec467373f8d033"
Sep 17 16:05:52.577680 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:05:52.577660    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a866e1d403794acf26a5de0df157f4e5d84211871a1b37e6a0ec467373f8d033\": container with ID starting with a866e1d403794acf26a5de0df157f4e5d84211871a1b37e6a0ec467373f8d033 not found: ID does not exist" containerID="a866e1d403794acf26a5de0df157f4e5d84211871a1b37e6a0ec467373f8d033"
Sep 17 16:05:52.577739 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:05:52.577689    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a866e1d403794acf26a5de0df157f4e5d84211871a1b37e6a0ec467373f8d033"} err="failed to get container status \"a866e1d403794acf26a5de0df157f4e5d84211871a1b37e6a0ec467373f8d033\": rpc error: code = NotFound desc = could not find container \"a866e1d403794acf26a5de0df157f4e5d84211871a1b37e6a0ec467373f8d033\": container with ID starting with a866e1d403794acf26a5de0df157f4e5d84211871a1b37e6a0ec467373f8d033 not found: ID does not exist"
Sep 17 16:05:52.584576 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:05:52.584549    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-p2mcb"]
Sep 17 16:05:52.588503 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:05:52.588480    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-p2mcb"]
Sep 17 16:05:52.618085 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:05:52.618040    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-nnnmx"]
Sep 17 16:05:52.618386 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:05:52.618372    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ea26ad2-9493-4b7b-a536-de88a1b39830" containerName="seaweedfs-tls-custom"
Sep 17 16:05:52.618452 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:05:52.618388    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ea26ad2-9493-4b7b-a536-de88a1b39830" containerName="seaweedfs-tls-custom"
Sep 17 16:05:52.618492 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:05:52.618478    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ea26ad2-9493-4b7b-a536-de88a1b39830" containerName="seaweedfs-tls-custom"
Sep 17 16:05:52.621300 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:05:52.621278    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-nnnmx"
Sep 17 16:05:52.624166 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:05:52.624145    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom\""
Sep 17 16:05:52.624264 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:05:52.624190    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 17 16:05:52.630751 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:05:52.630725    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-nnnmx"]
Sep 17 16:05:52.709563 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:05:52.709519    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4mpx\" (UniqueName: \"kubernetes.io/projected/9a0d05c9-f3bf-41dd-afd2-ce3f47a7ab4d-kube-api-access-x4mpx\") pod \"seaweedfs-tls-custom-8b44d767d-nnnmx\" (UID: \"9a0d05c9-f3bf-41dd-afd2-ce3f47a7ab4d\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-nnnmx"
Sep 17 16:05:52.709563 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:05:52.709568    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/9a0d05c9-f3bf-41dd-afd2-ce3f47a7ab4d-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-nnnmx\" (UID: \"9a0d05c9-f3bf-41dd-afd2-ce3f47a7ab4d\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-nnnmx"
Sep 17 16:05:52.709793 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:05:52.709639    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9a0d05c9-f3bf-41dd-afd2-ce3f47a7ab4d-data\") pod \"seaweedfs-tls-custom-8b44d767d-nnnmx\" (UID: \"9a0d05c9-f3bf-41dd-afd2-ce3f47a7ab4d\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-nnnmx"
Sep 17 16:05:52.810571 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:05:52.810546    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4mpx\" (UniqueName: \"kubernetes.io/projected/9a0d05c9-f3bf-41dd-afd2-ce3f47a7ab4d-kube-api-access-x4mpx\") pod \"seaweedfs-tls-custom-8b44d767d-nnnmx\" (UID: \"9a0d05c9-f3bf-41dd-afd2-ce3f47a7ab4d\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-nnnmx"
Sep 17 16:05:52.810843 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:05:52.810589    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/9a0d05c9-f3bf-41dd-afd2-ce3f47a7ab4d-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-nnnmx\" (UID: \"9a0d05c9-f3bf-41dd-afd2-ce3f47a7ab4d\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-nnnmx"
Sep 17 16:05:52.810843 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:05:52.810645    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9a0d05c9-f3bf-41dd-afd2-ce3f47a7ab4d-data\") pod \"seaweedfs-tls-custom-8b44d767d-nnnmx\" (UID: \"9a0d05c9-f3bf-41dd-afd2-ce3f47a7ab4d\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-nnnmx"
Sep 17 16:05:52.811060 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:05:52.811038    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9a0d05c9-f3bf-41dd-afd2-ce3f47a7ab4d-data\") pod \"seaweedfs-tls-custom-8b44d767d-nnnmx\" (UID: \"9a0d05c9-f3bf-41dd-afd2-ce3f47a7ab4d\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-nnnmx"
Sep 17 16:05:52.813177 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:05:52.813158    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/9a0d05c9-f3bf-41dd-afd2-ce3f47a7ab4d-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-nnnmx\" (UID: \"9a0d05c9-f3bf-41dd-afd2-ce3f47a7ab4d\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-nnnmx"
Sep 17 16:05:52.819751 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:05:52.819729    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4mpx\" (UniqueName: \"kubernetes.io/projected/9a0d05c9-f3bf-41dd-afd2-ce3f47a7ab4d-kube-api-access-x4mpx\") pod \"seaweedfs-tls-custom-8b44d767d-nnnmx\" (UID: \"9a0d05c9-f3bf-41dd-afd2-ce3f47a7ab4d\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-nnnmx"
Sep 17 16:05:52.931054 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:05:52.931008    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-nnnmx"
Sep 17 16:05:53.199576 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:05:53.199493    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-nnnmx"]
Sep 17 16:05:53.202299 ip-10-0-135-43 kubenswrapper[2585]: W0917 16:05:53.202267    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a0d05c9_f3bf_41dd_afd2_ce3f47a7ab4d.slice/crio-31881c96f0bcf5f3c28b3feb704df0a57fd4aa0b4bbf1c6bd2af097a3d56d1c9 WatchSource:0}: Error finding container 31881c96f0bcf5f3c28b3feb704df0a57fd4aa0b4bbf1c6bd2af097a3d56d1c9: Status 404 returned error can't find the container with id 31881c96f0bcf5f3c28b3feb704df0a57fd4aa0b4bbf1c6bd2af097a3d56d1c9
Sep 17 16:05:53.578059 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:05:53.578023    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-nnnmx" event={"ID":"9a0d05c9-f3bf-41dd-afd2-ce3f47a7ab4d","Type":"ContainerStarted","Data":"e93810d55f4362d7598c47a70f4ad079de2e053a979fd4b2e87b99f42a2dd5ab"}
Sep 17 16:05:53.578233 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:05:53.578076    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-nnnmx" event={"ID":"9a0d05c9-f3bf-41dd-afd2-ce3f47a7ab4d","Type":"ContainerStarted","Data":"31881c96f0bcf5f3c28b3feb704df0a57fd4aa0b4bbf1c6bd2af097a3d56d1c9"}
Sep 17 16:05:53.598904 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:05:53.598850    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-8b44d767d-nnnmx" podStartSLOduration=1.357165326 podStartE2EDuration="1.598833443s" totalImagesPullingTime="241.668117ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:05:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:05:53.597368287 +0000 UTC m=+709.367700086" watchObservedRunningTime="2026-09-17 16:05:53.598833443 +0000 UTC m=+709.369165241"
Sep 17 16:05:54.512713 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:05:54.512679    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ea26ad2-9493-4b7b-a536-de88a1b39830" path="/var/lib/kubelet/pods/8ea26ad2-9493-4b7b-a536-de88a1b39830/volumes"
Sep 17 16:09:06.426363 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:09:06.426333    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2b47p_a1850a4d-4d01-4a71-8636-b5918f1426c2/ovn-acl-logging/0.log"
Sep 17 16:09:06.426886 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:09:06.426651    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2b47p_a1850a4d-4d01-4a71-8636-b5918f1426c2/ovn-acl-logging/0.log"
Sep 17 16:09:09.352265 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:09:09.352230    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9mttq"]
Sep 17 16:09:09.354695 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:09:09.354673    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9mttq"
Sep 17 16:09:09.357661 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:09:09.357635    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
Sep 17 16:09:09.357914 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:09:09.357891    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
Sep 17 16:09:09.358016 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:09:09.357924    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 16:09:09.358091 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:09:09.358073    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:09:09.358149 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:09:09.358118    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 16:09:09.358231 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:09:09.358119    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xkc4g\""
Sep 17 16:09:09.358890 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:09:09.358874    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 16:09:09.369533 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:09:09.369505    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9mttq"]
Sep 17 16:09:09.477913 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:09:09.477872    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb5dbc11-1689-4a57-be19-4a2aae1261f7-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-9mttq\" (UID: \"eb5dbc11-1689-4a57-be19-4a2aae1261f7\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9mttq"
Sep 17 16:09:09.478093 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:09:09.477921    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb5dbc11-1689-4a57-be19-4a2aae1261f7-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-9mttq\" (UID: \"eb5dbc11-1689-4a57-be19-4a2aae1261f7\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9mttq"
Sep 17 16:09:09.478093 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:09:09.477986    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb5dbc11-1689-4a57-be19-4a2aae1261f7-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-9mttq\" (UID: \"eb5dbc11-1689-4a57-be19-4a2aae1261f7\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9mttq"
Sep 17 16:09:09.478093 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:09:09.478041    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb5dbc11-1689-4a57-be19-4a2aae1261f7-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-9mttq\" (UID: \"eb5dbc11-1689-4a57-be19-4a2aae1261f7\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9mttq"
Sep 17 16:09:09.478093 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:09:09.478059    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9qpj\" (UniqueName: \"kubernetes.io/projected/eb5dbc11-1689-4a57-be19-4a2aae1261f7-kube-api-access-f9qpj\") pod \"isvc-predictive-xgboost-predictor-695f978978-9mttq\" (UID: \"eb5dbc11-1689-4a57-be19-4a2aae1261f7\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9mttq"
Sep 17 16:09:09.579052 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:09:09.579006    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb5dbc11-1689-4a57-be19-4a2aae1261f7-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-9mttq\" (UID: \"eb5dbc11-1689-4a57-be19-4a2aae1261f7\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9mttq"
Sep 17 16:09:09.579234 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:09:09.579066    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb5dbc11-1689-4a57-be19-4a2aae1261f7-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-9mttq\" (UID: \"eb5dbc11-1689-4a57-be19-4a2aae1261f7\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9mttq"
Sep 17 16:09:09.579234 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:09:09.579086    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9qpj\" (UniqueName: \"kubernetes.io/projected/eb5dbc11-1689-4a57-be19-4a2aae1261f7-kube-api-access-f9qpj\") pod \"isvc-predictive-xgboost-predictor-695f978978-9mttq\" (UID: \"eb5dbc11-1689-4a57-be19-4a2aae1261f7\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9mttq"
Sep 17 16:09:09.579234 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:09:09.579134    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb5dbc11-1689-4a57-be19-4a2aae1261f7-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-9mttq\" (UID: \"eb5dbc11-1689-4a57-be19-4a2aae1261f7\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9mttq"
Sep 17 16:09:09.579234 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:09:09.579159    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb5dbc11-1689-4a57-be19-4a2aae1261f7-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-9mttq\" (UID: \"eb5dbc11-1689-4a57-be19-4a2aae1261f7\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9mttq"
Sep 17 16:09:09.579535 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:09:09.579513    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb5dbc11-1689-4a57-be19-4a2aae1261f7-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-9mttq\" (UID: \"eb5dbc11-1689-4a57-be19-4a2aae1261f7\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9mttq"
Sep 17 16:09:09.579820 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:09:09.579798    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb5dbc11-1689-4a57-be19-4a2aae1261f7-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-9mttq\" (UID: \"eb5dbc11-1689-4a57-be19-4a2aae1261f7\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9mttq"
Sep 17 16:09:09.579820 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:09:09.579810    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb5dbc11-1689-4a57-be19-4a2aae1261f7-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-9mttq\" (UID: \"eb5dbc11-1689-4a57-be19-4a2aae1261f7\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9mttq"
Sep 17 16:09:09.581687 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:09:09.581664    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb5dbc11-1689-4a57-be19-4a2aae1261f7-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-9mttq\" (UID: \"eb5dbc11-1689-4a57-be19-4a2aae1261f7\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9mttq"
Sep 17 16:09:09.587355 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:09:09.587331    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9qpj\" (UniqueName: \"kubernetes.io/projected/eb5dbc11-1689-4a57-be19-4a2aae1261f7-kube-api-access-f9qpj\") pod \"isvc-predictive-xgboost-predictor-695f978978-9mttq\" (UID: \"eb5dbc11-1689-4a57-be19-4a2aae1261f7\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9mttq"
Sep 17 16:09:09.666556 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:09:09.666464    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9mttq"
Sep 17 16:09:09.948755 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:09:09.948729    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9mttq"]
Sep 17 16:09:09.951533 ip-10-0-135-43 kubenswrapper[2585]: W0917 16:09:09.951395    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb5dbc11_1689_4a57_be19_4a2aae1261f7.slice/crio-e637160aff082b28787a8e03787dbea8793ff0cc7e4b4ec51701c8e9fca43a79 WatchSource:0}: Error finding container e637160aff082b28787a8e03787dbea8793ff0cc7e4b4ec51701c8e9fca43a79: Status 404 returned error can't find the container with id e637160aff082b28787a8e03787dbea8793ff0cc7e4b4ec51701c8e9fca43a79
Sep 17 16:09:10.456555 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:09:10.456523    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9mttq" event={"ID":"eb5dbc11-1689-4a57-be19-4a2aae1261f7","Type":"ContainerStarted","Data":"e637160aff082b28787a8e03787dbea8793ff0cc7e4b4ec51701c8e9fca43a79"}
Sep 17 16:09:14.480561 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:09:14.480513    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9mttq" event={"ID":"eb5dbc11-1689-4a57-be19-4a2aae1261f7","Type":"ContainerStarted","Data":"e93d75d81a0117acf85cbc2b6db040715aeb683010241bfafbaef69b7da6af66"}
Sep 17 16:09:18.499886 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:09:18.499849    2585 generic.go:356] "Generic (PLEG): container finished" podID="eb5dbc11-1689-4a57-be19-4a2aae1261f7" containerID="e93d75d81a0117acf85cbc2b6db040715aeb683010241bfafbaef69b7da6af66" exitCode=0
Sep 17 16:09:18.500331 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:09:18.499924    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9mttq" event={"ID":"eb5dbc11-1689-4a57-be19-4a2aae1261f7","Type":"ContainerDied","Data":"e93d75d81a0117acf85cbc2b6db040715aeb683010241bfafbaef69b7da6af66"}
Sep 17 16:09:41.628793 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:09:41.628760    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9mttq" event={"ID":"eb5dbc11-1689-4a57-be19-4a2aae1261f7","Type":"ContainerStarted","Data":"ce730be620c6394dc7b909a9d0e3bc20dfe77fd98aa152aee4f28264b5b85207"}
Sep 17 16:09:44.645994 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:09:44.645904    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9mttq" event={"ID":"eb5dbc11-1689-4a57-be19-4a2aae1261f7","Type":"ContainerStarted","Data":"539fee129a4e71b6b793ae4bef6c72872e7afafd74796c71d2d23f2d34a9c3de"}
Sep 17 16:09:44.646489 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:09:44.646031    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9mttq"
Sep 17 16:09:44.678949 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:09:44.678894    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9mttq" podStartSLOduration=2.427009457 podStartE2EDuration="35.678878196s" totalImagesPullingTime="29.582040472s" totalInitContainerRuntime="3.669828267s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:09:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:09:44.678315416 +0000 UTC m=+940.448647216" watchObservedRunningTime="2026-09-17 16:09:44.678878196 +0000 UTC m=+940.449209995"
Sep 17 16:09:45.650144 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:09:45.650109    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9mttq"
Sep 17 16:09:45.651434 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:09:45.651392    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9mttq" podUID="eb5dbc11-1689-4a57-be19-4a2aae1261f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 16:09:46.654636 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:09:46.654590    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9mttq" podUID="eb5dbc11-1689-4a57-be19-4a2aae1261f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 16:09:51.659067 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:09:51.659036    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9mttq"
Sep 17 16:09:51.659589 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:09:51.659562    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9mttq" podUID="eb5dbc11-1689-4a57-be19-4a2aae1261f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 16:10:01.660077 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:10:01.660029    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9mttq" podUID="eb5dbc11-1689-4a57-be19-4a2aae1261f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 16:10:11.659697 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:10:11.659657    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9mttq" podUID="eb5dbc11-1689-4a57-be19-4a2aae1261f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 16:10:18.676076 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:10:18.675986    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-lmmn7"]
Sep 17 16:10:18.690074 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:10:18.690038    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-lmmn7"
Sep 17 16:10:18.692461 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:10:18.692415    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-lmmn7"]
Sep 17 16:10:18.692845 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:10:18.692825    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Sep 17 16:10:18.693413 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:10:18.693393    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Sep 17 16:10:18.806923 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:10:18.806885    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/118c84e2-49e5-4a5c-991f-33a2482ccdd1-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-lmmn7\" (UID: \"118c84e2-49e5-4a5c-991f-33a2482ccdd1\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-lmmn7"
Sep 17 16:10:18.806923 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:10:18.806921    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz9gv\" (UniqueName: \"kubernetes.io/projected/118c84e2-49e5-4a5c-991f-33a2482ccdd1-kube-api-access-xz9gv\") pod \"message-dumper-predictor-6575bcfdfd-lmmn7\" (UID: \"118c84e2-49e5-4a5c-991f-33a2482ccdd1\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-lmmn7"
Sep 17 16:10:18.807168 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:10:18.806951    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/118c84e2-49e5-4a5c-991f-33a2482ccdd1-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-lmmn7\" (UID: \"118c84e2-49e5-4a5c-991f-33a2482ccdd1\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-lmmn7"
Sep 17 16:10:18.907475 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:10:18.907412    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/118c84e2-49e5-4a5c-991f-33a2482ccdd1-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-lmmn7\" (UID: \"118c84e2-49e5-4a5c-991f-33a2482ccdd1\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-lmmn7"
Sep 17 16:10:18.907687 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:10:18.907484    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz9gv\" (UniqueName: \"kubernetes.io/projected/118c84e2-49e5-4a5c-991f-33a2482ccdd1-kube-api-access-xz9gv\") pod \"message-dumper-predictor-6575bcfdfd-lmmn7\" (UID: \"118c84e2-49e5-4a5c-991f-33a2482ccdd1\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-lmmn7"
Sep 17 16:10:18.907687 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:10:18.907524    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/118c84e2-49e5-4a5c-991f-33a2482ccdd1-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-lmmn7\" (UID: \"118c84e2-49e5-4a5c-991f-33a2482ccdd1\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-lmmn7"
Sep 17 16:10:18.907687 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:10:18.907546    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-predictor-serving-cert: secret "message-dumper-predictor-serving-cert" not found
Sep 17 16:10:18.907687 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:10:18.907654    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/118c84e2-49e5-4a5c-991f-33a2482ccdd1-proxy-tls podName:118c84e2-49e5-4a5c-991f-33a2482ccdd1 nodeName:}" failed. No retries permitted until 2026-09-17 16:10:19.407632766 +0000 UTC m=+975.177964545 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/118c84e2-49e5-4a5c-991f-33a2482ccdd1-proxy-tls") pod "message-dumper-predictor-6575bcfdfd-lmmn7" (UID: "118c84e2-49e5-4a5c-991f-33a2482ccdd1") : secret "message-dumper-predictor-serving-cert" not found
Sep 17 16:10:18.908251 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:10:18.908230    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/118c84e2-49e5-4a5c-991f-33a2482ccdd1-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-lmmn7\" (UID: \"118c84e2-49e5-4a5c-991f-33a2482ccdd1\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-lmmn7"
Sep 17 16:10:18.917601 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:10:18.917566    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz9gv\" (UniqueName: \"kubernetes.io/projected/118c84e2-49e5-4a5c-991f-33a2482ccdd1-kube-api-access-xz9gv\") pod \"message-dumper-predictor-6575bcfdfd-lmmn7\" (UID: \"118c84e2-49e5-4a5c-991f-33a2482ccdd1\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-lmmn7"
Sep 17 16:10:19.413936 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:10:19.413888    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/118c84e2-49e5-4a5c-991f-33a2482ccdd1-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-lmmn7\" (UID: \"118c84e2-49e5-4a5c-991f-33a2482ccdd1\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-lmmn7"
Sep 17 16:10:19.416551 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:10:19.416509    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/118c84e2-49e5-4a5c-991f-33a2482ccdd1-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-lmmn7\" (UID: \"118c84e2-49e5-4a5c-991f-33a2482ccdd1\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-lmmn7"
Sep 17 16:10:19.602755 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:10:19.602711    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-lmmn7"
Sep 17 16:10:19.879285 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:10:19.879216    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-lmmn7"]
Sep 17 16:10:19.881720 ip-10-0-135-43 kubenswrapper[2585]: W0917 16:10:19.881692    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod118c84e2_49e5_4a5c_991f_33a2482ccdd1.slice/crio-e6eb990d152666fb953f4d09e2203162fd51d1a9adf8caf3eca89e79b9625266 WatchSource:0}: Error finding container e6eb990d152666fb953f4d09e2203162fd51d1a9adf8caf3eca89e79b9625266: Status 404 returned error can't find the container with id e6eb990d152666fb953f4d09e2203162fd51d1a9adf8caf3eca89e79b9625266
Sep 17 16:10:20.810265 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:10:20.810228    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-lmmn7" event={"ID":"118c84e2-49e5-4a5c-991f-33a2482ccdd1","Type":"ContainerStarted","Data":"e6eb990d152666fb953f4d09e2203162fd51d1a9adf8caf3eca89e79b9625266"}
Sep 17 16:10:21.659841 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:10:21.659801    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9mttq" podUID="eb5dbc11-1689-4a57-be19-4a2aae1261f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 16:10:21.817119 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:10:21.817084    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-lmmn7" event={"ID":"118c84e2-49e5-4a5c-991f-33a2482ccdd1","Type":"ContainerStarted","Data":"9bce24fc14125e134c785bec0804b63bc3e2a7c2d4fdce9eb05b8c2a000ec203"}
Sep 17 16:10:21.817328 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:10:21.817143    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-lmmn7" event={"ID":"118c84e2-49e5-4a5c-991f-33a2482ccdd1","Type":"ContainerStarted","Data":"9849ef2a0a54319334b2a6128a3fcc5fe67f62c06d21eca7ec8ee03e5054a7f2"}
Sep 17 16:10:21.817328 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:10:21.817247    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-lmmn7"
Sep 17 16:10:21.857779 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:10:21.857724    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-lmmn7" podStartSLOduration=2.817112636 podStartE2EDuration="3.857707411s" totalImagesPullingTime="1.040594775s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:10:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:10:21.854916616 +0000 UTC m=+977.625248413" watchObservedRunningTime="2026-09-17 16:10:21.857707411 +0000 UTC m=+977.628039208"
Sep 17 16:10:22.822199 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:10:22.822165    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-lmmn7"
Sep 17 16:10:22.824017 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:10:22.823991    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-lmmn7"
Sep 17 16:10:29.836057 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:10:29.836028    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-lmmn7"
Sep 17 16:10:31.660295 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:10:31.660257    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9mttq" podUID="eb5dbc11-1689-4a57-be19-4a2aae1261f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 16:10:41.659647 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:10:41.659603    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9mttq" podUID="eb5dbc11-1689-4a57-be19-4a2aae1261f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 16:10:51.659743 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:10:51.659697    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9mttq" podUID="eb5dbc11-1689-4a57-be19-4a2aae1261f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 16:11:01.660577 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:01.660542    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9mttq"
Sep 17 16:11:09.922674 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:09.922634    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9mttq"]
Sep 17 16:11:09.923140 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:09.922984    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9mttq" podUID="eb5dbc11-1689-4a57-be19-4a2aae1261f7" containerName="kserve-container" containerID="cri-o://ce730be620c6394dc7b909a9d0e3bc20dfe77fd98aa152aee4f28264b5b85207" gracePeriod=30
Sep 17 16:11:09.923221 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:09.923179    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9mttq" podUID="eb5dbc11-1689-4a57-be19-4a2aae1261f7" containerName="kube-rbac-proxy" containerID="cri-o://539fee129a4e71b6b793ae4bef6c72872e7afafd74796c71d2d23f2d34a9c3de" gracePeriod=30
Sep 17 16:11:10.039495 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:10.039467    2585 generic.go:356] "Generic (PLEG): container finished" podID="eb5dbc11-1689-4a57-be19-4a2aae1261f7" containerID="539fee129a4e71b6b793ae4bef6c72872e7afafd74796c71d2d23f2d34a9c3de" exitCode=2
Sep 17 16:11:10.039623 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:10.039531    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9mttq" event={"ID":"eb5dbc11-1689-4a57-be19-4a2aae1261f7","Type":"ContainerDied","Data":"539fee129a4e71b6b793ae4bef6c72872e7afafd74796c71d2d23f2d34a9c3de"}
Sep 17 16:11:11.655084 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:11.655046    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9mttq" podUID="eb5dbc11-1689-4a57-be19-4a2aae1261f7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Sep 17 16:11:11.660460 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:11.660407    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9mttq" podUID="eb5dbc11-1689-4a57-be19-4a2aae1261f7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 17 16:11:15.161640 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:15.161603    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz"]
Sep 17 16:11:15.170198 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:15.170163    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz"
Sep 17 16:11:15.173171 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:15.173127    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
Sep 17 16:11:15.173322 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:15.173140    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
Sep 17 16:11:15.180565 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:15.180528    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz"]
Sep 17 16:11:15.274071 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:15.274031    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5bf696c-3e84-4814-92d5-127495c74bce-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz\" (UID: \"e5bf696c-3e84-4814-92d5-127495c74bce\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz"
Sep 17 16:11:15.274257 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:15.274098    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5bf696c-3e84-4814-92d5-127495c74bce-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz\" (UID: \"e5bf696c-3e84-4814-92d5-127495c74bce\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz"
Sep 17 16:11:15.274257 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:15.274136    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5bf696c-3e84-4814-92d5-127495c74bce-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz\" (UID: \"e5bf696c-3e84-4814-92d5-127495c74bce\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz"
Sep 17 16:11:15.274257 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:15.274218    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khbtc\" (UniqueName: \"kubernetes.io/projected/e5bf696c-3e84-4814-92d5-127495c74bce-kube-api-access-khbtc\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz\" (UID: \"e5bf696c-3e84-4814-92d5-127495c74bce\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz"
Sep 17 16:11:15.274402 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:15.274279    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e5bf696c-3e84-4814-92d5-127495c74bce-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz\" (UID: \"e5bf696c-3e84-4814-92d5-127495c74bce\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz"
Sep 17 16:11:15.374403 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:15.374376    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9mttq"
Sep 17 16:11:15.374811 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:15.374788    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e5bf696c-3e84-4814-92d5-127495c74bce-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz\" (UID: \"e5bf696c-3e84-4814-92d5-127495c74bce\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz"
Sep 17 16:11:15.374914 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:15.374837    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5bf696c-3e84-4814-92d5-127495c74bce-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz\" (UID: \"e5bf696c-3e84-4814-92d5-127495c74bce\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz"
Sep 17 16:11:15.374914 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:15.374875    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5bf696c-3e84-4814-92d5-127495c74bce-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz\" (UID: \"e5bf696c-3e84-4814-92d5-127495c74bce\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz"
Sep 17 16:11:15.374914 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:15.374901    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5bf696c-3e84-4814-92d5-127495c74bce-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz\" (UID: \"e5bf696c-3e84-4814-92d5-127495c74bce\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz"
Sep 17 16:11:15.375081 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:15.374946    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khbtc\" (UniqueName: \"kubernetes.io/projected/e5bf696c-3e84-4814-92d5-127495c74bce-kube-api-access-khbtc\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz\" (UID: \"e5bf696c-3e84-4814-92d5-127495c74bce\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz"
Sep 17 16:11:15.375338 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:15.375315    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5bf696c-3e84-4814-92d5-127495c74bce-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz\" (UID: \"e5bf696c-3e84-4814-92d5-127495c74bce\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz"
Sep 17 16:11:15.375588 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:15.375568    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5bf696c-3e84-4814-92d5-127495c74bce-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz\" (UID: \"e5bf696c-3e84-4814-92d5-127495c74bce\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz"
Sep 17 16:11:15.375685 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:15.375620    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e5bf696c-3e84-4814-92d5-127495c74bce-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz\" (UID: \"e5bf696c-3e84-4814-92d5-127495c74bce\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz"
Sep 17 16:11:15.377555 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:15.377534    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5bf696c-3e84-4814-92d5-127495c74bce-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz\" (UID: \"e5bf696c-3e84-4814-92d5-127495c74bce\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz"
Sep 17 16:11:15.384254 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:15.384232    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khbtc\" (UniqueName: \"kubernetes.io/projected/e5bf696c-3e84-4814-92d5-127495c74bce-kube-api-access-khbtc\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz\" (UID: \"e5bf696c-3e84-4814-92d5-127495c74bce\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz"
Sep 17 16:11:15.475229 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:15.475192    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb5dbc11-1689-4a57-be19-4a2aae1261f7-cabundle-cert\") pod \"eb5dbc11-1689-4a57-be19-4a2aae1261f7\" (UID: \"eb5dbc11-1689-4a57-be19-4a2aae1261f7\") "
Sep 17 16:11:15.475441 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:15.475244    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb5dbc11-1689-4a57-be19-4a2aae1261f7-kube-rbac-proxy-sar-config\") pod \"eb5dbc11-1689-4a57-be19-4a2aae1261f7\" (UID: \"eb5dbc11-1689-4a57-be19-4a2aae1261f7\") "
Sep 17 16:11:15.475441 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:15.475262    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb5dbc11-1689-4a57-be19-4a2aae1261f7-kserve-provision-location\") pod \"eb5dbc11-1689-4a57-be19-4a2aae1261f7\" (UID: \"eb5dbc11-1689-4a57-be19-4a2aae1261f7\") "
Sep 17 16:11:15.475441 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:15.475402    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb5dbc11-1689-4a57-be19-4a2aae1261f7-proxy-tls\") pod \"eb5dbc11-1689-4a57-be19-4a2aae1261f7\" (UID: \"eb5dbc11-1689-4a57-be19-4a2aae1261f7\") "
Sep 17 16:11:15.475635 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:15.475487    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9qpj\" (UniqueName: \"kubernetes.io/projected/eb5dbc11-1689-4a57-be19-4a2aae1261f7-kube-api-access-f9qpj\") pod \"eb5dbc11-1689-4a57-be19-4a2aae1261f7\" (UID: \"eb5dbc11-1689-4a57-be19-4a2aae1261f7\") "
Sep 17 16:11:15.475685 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:15.475635    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb5dbc11-1689-4a57-be19-4a2aae1261f7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "eb5dbc11-1689-4a57-be19-4a2aae1261f7" (UID: "eb5dbc11-1689-4a57-be19-4a2aae1261f7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:11:15.475685 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:15.475639    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb5dbc11-1689-4a57-be19-4a2aae1261f7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "eb5dbc11-1689-4a57-be19-4a2aae1261f7" (UID: "eb5dbc11-1689-4a57-be19-4a2aae1261f7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:11:15.475685 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:15.475667    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb5dbc11-1689-4a57-be19-4a2aae1261f7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "eb5dbc11-1689-4a57-be19-4a2aae1261f7" (UID: "eb5dbc11-1689-4a57-be19-4a2aae1261f7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:11:15.475806 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:15.475769    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb5dbc11-1689-4a57-be19-4a2aae1261f7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:11:15.475806 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:15.475781    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb5dbc11-1689-4a57-be19-4a2aae1261f7-kserve-provision-location\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:11:15.475806 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:15.475795    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb5dbc11-1689-4a57-be19-4a2aae1261f7-cabundle-cert\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:11:15.477779 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:15.477757    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb5dbc11-1689-4a57-be19-4a2aae1261f7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "eb5dbc11-1689-4a57-be19-4a2aae1261f7" (UID: "eb5dbc11-1689-4a57-be19-4a2aae1261f7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:11:15.477859 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:15.477774    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb5dbc11-1689-4a57-be19-4a2aae1261f7-kube-api-access-f9qpj" (OuterVolumeSpecName: "kube-api-access-f9qpj") pod "eb5dbc11-1689-4a57-be19-4a2aae1261f7" (UID: "eb5dbc11-1689-4a57-be19-4a2aae1261f7"). InnerVolumeSpecName "kube-api-access-f9qpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:11:15.483958 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:15.483937    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz"
Sep 17 16:11:15.577207 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:15.577166    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb5dbc11-1689-4a57-be19-4a2aae1261f7-proxy-tls\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:11:15.577207 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:15.577199    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9qpj\" (UniqueName: \"kubernetes.io/projected/eb5dbc11-1689-4a57-be19-4a2aae1261f7-kube-api-access-f9qpj\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:11:15.761538 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:15.761513    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz"]
Sep 17 16:11:15.763758 ip-10-0-135-43 kubenswrapper[2585]: W0917 16:11:15.763731    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5bf696c_3e84_4814_92d5_127495c74bce.slice/crio-b94f564d0b1fc767c880e95a53c1672d5d4e6846c2dac49ab8c75644f3bc4086 WatchSource:0}: Error finding container b94f564d0b1fc767c880e95a53c1672d5d4e6846c2dac49ab8c75644f3bc4086: Status 404 returned error can't find the container with id b94f564d0b1fc767c880e95a53c1672d5d4e6846c2dac49ab8c75644f3bc4086
Sep 17 16:11:16.071177 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:16.071077    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz" event={"ID":"e5bf696c-3e84-4814-92d5-127495c74bce","Type":"ContainerStarted","Data":"e625fada71a41a8d220bc32720fb365c4647148562ed3aabfd194d4bfccb595a"}
Sep 17 16:11:16.071177 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:16.071144    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz" event={"ID":"e5bf696c-3e84-4814-92d5-127495c74bce","Type":"ContainerStarted","Data":"b94f564d0b1fc767c880e95a53c1672d5d4e6846c2dac49ab8c75644f3bc4086"}
Sep 17 16:11:16.073127 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:16.073093    2585 generic.go:356] "Generic (PLEG): container finished" podID="eb5dbc11-1689-4a57-be19-4a2aae1261f7" containerID="ce730be620c6394dc7b909a9d0e3bc20dfe77fd98aa152aee4f28264b5b85207" exitCode=0
Sep 17 16:11:16.073275 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:16.073152    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9mttq"
Sep 17 16:11:16.073275 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:16.073161    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9mttq" event={"ID":"eb5dbc11-1689-4a57-be19-4a2aae1261f7","Type":"ContainerDied","Data":"ce730be620c6394dc7b909a9d0e3bc20dfe77fd98aa152aee4f28264b5b85207"}
Sep 17 16:11:16.073275 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:16.073218    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9mttq" event={"ID":"eb5dbc11-1689-4a57-be19-4a2aae1261f7","Type":"ContainerDied","Data":"e637160aff082b28787a8e03787dbea8793ff0cc7e4b4ec51701c8e9fca43a79"}
Sep 17 16:11:16.073275 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:16.073256    2585 scope.go:117] "RemoveContainer" containerID="539fee129a4e71b6b793ae4bef6c72872e7afafd74796c71d2d23f2d34a9c3de"
Sep 17 16:11:16.083650 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:16.083627    2585 scope.go:117] "RemoveContainer" containerID="ce730be620c6394dc7b909a9d0e3bc20dfe77fd98aa152aee4f28264b5b85207"
Sep 17 16:11:16.093859 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:16.093835    2585 scope.go:117] "RemoveContainer" containerID="e93d75d81a0117acf85cbc2b6db040715aeb683010241bfafbaef69b7da6af66"
Sep 17 16:11:16.104403 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:16.104382    2585 scope.go:117] "RemoveContainer" containerID="539fee129a4e71b6b793ae4bef6c72872e7afafd74796c71d2d23f2d34a9c3de"
Sep 17 16:11:16.104826 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:11:16.104803    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"539fee129a4e71b6b793ae4bef6c72872e7afafd74796c71d2d23f2d34a9c3de\": container with ID starting with 539fee129a4e71b6b793ae4bef6c72872e7afafd74796c71d2d23f2d34a9c3de not found: ID does not exist" containerID="539fee129a4e71b6b793ae4bef6c72872e7afafd74796c71d2d23f2d34a9c3de"
Sep 17 16:11:16.104910 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:16.104837    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"539fee129a4e71b6b793ae4bef6c72872e7afafd74796c71d2d23f2d34a9c3de"} err="failed to get container status \"539fee129a4e71b6b793ae4bef6c72872e7afafd74796c71d2d23f2d34a9c3de\": rpc error: code = NotFound desc = could not find container \"539fee129a4e71b6b793ae4bef6c72872e7afafd74796c71d2d23f2d34a9c3de\": container with ID starting with 539fee129a4e71b6b793ae4bef6c72872e7afafd74796c71d2d23f2d34a9c3de not found: ID does not exist"
Sep 17 16:11:16.104910 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:16.104861    2585 scope.go:117] "RemoveContainer" containerID="ce730be620c6394dc7b909a9d0e3bc20dfe77fd98aa152aee4f28264b5b85207"
Sep 17 16:11:16.105052 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:16.105028    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9mttq"]
Sep 17 16:11:16.105158 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:11:16.105139    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce730be620c6394dc7b909a9d0e3bc20dfe77fd98aa152aee4f28264b5b85207\": container with ID starting with ce730be620c6394dc7b909a9d0e3bc20dfe77fd98aa152aee4f28264b5b85207 not found: ID does not exist" containerID="ce730be620c6394dc7b909a9d0e3bc20dfe77fd98aa152aee4f28264b5b85207"
Sep 17 16:11:16.105218 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:16.105168    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce730be620c6394dc7b909a9d0e3bc20dfe77fd98aa152aee4f28264b5b85207"} err="failed to get container status \"ce730be620c6394dc7b909a9d0e3bc20dfe77fd98aa152aee4f28264b5b85207\": rpc error: code = NotFound desc = could not find container \"ce730be620c6394dc7b909a9d0e3bc20dfe77fd98aa152aee4f28264b5b85207\": container with ID starting with ce730be620c6394dc7b909a9d0e3bc20dfe77fd98aa152aee4f28264b5b85207 not found: ID does not exist"
Sep 17 16:11:16.105218 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:16.105188    2585 scope.go:117] "RemoveContainer" containerID="e93d75d81a0117acf85cbc2b6db040715aeb683010241bfafbaef69b7da6af66"
Sep 17 16:11:16.105447 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:11:16.105415    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e93d75d81a0117acf85cbc2b6db040715aeb683010241bfafbaef69b7da6af66\": container with ID starting with e93d75d81a0117acf85cbc2b6db040715aeb683010241bfafbaef69b7da6af66 not found: ID does not exist" containerID="e93d75d81a0117acf85cbc2b6db040715aeb683010241bfafbaef69b7da6af66"
Sep 17 16:11:16.105509 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:16.105453    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e93d75d81a0117acf85cbc2b6db040715aeb683010241bfafbaef69b7da6af66"} err="failed to get container status \"e93d75d81a0117acf85cbc2b6db040715aeb683010241bfafbaef69b7da6af66\": rpc error: code = NotFound desc = could not find container \"e93d75d81a0117acf85cbc2b6db040715aeb683010241bfafbaef69b7da6af66\": container with ID starting with e93d75d81a0117acf85cbc2b6db040715aeb683010241bfafbaef69b7da6af66 not found: ID does not exist"
Sep 17 16:11:16.109541 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:16.109515    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-9mttq"]
Sep 17 16:11:16.513359 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:16.513325    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb5dbc11-1689-4a57-be19-4a2aae1261f7" path="/var/lib/kubelet/pods/eb5dbc11-1689-4a57-be19-4a2aae1261f7/volumes"
Sep 17 16:11:20.095359 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:20.095328    2585 generic.go:356] "Generic (PLEG): container finished" podID="e5bf696c-3e84-4814-92d5-127495c74bce" containerID="e625fada71a41a8d220bc32720fb365c4647148562ed3aabfd194d4bfccb595a" exitCode=0
Sep 17 16:11:20.095768 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:20.095400    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz" event={"ID":"e5bf696c-3e84-4814-92d5-127495c74bce","Type":"ContainerDied","Data":"e625fada71a41a8d220bc32720fb365c4647148562ed3aabfd194d4bfccb595a"}
Sep 17 16:11:21.103607 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:21.103573    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz" event={"ID":"e5bf696c-3e84-4814-92d5-127495c74bce","Type":"ContainerStarted","Data":"b20c3e97c377d1737fff999198571b405d1af0fc9c4ff7b1bfdfad071a9064ba"}
Sep 17 16:11:21.104018 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:21.103636    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz" event={"ID":"e5bf696c-3e84-4814-92d5-127495c74bce","Type":"ContainerStarted","Data":"4c3863e7918f79052d818e3e425e255815776329324147229c555899a8a2cbb1"}
Sep 17 16:11:21.104018 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:21.103918    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz"
Sep 17 16:11:21.104091 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:21.104039    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz"
Sep 17 16:11:21.105453 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:21.105405    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz" podUID="e5bf696c-3e84-4814-92d5-127495c74bce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 16:11:21.125192 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:21.125091    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz" podStartSLOduration=2.025107368 podStartE2EDuration="6.125074914s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.099967546s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:11:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:11:21.123363107 +0000 UTC m=+1036.893694928" watchObservedRunningTime="2026-09-17 16:11:21.125074914 +0000 UTC m=+1036.895406716"
Sep 17 16:11:22.109002 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:22.108966    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz" podUID="e5bf696c-3e84-4814-92d5-127495c74bce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 16:11:27.113730 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:27.113697    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz"
Sep 17 16:11:27.114311 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:27.114280    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz" podUID="e5bf696c-3e84-4814-92d5-127495c74bce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 16:11:37.115047 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:37.115000    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz" podUID="e5bf696c-3e84-4814-92d5-127495c74bce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 16:11:47.114254 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:47.114207    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz" podUID="e5bf696c-3e84-4814-92d5-127495c74bce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 16:11:57.114609 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:11:57.114569    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz" podUID="e5bf696c-3e84-4814-92d5-127495c74bce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 16:12:07.114688 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:07.114651    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz" podUID="e5bf696c-3e84-4814-92d5-127495c74bce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 16:12:13.732617 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:13.732583    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-lmmn7_118c84e2-49e5-4a5c-991f-33a2482ccdd1/kserve-container/0.log"
Sep 17 16:12:13.943483 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:13.943450    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-lmmn7"]
Sep 17 16:12:13.943704 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:13.943683    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-lmmn7" podUID="118c84e2-49e5-4a5c-991f-33a2482ccdd1" containerName="kserve-container" containerID="cri-o://9849ef2a0a54319334b2a6128a3fcc5fe67f62c06d21eca7ec8ee03e5054a7f2" gracePeriod=30
Sep 17 16:12:13.943799 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:13.943754    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-lmmn7" podUID="118c84e2-49e5-4a5c-991f-33a2482ccdd1" containerName="kube-rbac-proxy" containerID="cri-o://9bce24fc14125e134c785bec0804b63bc3e2a7c2d4fdce9eb05b8c2a000ec203" gracePeriod=30
Sep 17 16:12:14.055841 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:14.055770    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-znz46"]
Sep 17 16:12:14.056183 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:14.056170    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb5dbc11-1689-4a57-be19-4a2aae1261f7" containerName="kserve-container"
Sep 17 16:12:14.056244 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:14.056186    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb5dbc11-1689-4a57-be19-4a2aae1261f7" containerName="kserve-container"
Sep 17 16:12:14.056244 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:14.056215    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb5dbc11-1689-4a57-be19-4a2aae1261f7" containerName="storage-initializer"
Sep 17 16:12:14.056244 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:14.056221    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb5dbc11-1689-4a57-be19-4a2aae1261f7" containerName="storage-initializer"
Sep 17 16:12:14.056244 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:14.056229    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb5dbc11-1689-4a57-be19-4a2aae1261f7" containerName="kube-rbac-proxy"
Sep 17 16:12:14.056244 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:14.056235    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb5dbc11-1689-4a57-be19-4a2aae1261f7" containerName="kube-rbac-proxy"
Sep 17 16:12:14.056394 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:14.056295    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb5dbc11-1689-4a57-be19-4a2aae1261f7" containerName="kube-rbac-proxy"
Sep 17 16:12:14.056394 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:14.056303    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb5dbc11-1689-4a57-be19-4a2aae1261f7" containerName="kserve-container"
Sep 17 16:12:14.059683 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:14.059658    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-znz46"
Sep 17 16:12:14.062300 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:14.062278    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Sep 17 16:12:14.062405 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:14.062286    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Sep 17 16:12:14.071295 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:14.071268    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-znz46"]
Sep 17 16:12:14.186626 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:14.186598    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/286dade9-7788-47b5-8d29-0056a17107d9-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-znz46\" (UID: \"286dade9-7788-47b5-8d29-0056a17107d9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-znz46"
Sep 17 16:12:14.186786 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:14.186643    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqxvx\" (UniqueName: \"kubernetes.io/projected/286dade9-7788-47b5-8d29-0056a17107d9-kube-api-access-tqxvx\") pod \"isvc-autogluon-v1-predictor-574b5b7495-znz46\" (UID: \"286dade9-7788-47b5-8d29-0056a17107d9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-znz46"
Sep 17 16:12:14.186786 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:14.186730    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/286dade9-7788-47b5-8d29-0056a17107d9-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-znz46\" (UID: \"286dade9-7788-47b5-8d29-0056a17107d9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-znz46"
Sep 17 16:12:14.186900 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:14.186797    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/286dade9-7788-47b5-8d29-0056a17107d9-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-znz46\" (UID: \"286dade9-7788-47b5-8d29-0056a17107d9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-znz46"
Sep 17 16:12:14.186900 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:14.186873    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/286dade9-7788-47b5-8d29-0056a17107d9-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-znz46\" (UID: \"286dade9-7788-47b5-8d29-0056a17107d9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-znz46"
Sep 17 16:12:14.194238 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:14.194218    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-lmmn7"
Sep 17 16:12:14.287973 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:14.287936    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/118c84e2-49e5-4a5c-991f-33a2482ccdd1-kube-rbac-proxy-sar-config\") pod \"118c84e2-49e5-4a5c-991f-33a2482ccdd1\" (UID: \"118c84e2-49e5-4a5c-991f-33a2482ccdd1\") "
Sep 17 16:12:14.287973 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:14.287980    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/118c84e2-49e5-4a5c-991f-33a2482ccdd1-proxy-tls\") pod \"118c84e2-49e5-4a5c-991f-33a2482ccdd1\" (UID: \"118c84e2-49e5-4a5c-991f-33a2482ccdd1\") "
Sep 17 16:12:14.288227 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:14.288011    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xz9gv\" (UniqueName: \"kubernetes.io/projected/118c84e2-49e5-4a5c-991f-33a2482ccdd1-kube-api-access-xz9gv\") pod \"118c84e2-49e5-4a5c-991f-33a2482ccdd1\" (UID: \"118c84e2-49e5-4a5c-991f-33a2482ccdd1\") "
Sep 17 16:12:14.288227 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:14.288095    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/286dade9-7788-47b5-8d29-0056a17107d9-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-znz46\" (UID: \"286dade9-7788-47b5-8d29-0056a17107d9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-znz46"
Sep 17 16:12:14.288227 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:14.288127    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqxvx\" (UniqueName: \"kubernetes.io/projected/286dade9-7788-47b5-8d29-0056a17107d9-kube-api-access-tqxvx\") pod \"isvc-autogluon-v1-predictor-574b5b7495-znz46\" (UID: \"286dade9-7788-47b5-8d29-0056a17107d9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-znz46"
Sep 17 16:12:14.288227 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:14.288161    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/286dade9-7788-47b5-8d29-0056a17107d9-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-znz46\" (UID: \"286dade9-7788-47b5-8d29-0056a17107d9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-znz46"
Sep 17 16:12:14.288489 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:12:14.288277    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v1-predictor-serving-cert: secret "isvc-autogluon-v1-predictor-serving-cert" not found
Sep 17 16:12:14.288489 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:14.288292    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/286dade9-7788-47b5-8d29-0056a17107d9-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-znz46\" (UID: \"286dade9-7788-47b5-8d29-0056a17107d9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-znz46"
Sep 17 16:12:14.288489 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:14.288336    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/118c84e2-49e5-4a5c-991f-33a2482ccdd1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "118c84e2-49e5-4a5c-991f-33a2482ccdd1" (UID: "118c84e2-49e5-4a5c-991f-33a2482ccdd1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:12:14.288489 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:12:14.288345    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/286dade9-7788-47b5-8d29-0056a17107d9-proxy-tls podName:286dade9-7788-47b5-8d29-0056a17107d9 nodeName:}" failed. No retries permitted until 2026-09-17 16:12:14.788324411 +0000 UTC m=+1090.558656214 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/286dade9-7788-47b5-8d29-0056a17107d9-proxy-tls") pod "isvc-autogluon-v1-predictor-574b5b7495-znz46" (UID: "286dade9-7788-47b5-8d29-0056a17107d9") : secret "isvc-autogluon-v1-predictor-serving-cert" not found
Sep 17 16:12:14.288489 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:14.288469    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/286dade9-7788-47b5-8d29-0056a17107d9-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-znz46\" (UID: \"286dade9-7788-47b5-8d29-0056a17107d9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-znz46"
Sep 17 16:12:14.288748 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:14.288537    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/118c84e2-49e5-4a5c-991f-33a2482ccdd1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:12:14.288748 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:14.288585    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/286dade9-7788-47b5-8d29-0056a17107d9-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-znz46\" (UID: \"286dade9-7788-47b5-8d29-0056a17107d9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-znz46"
Sep 17 16:12:14.289056 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:14.289031    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/286dade9-7788-47b5-8d29-0056a17107d9-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-znz46\" (UID: \"286dade9-7788-47b5-8d29-0056a17107d9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-znz46"
Sep 17 16:12:14.289134 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:14.289084    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/286dade9-7788-47b5-8d29-0056a17107d9-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-znz46\" (UID: \"286dade9-7788-47b5-8d29-0056a17107d9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-znz46"
Sep 17 16:12:14.290345 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:14.290312    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/118c84e2-49e5-4a5c-991f-33a2482ccdd1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "118c84e2-49e5-4a5c-991f-33a2482ccdd1" (UID: "118c84e2-49e5-4a5c-991f-33a2482ccdd1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:12:14.290475 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:14.290351    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/118c84e2-49e5-4a5c-991f-33a2482ccdd1-kube-api-access-xz9gv" (OuterVolumeSpecName: "kube-api-access-xz9gv") pod "118c84e2-49e5-4a5c-991f-33a2482ccdd1" (UID: "118c84e2-49e5-4a5c-991f-33a2482ccdd1"). InnerVolumeSpecName "kube-api-access-xz9gv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:12:14.297369 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:14.297343    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqxvx\" (UniqueName: \"kubernetes.io/projected/286dade9-7788-47b5-8d29-0056a17107d9-kube-api-access-tqxvx\") pod \"isvc-autogluon-v1-predictor-574b5b7495-znz46\" (UID: \"286dade9-7788-47b5-8d29-0056a17107d9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-znz46"
Sep 17 16:12:14.348243 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:14.348153    2585 generic.go:356] "Generic (PLEG): container finished" podID="118c84e2-49e5-4a5c-991f-33a2482ccdd1" containerID="9bce24fc14125e134c785bec0804b63bc3e2a7c2d4fdce9eb05b8c2a000ec203" exitCode=2
Sep 17 16:12:14.348243 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:14.348180    2585 generic.go:356] "Generic (PLEG): container finished" podID="118c84e2-49e5-4a5c-991f-33a2482ccdd1" containerID="9849ef2a0a54319334b2a6128a3fcc5fe67f62c06d21eca7ec8ee03e5054a7f2" exitCode=2
Sep 17 16:12:14.348243 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:14.348197    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-lmmn7"
Sep 17 16:12:14.348243 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:14.348239    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-lmmn7" event={"ID":"118c84e2-49e5-4a5c-991f-33a2482ccdd1","Type":"ContainerDied","Data":"9bce24fc14125e134c785bec0804b63bc3e2a7c2d4fdce9eb05b8c2a000ec203"}
Sep 17 16:12:14.348577 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:14.348312    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-lmmn7" event={"ID":"118c84e2-49e5-4a5c-991f-33a2482ccdd1","Type":"ContainerDied","Data":"9849ef2a0a54319334b2a6128a3fcc5fe67f62c06d21eca7ec8ee03e5054a7f2"}
Sep 17 16:12:14.348577 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:14.348345    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-lmmn7" event={"ID":"118c84e2-49e5-4a5c-991f-33a2482ccdd1","Type":"ContainerDied","Data":"e6eb990d152666fb953f4d09e2203162fd51d1a9adf8caf3eca89e79b9625266"}
Sep 17 16:12:14.348577 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:14.348380    2585 scope.go:117] "RemoveContainer" containerID="9bce24fc14125e134c785bec0804b63bc3e2a7c2d4fdce9eb05b8c2a000ec203"
Sep 17 16:12:14.358018 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:14.358001    2585 scope.go:117] "RemoveContainer" containerID="9849ef2a0a54319334b2a6128a3fcc5fe67f62c06d21eca7ec8ee03e5054a7f2"
Sep 17 16:12:14.368504 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:14.368471    2585 scope.go:117] "RemoveContainer" containerID="9bce24fc14125e134c785bec0804b63bc3e2a7c2d4fdce9eb05b8c2a000ec203"
Sep 17 16:12:14.368888 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:12:14.368866    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bce24fc14125e134c785bec0804b63bc3e2a7c2d4fdce9eb05b8c2a000ec203\": container with ID starting with 9bce24fc14125e134c785bec0804b63bc3e2a7c2d4fdce9eb05b8c2a000ec203 not found: ID does not exist" containerID="9bce24fc14125e134c785bec0804b63bc3e2a7c2d4fdce9eb05b8c2a000ec203"
Sep 17 16:12:14.368951 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:14.368896    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bce24fc14125e134c785bec0804b63bc3e2a7c2d4fdce9eb05b8c2a000ec203"} err="failed to get container status \"9bce24fc14125e134c785bec0804b63bc3e2a7c2d4fdce9eb05b8c2a000ec203\": rpc error: code = NotFound desc = could not find container \"9bce24fc14125e134c785bec0804b63bc3e2a7c2d4fdce9eb05b8c2a000ec203\": container with ID starting with 9bce24fc14125e134c785bec0804b63bc3e2a7c2d4fdce9eb05b8c2a000ec203 not found: ID does not exist"
Sep 17 16:12:14.368951 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:14.368914    2585 scope.go:117] "RemoveContainer" containerID="9849ef2a0a54319334b2a6128a3fcc5fe67f62c06d21eca7ec8ee03e5054a7f2"
Sep 17 16:12:14.369221 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:12:14.369202    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9849ef2a0a54319334b2a6128a3fcc5fe67f62c06d21eca7ec8ee03e5054a7f2\": container with ID starting with 9849ef2a0a54319334b2a6128a3fcc5fe67f62c06d21eca7ec8ee03e5054a7f2 not found: ID does not exist" containerID="9849ef2a0a54319334b2a6128a3fcc5fe67f62c06d21eca7ec8ee03e5054a7f2"
Sep 17 16:12:14.369277 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:14.369224    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9849ef2a0a54319334b2a6128a3fcc5fe67f62c06d21eca7ec8ee03e5054a7f2"} err="failed to get container status \"9849ef2a0a54319334b2a6128a3fcc5fe67f62c06d21eca7ec8ee03e5054a7f2\": rpc error: code = NotFound desc = could not find container \"9849ef2a0a54319334b2a6128a3fcc5fe67f62c06d21eca7ec8ee03e5054a7f2\": container with ID starting with 9849ef2a0a54319334b2a6128a3fcc5fe67f62c06d21eca7ec8ee03e5054a7f2 not found: ID does not exist"
Sep 17 16:12:14.369277 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:14.369240    2585 scope.go:117] "RemoveContainer" containerID="9bce24fc14125e134c785bec0804b63bc3e2a7c2d4fdce9eb05b8c2a000ec203"
Sep 17 16:12:14.369514 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:14.369492    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bce24fc14125e134c785bec0804b63bc3e2a7c2d4fdce9eb05b8c2a000ec203"} err="failed to get container status \"9bce24fc14125e134c785bec0804b63bc3e2a7c2d4fdce9eb05b8c2a000ec203\": rpc error: code = NotFound desc = could not find container \"9bce24fc14125e134c785bec0804b63bc3e2a7c2d4fdce9eb05b8c2a000ec203\": container with ID starting with 9bce24fc14125e134c785bec0804b63bc3e2a7c2d4fdce9eb05b8c2a000ec203 not found: ID does not exist"
Sep 17 16:12:14.369514 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:14.369512    2585 scope.go:117] "RemoveContainer" containerID="9849ef2a0a54319334b2a6128a3fcc5fe67f62c06d21eca7ec8ee03e5054a7f2"
Sep 17 16:12:14.369783 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:14.369767    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9849ef2a0a54319334b2a6128a3fcc5fe67f62c06d21eca7ec8ee03e5054a7f2"} err="failed to get container status \"9849ef2a0a54319334b2a6128a3fcc5fe67f62c06d21eca7ec8ee03e5054a7f2\": rpc error: code = NotFound desc = could not find container \"9849ef2a0a54319334b2a6128a3fcc5fe67f62c06d21eca7ec8ee03e5054a7f2\": container with ID starting with 9849ef2a0a54319334b2a6128a3fcc5fe67f62c06d21eca7ec8ee03e5054a7f2 not found: ID does not exist"
Sep 17 16:12:14.373151 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:14.373133    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-lmmn7"]
Sep 17 16:12:14.377070 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:14.377047    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-lmmn7"]
Sep 17 16:12:14.389558 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:14.389537    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/118c84e2-49e5-4a5c-991f-33a2482ccdd1-proxy-tls\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:12:14.389558 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:14.389556    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xz9gv\" (UniqueName: \"kubernetes.io/projected/118c84e2-49e5-4a5c-991f-33a2482ccdd1-kube-api-access-xz9gv\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:12:14.512482 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:14.512449    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="118c84e2-49e5-4a5c-991f-33a2482ccdd1" path="/var/lib/kubelet/pods/118c84e2-49e5-4a5c-991f-33a2482ccdd1/volumes"
Sep 17 16:12:14.793372 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:14.793337    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/286dade9-7788-47b5-8d29-0056a17107d9-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-znz46\" (UID: \"286dade9-7788-47b5-8d29-0056a17107d9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-znz46"
Sep 17 16:12:14.795792 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:14.795771    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/286dade9-7788-47b5-8d29-0056a17107d9-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-znz46\" (UID: \"286dade9-7788-47b5-8d29-0056a17107d9\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-znz46"
Sep 17 16:12:14.971674 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:14.971628    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-znz46"
Sep 17 16:12:15.250980 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:15.250954    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-znz46"]
Sep 17 16:12:15.253711 ip-10-0-135-43 kubenswrapper[2585]: W0917 16:12:15.253683    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod286dade9_7788_47b5_8d29_0056a17107d9.slice/crio-9e5ab677252dd8194cd94e97fdbed9df6bfbc301b62074dbf61c400e177690eb WatchSource:0}: Error finding container 9e5ab677252dd8194cd94e97fdbed9df6bfbc301b62074dbf61c400e177690eb: Status 404 returned error can't find the container with id 9e5ab677252dd8194cd94e97fdbed9df6bfbc301b62074dbf61c400e177690eb
Sep 17 16:12:15.355604 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:15.355560    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-znz46" event={"ID":"286dade9-7788-47b5-8d29-0056a17107d9","Type":"ContainerStarted","Data":"f96996527957f8f408592d53df96f1790d8c24d6a055545155a0852480f208e1"}
Sep 17 16:12:15.355801 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:15.355646    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-znz46" event={"ID":"286dade9-7788-47b5-8d29-0056a17107d9","Type":"ContainerStarted","Data":"9e5ab677252dd8194cd94e97fdbed9df6bfbc301b62074dbf61c400e177690eb"}
Sep 17 16:12:17.114699 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:17.114652    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz" podUID="e5bf696c-3e84-4814-92d5-127495c74bce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 16:12:27.114607 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:27.114562    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz" podUID="e5bf696c-3e84-4814-92d5-127495c74bce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 16:12:36.455973 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:36.455939    2585 generic.go:356] "Generic (PLEG): container finished" podID="286dade9-7788-47b5-8d29-0056a17107d9" containerID="f96996527957f8f408592d53df96f1790d8c24d6a055545155a0852480f208e1" exitCode=0
Sep 17 16:12:36.456365 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:36.456012    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-znz46" event={"ID":"286dade9-7788-47b5-8d29-0056a17107d9","Type":"ContainerDied","Data":"f96996527957f8f408592d53df96f1790d8c24d6a055545155a0852480f208e1"}
Sep 17 16:12:36.457241 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:36.457227    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 16:12:37.114577 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:37.114139    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz" podUID="e5bf696c-3e84-4814-92d5-127495c74bce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 16:12:47.115218 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:47.115184    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz"
Sep 17 16:12:56.301498 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:56.301463    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz"]
Sep 17 16:12:56.301966 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:56.301716    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz" podUID="e5bf696c-3e84-4814-92d5-127495c74bce" containerName="kserve-container" containerID="cri-o://4c3863e7918f79052d818e3e425e255815776329324147229c555899a8a2cbb1" gracePeriod=30
Sep 17 16:12:56.301966 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:56.301818    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz" podUID="e5bf696c-3e84-4814-92d5-127495c74bce" containerName="kube-rbac-proxy" containerID="cri-o://b20c3e97c377d1737fff999198571b405d1af0fc9c4ff7b1bfdfad071a9064ba" gracePeriod=30
Sep 17 16:12:56.570554 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:56.570471    2585 generic.go:356] "Generic (PLEG): container finished" podID="e5bf696c-3e84-4814-92d5-127495c74bce" containerID="b20c3e97c377d1737fff999198571b405d1af0fc9c4ff7b1bfdfad071a9064ba" exitCode=2
Sep 17 16:12:56.570554 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:56.570522    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz" event={"ID":"e5bf696c-3e84-4814-92d5-127495c74bce","Type":"ContainerDied","Data":"b20c3e97c377d1737fff999198571b405d1af0fc9c4ff7b1bfdfad071a9064ba"}
Sep 17 16:12:57.109949 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:57.109902    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz" podUID="e5bf696c-3e84-4814-92d5-127495c74bce" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Sep 17 16:12:57.114294 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:12:57.114257    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz" podUID="e5bf696c-3e84-4814-92d5-127495c74bce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 17 16:13:01.977543 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:13:01.977473    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz"
Sep 17 16:13:02.011851 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:13:02.011819    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-khbtc\" (UniqueName: \"kubernetes.io/projected/e5bf696c-3e84-4814-92d5-127495c74bce-kube-api-access-khbtc\") pod \"e5bf696c-3e84-4814-92d5-127495c74bce\" (UID: \"e5bf696c-3e84-4814-92d5-127495c74bce\") "
Sep 17 16:13:02.012039 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:13:02.011880    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e5bf696c-3e84-4814-92d5-127495c74bce-cabundle-cert\") pod \"e5bf696c-3e84-4814-92d5-127495c74bce\" (UID: \"e5bf696c-3e84-4814-92d5-127495c74bce\") "
Sep 17 16:13:02.012039 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:13:02.011993    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5bf696c-3e84-4814-92d5-127495c74bce-kserve-provision-location\") pod \"e5bf696c-3e84-4814-92d5-127495c74bce\" (UID: \"e5bf696c-3e84-4814-92d5-127495c74bce\") "
Sep 17 16:13:02.012039 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:13:02.012036    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5bf696c-3e84-4814-92d5-127495c74bce-proxy-tls\") pod \"e5bf696c-3e84-4814-92d5-127495c74bce\" (UID: \"e5bf696c-3e84-4814-92d5-127495c74bce\") "
Sep 17 16:13:02.012203 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:13:02.012066    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5bf696c-3e84-4814-92d5-127495c74bce-kube-rbac-proxy-sar-config\") pod \"e5bf696c-3e84-4814-92d5-127495c74bce\" (UID: \"e5bf696c-3e84-4814-92d5-127495c74bce\") "
Sep 17 16:13:02.012340 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:13:02.012310    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5bf696c-3e84-4814-92d5-127495c74bce-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e5bf696c-3e84-4814-92d5-127495c74bce" (UID: "e5bf696c-3e84-4814-92d5-127495c74bce"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:13:02.012397 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:13:02.012313    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5bf696c-3e84-4814-92d5-127495c74bce-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e5bf696c-3e84-4814-92d5-127495c74bce" (UID: "e5bf696c-3e84-4814-92d5-127495c74bce"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:13:02.012641 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:13:02.012606    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5bf696c-3e84-4814-92d5-127495c74bce-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e5bf696c-3e84-4814-92d5-127495c74bce" (UID: "e5bf696c-3e84-4814-92d5-127495c74bce"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:13:02.014138 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:13:02.014103    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5bf696c-3e84-4814-92d5-127495c74bce-kube-api-access-khbtc" (OuterVolumeSpecName: "kube-api-access-khbtc") pod "e5bf696c-3e84-4814-92d5-127495c74bce" (UID: "e5bf696c-3e84-4814-92d5-127495c74bce"). InnerVolumeSpecName "kube-api-access-khbtc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:13:02.014252 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:13:02.014207    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5bf696c-3e84-4814-92d5-127495c74bce-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e5bf696c-3e84-4814-92d5-127495c74bce" (UID: "e5bf696c-3e84-4814-92d5-127495c74bce"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:13:02.113268 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:13:02.113227    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khbtc\" (UniqueName: \"kubernetes.io/projected/e5bf696c-3e84-4814-92d5-127495c74bce-kube-api-access-khbtc\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:13:02.113268 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:13:02.113266    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e5bf696c-3e84-4814-92d5-127495c74bce-cabundle-cert\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:13:02.113498 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:13:02.113281    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5bf696c-3e84-4814-92d5-127495c74bce-kserve-provision-location\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:13:02.113498 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:13:02.113297    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e5bf696c-3e84-4814-92d5-127495c74bce-proxy-tls\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:13:02.113498 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:13:02.113314    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e5bf696c-3e84-4814-92d5-127495c74bce-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:13:02.605432 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:13:02.605390    2585 generic.go:356] "Generic (PLEG): container finished" podID="e5bf696c-3e84-4814-92d5-127495c74bce" containerID="4c3863e7918f79052d818e3e425e255815776329324147229c555899a8a2cbb1" exitCode=0
Sep 17 16:13:02.605602 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:13:02.605507    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz" event={"ID":"e5bf696c-3e84-4814-92d5-127495c74bce","Type":"ContainerDied","Data":"4c3863e7918f79052d818e3e425e255815776329324147229c555899a8a2cbb1"}
Sep 17 16:13:02.605602 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:13:02.605522    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz"
Sep 17 16:13:02.605602 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:13:02.605566    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz" event={"ID":"e5bf696c-3e84-4814-92d5-127495c74bce","Type":"ContainerDied","Data":"b94f564d0b1fc767c880e95a53c1672d5d4e6846c2dac49ab8c75644f3bc4086"}
Sep 17 16:13:02.605751 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:13:02.605610    2585 scope.go:117] "RemoveContainer" containerID="b20c3e97c377d1737fff999198571b405d1af0fc9c4ff7b1bfdfad071a9064ba"
Sep 17 16:13:02.616319 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:13:02.616290    2585 scope.go:117] "RemoveContainer" containerID="4c3863e7918f79052d818e3e425e255815776329324147229c555899a8a2cbb1"
Sep 17 16:13:02.627411 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:13:02.627383    2585 scope.go:117] "RemoveContainer" containerID="e625fada71a41a8d220bc32720fb365c4647148562ed3aabfd194d4bfccb595a"
Sep 17 16:13:02.627620 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:13:02.627594    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz"]
Sep 17 16:13:02.631498 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:13:02.631473    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-b89kz"]
Sep 17 16:13:02.638689 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:13:02.638662    2585 scope.go:117] "RemoveContainer" containerID="b20c3e97c377d1737fff999198571b405d1af0fc9c4ff7b1bfdfad071a9064ba"
Sep 17 16:13:02.639095 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:13:02.639061    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b20c3e97c377d1737fff999198571b405d1af0fc9c4ff7b1bfdfad071a9064ba\": container with ID starting with b20c3e97c377d1737fff999198571b405d1af0fc9c4ff7b1bfdfad071a9064ba not found: ID does not exist" containerID="b20c3e97c377d1737fff999198571b405d1af0fc9c4ff7b1bfdfad071a9064ba"
Sep 17 16:13:02.639273 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:13:02.639106    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b20c3e97c377d1737fff999198571b405d1af0fc9c4ff7b1bfdfad071a9064ba"} err="failed to get container status \"b20c3e97c377d1737fff999198571b405d1af0fc9c4ff7b1bfdfad071a9064ba\": rpc error: code = NotFound desc = could not find container \"b20c3e97c377d1737fff999198571b405d1af0fc9c4ff7b1bfdfad071a9064ba\": container with ID starting with b20c3e97c377d1737fff999198571b405d1af0fc9c4ff7b1bfdfad071a9064ba not found: ID does not exist"
Sep 17 16:13:02.639273 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:13:02.639136    2585 scope.go:117] "RemoveContainer" containerID="4c3863e7918f79052d818e3e425e255815776329324147229c555899a8a2cbb1"
Sep 17 16:13:02.639498 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:13:02.639474    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c3863e7918f79052d818e3e425e255815776329324147229c555899a8a2cbb1\": container with ID starting with 4c3863e7918f79052d818e3e425e255815776329324147229c555899a8a2cbb1 not found: ID does not exist" containerID="4c3863e7918f79052d818e3e425e255815776329324147229c555899a8a2cbb1"
Sep 17 16:13:02.639613 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:13:02.639503    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c3863e7918f79052d818e3e425e255815776329324147229c555899a8a2cbb1"} err="failed to get container status \"4c3863e7918f79052d818e3e425e255815776329324147229c555899a8a2cbb1\": rpc error: code = NotFound desc = could not find container \"4c3863e7918f79052d818e3e425e255815776329324147229c555899a8a2cbb1\": container with ID starting with 4c3863e7918f79052d818e3e425e255815776329324147229c555899a8a2cbb1 not found: ID does not exist"
Sep 17 16:13:02.639613 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:13:02.639526    2585 scope.go:117] "RemoveContainer" containerID="e625fada71a41a8d220bc32720fb365c4647148562ed3aabfd194d4bfccb595a"
Sep 17 16:13:02.639892 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:13:02.639867    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e625fada71a41a8d220bc32720fb365c4647148562ed3aabfd194d4bfccb595a\": container with ID starting with e625fada71a41a8d220bc32720fb365c4647148562ed3aabfd194d4bfccb595a not found: ID does not exist" containerID="e625fada71a41a8d220bc32720fb365c4647148562ed3aabfd194d4bfccb595a"
Sep 17 16:13:02.639950 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:13:02.639897    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e625fada71a41a8d220bc32720fb365c4647148562ed3aabfd194d4bfccb595a"} err="failed to get container status \"e625fada71a41a8d220bc32720fb365c4647148562ed3aabfd194d4bfccb595a\": rpc error: code = NotFound desc = could not find container \"e625fada71a41a8d220bc32720fb365c4647148562ed3aabfd194d4bfccb595a\": container with ID starting with e625fada71a41a8d220bc32720fb365c4647148562ed3aabfd194d4bfccb595a not found: ID does not exist"
Sep 17 16:13:04.512777 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:13:04.512745    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5bf696c-3e84-4814-92d5-127495c74bce" path="/var/lib/kubelet/pods/e5bf696c-3e84-4814-92d5-127495c74bce/volumes"
Sep 17 16:14:47.641904 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:14:47.641827    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2b47p_a1850a4d-4d01-4a71-8636-b5918f1426c2/ovn-acl-logging/0.log"
Sep 17 16:14:47.641904 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:14:47.641830    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2b47p_a1850a4d-4d01-4a71-8636-b5918f1426c2/ovn-acl-logging/0.log"
Sep 17 16:14:49.151124 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:14:49.151090    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-znz46" event={"ID":"286dade9-7788-47b5-8d29-0056a17107d9","Type":"ContainerStarted","Data":"1fbf71fa117d22f65fe0cd499983ebf84b324e3fea7208695da724167271f9c9"}
Sep 17 16:14:49.151640 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:14:49.151144    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-znz46" event={"ID":"286dade9-7788-47b5-8d29-0056a17107d9","Type":"ContainerStarted","Data":"81b500fa5d0275690c09c08114ab5fa49617ddd2cd8f6c813d56a06f0c903f2c"}
Sep 17 16:14:49.151640 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:14:49.151254    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-znz46"
Sep 17 16:14:49.151640 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:14:49.151312    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-znz46"
Sep 17 16:14:49.152634 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:14:49.152612    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-znz46" podUID="286dade9-7788-47b5-8d29-0056a17107d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 16:14:49.174896 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:14:49.174836    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-znz46" podStartSLOduration=2.737904535 podStartE2EDuration="2m35.174814248s" totalImagesPullingTime="2m11.823187146s" totalInitContainerRuntime="20.613722567s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:12:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:14:49.173217235 +0000 UTC m=+1244.943549071" watchObservedRunningTime="2026-09-17 16:14:49.174814248 +0000 UTC m=+1244.945146051"
Sep 17 16:14:50.156566 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:14:50.156530    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-znz46" podUID="286dade9-7788-47b5-8d29-0056a17107d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 16:14:55.161324 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:14:55.161289    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-znz46"
Sep 17 16:14:55.162026 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:14:55.161847    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-znz46" podUID="286dade9-7788-47b5-8d29-0056a17107d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 17 16:15:05.162508 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:05.162474    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-znz46"
Sep 17 16:15:20.811276 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:20.811241    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-znz46"]
Sep 17 16:15:20.811749 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:20.811517    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-znz46" podUID="286dade9-7788-47b5-8d29-0056a17107d9" containerName="kserve-container" containerID="cri-o://81b500fa5d0275690c09c08114ab5fa49617ddd2cd8f6c813d56a06f0c903f2c" gracePeriod=30
Sep 17 16:15:20.811749 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:20.811561    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-znz46" podUID="286dade9-7788-47b5-8d29-0056a17107d9" containerName="kube-rbac-proxy" containerID="cri-o://1fbf71fa117d22f65fe0cd499983ebf84b324e3fea7208695da724167271f9c9" gracePeriod=30
Sep 17 16:15:21.308643 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:21.308611    2585 generic.go:356] "Generic (PLEG): container finished" podID="286dade9-7788-47b5-8d29-0056a17107d9" containerID="1fbf71fa117d22f65fe0cd499983ebf84b324e3fea7208695da724167271f9c9" exitCode=2
Sep 17 16:15:21.308823 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:21.308690    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-znz46" event={"ID":"286dade9-7788-47b5-8d29-0056a17107d9","Type":"ContainerDied","Data":"1fbf71fa117d22f65fe0cd499983ebf84b324e3fea7208695da724167271f9c9"}
Sep 17 16:15:22.317555 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:22.317524    2585 generic.go:356] "Generic (PLEG): container finished" podID="286dade9-7788-47b5-8d29-0056a17107d9" containerID="81b500fa5d0275690c09c08114ab5fa49617ddd2cd8f6c813d56a06f0c903f2c" exitCode=0
Sep 17 16:15:22.317900 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:22.317597    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-znz46" event={"ID":"286dade9-7788-47b5-8d29-0056a17107d9","Type":"ContainerDied","Data":"81b500fa5d0275690c09c08114ab5fa49617ddd2cd8f6c813d56a06f0c903f2c"}
Sep 17 16:15:22.390770 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:22.390705    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-znz46"
Sep 17 16:15:22.473017 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:22.472983    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/286dade9-7788-47b5-8d29-0056a17107d9-cabundle-cert\") pod \"286dade9-7788-47b5-8d29-0056a17107d9\" (UID: \"286dade9-7788-47b5-8d29-0056a17107d9\") "
Sep 17 16:15:22.473222 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:22.473037    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/286dade9-7788-47b5-8d29-0056a17107d9-kube-rbac-proxy-sar-config\") pod \"286dade9-7788-47b5-8d29-0056a17107d9\" (UID: \"286dade9-7788-47b5-8d29-0056a17107d9\") "
Sep 17 16:15:22.473222 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:22.473087    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/286dade9-7788-47b5-8d29-0056a17107d9-proxy-tls\") pod \"286dade9-7788-47b5-8d29-0056a17107d9\" (UID: \"286dade9-7788-47b5-8d29-0056a17107d9\") "
Sep 17 16:15:22.473222 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:22.473132    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tqxvx\" (UniqueName: \"kubernetes.io/projected/286dade9-7788-47b5-8d29-0056a17107d9-kube-api-access-tqxvx\") pod \"286dade9-7788-47b5-8d29-0056a17107d9\" (UID: \"286dade9-7788-47b5-8d29-0056a17107d9\") "
Sep 17 16:15:22.473222 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:22.473171    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/286dade9-7788-47b5-8d29-0056a17107d9-kserve-provision-location\") pod \"286dade9-7788-47b5-8d29-0056a17107d9\" (UID: \"286dade9-7788-47b5-8d29-0056a17107d9\") "
Sep 17 16:15:22.473432 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:22.473377    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/286dade9-7788-47b5-8d29-0056a17107d9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "286dade9-7788-47b5-8d29-0056a17107d9" (UID: "286dade9-7788-47b5-8d29-0056a17107d9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:15:22.473552 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:22.473414    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/286dade9-7788-47b5-8d29-0056a17107d9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "286dade9-7788-47b5-8d29-0056a17107d9" (UID: "286dade9-7788-47b5-8d29-0056a17107d9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:15:22.475292 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:22.475264    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/286dade9-7788-47b5-8d29-0056a17107d9-kube-api-access-tqxvx" (OuterVolumeSpecName: "kube-api-access-tqxvx") pod "286dade9-7788-47b5-8d29-0056a17107d9" (UID: "286dade9-7788-47b5-8d29-0056a17107d9"). InnerVolumeSpecName "kube-api-access-tqxvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:15:22.475431 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:22.475333    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/286dade9-7788-47b5-8d29-0056a17107d9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "286dade9-7788-47b5-8d29-0056a17107d9" (UID: "286dade9-7788-47b5-8d29-0056a17107d9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:15:22.476742 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:22.476721    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/286dade9-7788-47b5-8d29-0056a17107d9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "286dade9-7788-47b5-8d29-0056a17107d9" (UID: "286dade9-7788-47b5-8d29-0056a17107d9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:15:22.574471 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:22.574410    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/286dade9-7788-47b5-8d29-0056a17107d9-proxy-tls\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:15:22.574471 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:22.574468    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqxvx\" (UniqueName: \"kubernetes.io/projected/286dade9-7788-47b5-8d29-0056a17107d9-kube-api-access-tqxvx\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:15:22.574471 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:22.574478    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/286dade9-7788-47b5-8d29-0056a17107d9-kserve-provision-location\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:15:22.574792 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:22.574487    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/286dade9-7788-47b5-8d29-0056a17107d9-cabundle-cert\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:15:22.574792 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:22.574497    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/286dade9-7788-47b5-8d29-0056a17107d9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:15:23.324170 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:23.324140    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-znz46"
Sep 17 16:15:23.324170 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:23.324158    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-znz46" event={"ID":"286dade9-7788-47b5-8d29-0056a17107d9","Type":"ContainerDied","Data":"9e5ab677252dd8194cd94e97fdbed9df6bfbc301b62074dbf61c400e177690eb"}
Sep 17 16:15:23.324744 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:23.324228    2585 scope.go:117] "RemoveContainer" containerID="1fbf71fa117d22f65fe0cd499983ebf84b324e3fea7208695da724167271f9c9"
Sep 17 16:15:23.332780 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:23.332757    2585 scope.go:117] "RemoveContainer" containerID="81b500fa5d0275690c09c08114ab5fa49617ddd2cd8f6c813d56a06f0c903f2c"
Sep 17 16:15:23.340442 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:23.340402    2585 scope.go:117] "RemoveContainer" containerID="f96996527957f8f408592d53df96f1790d8c24d6a055545155a0852480f208e1"
Sep 17 16:15:23.343355 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:23.343332    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-znz46"]
Sep 17 16:15:23.346502 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:23.346478    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-znz46"]
Sep 17 16:15:24.513112 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:24.513069    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="286dade9-7788-47b5-8d29-0056a17107d9" path="/var/lib/kubelet/pods/286dade9-7788-47b5-8d29-0056a17107d9/volumes"
Sep 17 16:15:26.022146 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:26.022112    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-84xms"]
Sep 17 16:15:26.022542 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:26.022480    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5bf696c-3e84-4814-92d5-127495c74bce" containerName="storage-initializer"
Sep 17 16:15:26.022542 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:26.022495    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5bf696c-3e84-4814-92d5-127495c74bce" containerName="storage-initializer"
Sep 17 16:15:26.022542 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:26.022507    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="118c84e2-49e5-4a5c-991f-33a2482ccdd1" containerName="kube-rbac-proxy"
Sep 17 16:15:26.022542 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:26.022514    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="118c84e2-49e5-4a5c-991f-33a2482ccdd1" containerName="kube-rbac-proxy"
Sep 17 16:15:26.022542 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:26.022520    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="286dade9-7788-47b5-8d29-0056a17107d9" containerName="kube-rbac-proxy"
Sep 17 16:15:26.022542 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:26.022527    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="286dade9-7788-47b5-8d29-0056a17107d9" containerName="kube-rbac-proxy"
Sep 17 16:15:26.022542 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:26.022534    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5bf696c-3e84-4814-92d5-127495c74bce" containerName="kserve-container"
Sep 17 16:15:26.022542 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:26.022540    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5bf696c-3e84-4814-92d5-127495c74bce" containerName="kserve-container"
Sep 17 16:15:26.022542 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:26.022546    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5bf696c-3e84-4814-92d5-127495c74bce" containerName="kube-rbac-proxy"
Sep 17 16:15:26.022811 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:26.022552    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5bf696c-3e84-4814-92d5-127495c74bce" containerName="kube-rbac-proxy"
Sep 17 16:15:26.022811 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:26.022562    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="286dade9-7788-47b5-8d29-0056a17107d9" containerName="storage-initializer"
Sep 17 16:15:26.022811 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:26.022570    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="286dade9-7788-47b5-8d29-0056a17107d9" containerName="storage-initializer"
Sep 17 16:15:26.022811 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:26.022576    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="286dade9-7788-47b5-8d29-0056a17107d9" containerName="kserve-container"
Sep 17 16:15:26.022811 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:26.022581    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="286dade9-7788-47b5-8d29-0056a17107d9" containerName="kserve-container"
Sep 17 16:15:26.022811 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:26.022592    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="118c84e2-49e5-4a5c-991f-33a2482ccdd1" containerName="kserve-container"
Sep 17 16:15:26.022811 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:26.022597    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="118c84e2-49e5-4a5c-991f-33a2482ccdd1" containerName="kserve-container"
Sep 17 16:15:26.022811 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:26.022651    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="286dade9-7788-47b5-8d29-0056a17107d9" containerName="kserve-container"
Sep 17 16:15:26.022811 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:26.022660    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="286dade9-7788-47b5-8d29-0056a17107d9" containerName="kube-rbac-proxy"
Sep 17 16:15:26.022811 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:26.022667    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5bf696c-3e84-4814-92d5-127495c74bce" containerName="kserve-container"
Sep 17 16:15:26.022811 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:26.022672    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="118c84e2-49e5-4a5c-991f-33a2482ccdd1" containerName="kserve-container"
Sep 17 16:15:26.022811 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:26.022678    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="118c84e2-49e5-4a5c-991f-33a2482ccdd1" containerName="kube-rbac-proxy"
Sep 17 16:15:26.022811 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:26.022684    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5bf696c-3e84-4814-92d5-127495c74bce" containerName="kube-rbac-proxy"
Sep 17 16:15:26.035684 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:26.035653    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-84xms"
Sep 17 16:15:26.040550 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:26.040522    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Sep 17 16:15:26.040955 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:26.040929    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 16:15:26.041091 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:26.040998    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 16:15:26.041091 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:26.041035    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 16:15:26.041317 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:26.041297    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:15:26.041493 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:26.041329    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Sep 17 16:15:26.041605 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:26.041407    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xkc4g\""
Sep 17 16:15:26.042893 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:26.042870    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-84xms"]
Sep 17 16:15:26.106439 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:26.106384    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlwv7\" (UniqueName: \"kubernetes.io/projected/79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0-kube-api-access-mlwv7\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-84xms\" (UID: \"79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-84xms"
Sep 17 16:15:26.106439 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:26.106443    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-84xms\" (UID: \"79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-84xms"
Sep 17 16:15:26.106643 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:26.106468    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-84xms\" (UID: \"79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-84xms"
Sep 17 16:15:26.106643 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:26.106504    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-84xms\" (UID: \"79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-84xms"
Sep 17 16:15:26.106643 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:26.106531    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-84xms\" (UID: \"79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-84xms"
Sep 17 16:15:26.207589 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:26.207541    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlwv7\" (UniqueName: \"kubernetes.io/projected/79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0-kube-api-access-mlwv7\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-84xms\" (UID: \"79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-84xms"
Sep 17 16:15:26.207589 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:26.207590    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-84xms\" (UID: \"79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-84xms"
Sep 17 16:15:26.207842 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:26.207773    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-84xms\" (UID: \"79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-84xms"
Sep 17 16:15:26.207842 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:26.207815    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-84xms\" (UID: \"79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-84xms"
Sep 17 16:15:26.207975 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:26.207854    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-84xms\" (UID: \"79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-84xms"
Sep 17 16:15:26.207975 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:15:26.207902    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-predictor-serving-cert: secret "isvc-autogluon-v2-predictor-serving-cert" not found
Sep 17 16:15:26.207975 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:15:26.207963    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0-proxy-tls podName:79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0 nodeName:}" failed. No retries permitted until 2026-09-17 16:15:26.707947296 +0000 UTC m=+1282.478279071 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0-proxy-tls") pod "isvc-autogluon-v2-predictor-9684b4cd9-84xms" (UID: "79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0") : secret "isvc-autogluon-v2-predictor-serving-cert" not found
Sep 17 16:15:26.208221 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:26.208182    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-84xms\" (UID: \"79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-84xms"
Sep 17 16:15:26.208491 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:26.208467    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-84xms\" (UID: \"79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-84xms"
Sep 17 16:15:26.208491 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:26.208484    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-84xms\" (UID: \"79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-84xms"
Sep 17 16:15:26.223659 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:26.223632    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlwv7\" (UniqueName: \"kubernetes.io/projected/79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0-kube-api-access-mlwv7\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-84xms\" (UID: \"79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-84xms"
Sep 17 16:15:26.712622 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:26.712569    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-84xms\" (UID: \"79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-84xms"
Sep 17 16:15:26.715053 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:26.715030    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-84xms\" (UID: \"79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-84xms"
Sep 17 16:15:26.948460 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:26.948380    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-84xms"
Sep 17 16:15:27.222301 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:27.222269    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-84xms"]
Sep 17 16:15:27.225813 ip-10-0-135-43 kubenswrapper[2585]: W0917 16:15:27.225786    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79ce7d71_463a_4c3f_8a08_5b6f0dd3a6f0.slice/crio-c2fe394436ebcbc91519fc53d6599aaf8bd9ea2bb26a730ad2d450f414fd5fac WatchSource:0}: Error finding container c2fe394436ebcbc91519fc53d6599aaf8bd9ea2bb26a730ad2d450f414fd5fac: Status 404 returned error can't find the container with id c2fe394436ebcbc91519fc53d6599aaf8bd9ea2bb26a730ad2d450f414fd5fac
Sep 17 16:15:27.345482 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:27.345441    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-84xms" event={"ID":"79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0","Type":"ContainerStarted","Data":"7dea591757c44a477e6d723a9c36902da859ae2ec0ea059b3481b3b1c3364082"}
Sep 17 16:15:27.345642 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:27.345512    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-84xms" event={"ID":"79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0","Type":"ContainerStarted","Data":"c2fe394436ebcbc91519fc53d6599aaf8bd9ea2bb26a730ad2d450f414fd5fac"}
Sep 17 16:15:57.482103 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:57.482063    2585 generic.go:356] "Generic (PLEG): container finished" podID="79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0" containerID="7dea591757c44a477e6d723a9c36902da859ae2ec0ea059b3481b3b1c3364082" exitCode=0
Sep 17 16:15:57.482736 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:57.482139    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-84xms" event={"ID":"79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0","Type":"ContainerDied","Data":"7dea591757c44a477e6d723a9c36902da859ae2ec0ea059b3481b3b1c3364082"}
Sep 17 16:15:58.489817 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:58.489782    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-84xms" event={"ID":"79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0","Type":"ContainerStarted","Data":"25b076da8d73616ac97d211ddaaa63d3525ae1f8f4f2a493962d8c11df111676"}
Sep 17 16:15:58.490213 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:58.489844    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-84xms" event={"ID":"79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0","Type":"ContainerStarted","Data":"c66c9bb07acc7713848a3bcf8c859e4759e4e7593bce7be5642a640c209f7213"}
Sep 17 16:15:58.490213 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:58.490031    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-84xms"
Sep 17 16:15:58.521986 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:58.521935    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-84xms" podStartSLOduration=3.153812341 podStartE2EDuration="32.521919469s" totalImagesPullingTime="0s" totalInitContainerRuntime="29.368107128s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:15:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:15:58.520849896 +0000 UTC m=+1314.291181696" watchObservedRunningTime="2026-09-17 16:15:58.521919469 +0000 UTC m=+1314.292251266"
Sep 17 16:15:59.498106 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:15:59.498075    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-84xms"
Sep 17 16:16:05.508817 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:16:05.508784    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-84xms"
Sep 17 16:17:36.544481 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:36.544437    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-84xms"
Sep 17 16:17:47.550684 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:47.550592    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-84xms" podUID="79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.36:8080/v2/models/isvc-autogluon-v2/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 17 16:17:52.464034 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:52.463994    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-84xms"]
Sep 17 16:17:52.464472 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:52.464226    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-84xms" podUID="79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0" containerName="kserve-container" containerID="cri-o://c66c9bb07acc7713848a3bcf8c859e4759e4e7593bce7be5642a640c209f7213" gracePeriod=30
Sep 17 16:17:52.464472 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:52.464271    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-84xms" podUID="79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0" containerName="kube-rbac-proxy" containerID="cri-o://25b076da8d73616ac97d211ddaaa63d3525ae1f8f4f2a493962d8c11df111676" gracePeriod=30
Sep 17 16:17:53.013797 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:53.013759    2585 generic.go:356] "Generic (PLEG): container finished" podID="79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0" containerID="25b076da8d73616ac97d211ddaaa63d3525ae1f8f4f2a493962d8c11df111676" exitCode=2
Sep 17 16:17:53.013962 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:53.013806    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-84xms" event={"ID":"79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0","Type":"ContainerDied","Data":"25b076da8d73616ac97d211ddaaa63d3525ae1f8f4f2a493962d8c11df111676"}
Sep 17 16:17:54.004732 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:54.004706    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-84xms"
Sep 17 16:17:54.020113 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:54.020086    2585 generic.go:356] "Generic (PLEG): container finished" podID="79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0" containerID="c66c9bb07acc7713848a3bcf8c859e4759e4e7593bce7be5642a640c209f7213" exitCode=0
Sep 17 16:17:54.020269 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:54.020125    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-84xms" event={"ID":"79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0","Type":"ContainerDied","Data":"c66c9bb07acc7713848a3bcf8c859e4759e4e7593bce7be5642a640c209f7213"}
Sep 17 16:17:54.020269 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:54.020145    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-84xms"
Sep 17 16:17:54.020269 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:54.020172    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-84xms" event={"ID":"79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0","Type":"ContainerDied","Data":"c2fe394436ebcbc91519fc53d6599aaf8bd9ea2bb26a730ad2d450f414fd5fac"}
Sep 17 16:17:54.020269 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:54.020200    2585 scope.go:117] "RemoveContainer" containerID="25b076da8d73616ac97d211ddaaa63d3525ae1f8f4f2a493962d8c11df111676"
Sep 17 16:17:54.032668 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:54.032576    2585 scope.go:117] "RemoveContainer" containerID="c66c9bb07acc7713848a3bcf8c859e4759e4e7593bce7be5642a640c209f7213"
Sep 17 16:17:54.041234 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:54.041208    2585 scope.go:117] "RemoveContainer" containerID="7dea591757c44a477e6d723a9c36902da859ae2ec0ea059b3481b3b1c3364082"
Sep 17 16:17:54.049342 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:54.049324    2585 scope.go:117] "RemoveContainer" containerID="25b076da8d73616ac97d211ddaaa63d3525ae1f8f4f2a493962d8c11df111676"
Sep 17 16:17:54.049653 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:17:54.049634    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25b076da8d73616ac97d211ddaaa63d3525ae1f8f4f2a493962d8c11df111676\": container with ID starting with 25b076da8d73616ac97d211ddaaa63d3525ae1f8f4f2a493962d8c11df111676 not found: ID does not exist" containerID="25b076da8d73616ac97d211ddaaa63d3525ae1f8f4f2a493962d8c11df111676"
Sep 17 16:17:54.049752 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:54.049661    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25b076da8d73616ac97d211ddaaa63d3525ae1f8f4f2a493962d8c11df111676"} err="failed to get container status \"25b076da8d73616ac97d211ddaaa63d3525ae1f8f4f2a493962d8c11df111676\": rpc error: code = NotFound desc = could not find container \"25b076da8d73616ac97d211ddaaa63d3525ae1f8f4f2a493962d8c11df111676\": container with ID starting with 25b076da8d73616ac97d211ddaaa63d3525ae1f8f4f2a493962d8c11df111676 not found: ID does not exist"
Sep 17 16:17:54.049752 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:54.049680    2585 scope.go:117] "RemoveContainer" containerID="c66c9bb07acc7713848a3bcf8c859e4759e4e7593bce7be5642a640c209f7213"
Sep 17 16:17:54.049976 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:17:54.049954    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c66c9bb07acc7713848a3bcf8c859e4759e4e7593bce7be5642a640c209f7213\": container with ID starting with c66c9bb07acc7713848a3bcf8c859e4759e4e7593bce7be5642a640c209f7213 not found: ID does not exist" containerID="c66c9bb07acc7713848a3bcf8c859e4759e4e7593bce7be5642a640c209f7213"
Sep 17 16:17:54.050022 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:54.049987    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c66c9bb07acc7713848a3bcf8c859e4759e4e7593bce7be5642a640c209f7213"} err="failed to get container status \"c66c9bb07acc7713848a3bcf8c859e4759e4e7593bce7be5642a640c209f7213\": rpc error: code = NotFound desc = could not find container \"c66c9bb07acc7713848a3bcf8c859e4759e4e7593bce7be5642a640c209f7213\": container with ID starting with c66c9bb07acc7713848a3bcf8c859e4759e4e7593bce7be5642a640c209f7213 not found: ID does not exist"
Sep 17 16:17:54.050022 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:54.050011    2585 scope.go:117] "RemoveContainer" containerID="7dea591757c44a477e6d723a9c36902da859ae2ec0ea059b3481b3b1c3364082"
Sep 17 16:17:54.050257 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:17:54.050242    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dea591757c44a477e6d723a9c36902da859ae2ec0ea059b3481b3b1c3364082\": container with ID starting with 7dea591757c44a477e6d723a9c36902da859ae2ec0ea059b3481b3b1c3364082 not found: ID does not exist" containerID="7dea591757c44a477e6d723a9c36902da859ae2ec0ea059b3481b3b1c3364082"
Sep 17 16:17:54.050303 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:54.050261    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dea591757c44a477e6d723a9c36902da859ae2ec0ea059b3481b3b1c3364082"} err="failed to get container status \"7dea591757c44a477e6d723a9c36902da859ae2ec0ea059b3481b3b1c3364082\": rpc error: code = NotFound desc = could not find container \"7dea591757c44a477e6d723a9c36902da859ae2ec0ea059b3481b3b1c3364082\": container with ID starting with 7dea591757c44a477e6d723a9c36902da859ae2ec0ea059b3481b3b1c3364082 not found: ID does not exist"
Sep 17 16:17:54.072561 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:54.072535    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0-cabundle-cert\") pod \"79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0\" (UID: \"79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0\") "
Sep 17 16:17:54.072683 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:54.072585    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mlwv7\" (UniqueName: \"kubernetes.io/projected/79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0-kube-api-access-mlwv7\") pod \"79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0\" (UID: \"79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0\") "
Sep 17 16:17:54.072729 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:54.072706    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0-kserve-provision-location\") pod \"79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0\" (UID: \"79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0\") "
Sep 17 16:17:54.072794 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:54.072779    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0-proxy-tls\") pod \"79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0\" (UID: \"79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0\") "
Sep 17 16:17:54.072837 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:54.072813    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0-kube-rbac-proxy-sar-config\") pod \"79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0\" (UID: \"79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0\") "
Sep 17 16:17:54.072998 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:54.072967    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0" (UID: "79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:17:54.073116 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:54.073078    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0-cabundle-cert\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:17:54.073233 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:54.073206    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0" (UID: "79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:17:54.074795 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:54.074762    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0" (UID: "79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:17:54.074879 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:54.074818    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0-kube-api-access-mlwv7" (OuterVolumeSpecName: "kube-api-access-mlwv7") pod "79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0" (UID: "79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0"). InnerVolumeSpecName "kube-api-access-mlwv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:17:54.079462 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:54.079414    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0" (UID: "79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:17:54.174194 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:54.174152    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:17:54.174194 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:54.174188    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlwv7\" (UniqueName: \"kubernetes.io/projected/79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0-kube-api-access-mlwv7\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:17:54.174194 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:54.174199    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0-kserve-provision-location\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:17:54.174471 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:54.174209    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0-proxy-tls\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:17:54.343563 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:54.343526    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-84xms"]
Sep 17 16:17:54.347047 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:54.347019    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-84xms"]
Sep 17 16:17:54.513136 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:54.513103    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0" path="/var/lib/kubelet/pods/79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0/volumes"
Sep 17 16:17:57.783614 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:57.783580    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7cbh5"]
Sep 17 16:17:57.784007 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:57.783906    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0" containerName="storage-initializer"
Sep 17 16:17:57.784007 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:57.783918    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0" containerName="storage-initializer"
Sep 17 16:17:57.784007 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:57.783929    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0" containerName="kserve-container"
Sep 17 16:17:57.784007 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:57.783935    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0" containerName="kserve-container"
Sep 17 16:17:57.784007 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:57.783948    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0" containerName="kube-rbac-proxy"
Sep 17 16:17:57.784007 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:57.783955    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0" containerName="kube-rbac-proxy"
Sep 17 16:17:57.784195 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:57.784017    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0" containerName="kserve-container"
Sep 17 16:17:57.784195 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:57.784024    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="79ce7d71-463a-4c3f-8a08-5b6f0dd3a6f0" containerName="kube-rbac-proxy"
Sep 17 16:17:57.786064 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:57.786047    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7cbh5"
Sep 17 16:17:57.794771 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:57.794753    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Sep 17 16:17:57.794951 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:57.794934    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 16:17:57.795002 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:57.794985    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 16:17:57.795257 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:57.795239    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 16:17:57.795335 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:57.795239    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Sep 17 16:17:57.795389 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:57.795362    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xkc4g\""
Sep 17 16:17:57.795472 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:57.795392    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:17:57.805831 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:57.805800    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13e8fa4e-9d68-4ac4-8be5-9f1e61b83215-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-7cbh5\" (UID: \"13e8fa4e-9d68-4ac4-8be5-9f1e61b83215\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7cbh5"
Sep 17 16:17:57.805937 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:57.805852    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13e8fa4e-9d68-4ac4-8be5-9f1e61b83215-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-7cbh5\" (UID: \"13e8fa4e-9d68-4ac4-8be5-9f1e61b83215\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7cbh5"
Sep 17 16:17:57.805937 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:57.805914    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13e8fa4e-9d68-4ac4-8be5-9f1e61b83215-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-7cbh5\" (UID: \"13e8fa4e-9d68-4ac4-8be5-9f1e61b83215\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7cbh5"
Sep 17 16:17:57.806043 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:57.805958    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13e8fa4e-9d68-4ac4-8be5-9f1e61b83215-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-7cbh5\" (UID: \"13e8fa4e-9d68-4ac4-8be5-9f1e61b83215\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7cbh5"
Sep 17 16:17:57.806043 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:57.806012    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqg5q\" (UniqueName: \"kubernetes.io/projected/13e8fa4e-9d68-4ac4-8be5-9f1e61b83215-kube-api-access-mqg5q\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-7cbh5\" (UID: \"13e8fa4e-9d68-4ac4-8be5-9f1e61b83215\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7cbh5"
Sep 17 16:17:57.819196 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:57.819166    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7cbh5"]
Sep 17 16:17:57.906602 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:57.906568    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqg5q\" (UniqueName: \"kubernetes.io/projected/13e8fa4e-9d68-4ac4-8be5-9f1e61b83215-kube-api-access-mqg5q\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-7cbh5\" (UID: \"13e8fa4e-9d68-4ac4-8be5-9f1e61b83215\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7cbh5"
Sep 17 16:17:57.906771 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:57.906615    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13e8fa4e-9d68-4ac4-8be5-9f1e61b83215-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-7cbh5\" (UID: \"13e8fa4e-9d68-4ac4-8be5-9f1e61b83215\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7cbh5"
Sep 17 16:17:57.906835 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:57.906798    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13e8fa4e-9d68-4ac4-8be5-9f1e61b83215-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-7cbh5\" (UID: \"13e8fa4e-9d68-4ac4-8be5-9f1e61b83215\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7cbh5"
Sep 17 16:17:57.906879 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:57.906861    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13e8fa4e-9d68-4ac4-8be5-9f1e61b83215-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-7cbh5\" (UID: \"13e8fa4e-9d68-4ac4-8be5-9f1e61b83215\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7cbh5"
Sep 17 16:17:57.906925 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:57.906910    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13e8fa4e-9d68-4ac4-8be5-9f1e61b83215-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-7cbh5\" (UID: \"13e8fa4e-9d68-4ac4-8be5-9f1e61b83215\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7cbh5"
Sep 17 16:17:57.907263 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:57.907238    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13e8fa4e-9d68-4ac4-8be5-9f1e61b83215-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-7cbh5\" (UID: \"13e8fa4e-9d68-4ac4-8be5-9f1e61b83215\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7cbh5"
Sep 17 16:17:57.907448 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:57.907399    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13e8fa4e-9d68-4ac4-8be5-9f1e61b83215-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-7cbh5\" (UID: \"13e8fa4e-9d68-4ac4-8be5-9f1e61b83215\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7cbh5"
Sep 17 16:17:57.907591 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:57.907543    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13e8fa4e-9d68-4ac4-8be5-9f1e61b83215-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-7cbh5\" (UID: \"13e8fa4e-9d68-4ac4-8be5-9f1e61b83215\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7cbh5"
Sep 17 16:17:57.909380 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:57.909363    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13e8fa4e-9d68-4ac4-8be5-9f1e61b83215-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-7cbh5\" (UID: \"13e8fa4e-9d68-4ac4-8be5-9f1e61b83215\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7cbh5"
Sep 17 16:17:57.917540 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:57.917517    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqg5q\" (UniqueName: \"kubernetes.io/projected/13e8fa4e-9d68-4ac4-8be5-9f1e61b83215-kube-api-access-mqg5q\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-7cbh5\" (UID: \"13e8fa4e-9d68-4ac4-8be5-9f1e61b83215\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7cbh5"
Sep 17 16:17:58.096652 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:58.096566    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7cbh5"
Sep 17 16:17:58.377866 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:58.377834    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7cbh5"]
Sep 17 16:17:58.378161 ip-10-0-135-43 kubenswrapper[2585]: W0917 16:17:58.378130    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13e8fa4e_9d68_4ac4_8be5_9f1e61b83215.slice/crio-1f5fcb1d3cc55a9067a0644c65f71bc09c8140720bf66304a08acee9266d4c3c WatchSource:0}: Error finding container 1f5fcb1d3cc55a9067a0644c65f71bc09c8140720bf66304a08acee9266d4c3c: Status 404 returned error can't find the container with id 1f5fcb1d3cc55a9067a0644c65f71bc09c8140720bf66304a08acee9266d4c3c
Sep 17 16:17:59.045620 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:59.045583    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7cbh5" event={"ID":"13e8fa4e-9d68-4ac4-8be5-9f1e61b83215","Type":"ContainerStarted","Data":"17019c169457850fb2a3680221d8acee156012ad611c8587305a4befe6a20651"}
Sep 17 16:17:59.046106 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:17:59.045660    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7cbh5" event={"ID":"13e8fa4e-9d68-4ac4-8be5-9f1e61b83215","Type":"ContainerStarted","Data":"1f5fcb1d3cc55a9067a0644c65f71bc09c8140720bf66304a08acee9266d4c3c"}
Sep 17 16:18:04.070662 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:18:04.070630    2585 generic.go:356] "Generic (PLEG): container finished" podID="13e8fa4e-9d68-4ac4-8be5-9f1e61b83215" containerID="17019c169457850fb2a3680221d8acee156012ad611c8587305a4befe6a20651" exitCode=0
Sep 17 16:18:04.071043 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:18:04.070697    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7cbh5" event={"ID":"13e8fa4e-9d68-4ac4-8be5-9f1e61b83215","Type":"ContainerDied","Data":"17019c169457850fb2a3680221d8acee156012ad611c8587305a4befe6a20651"}
Sep 17 16:18:05.078575 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:18:05.078538    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7cbh5" event={"ID":"13e8fa4e-9d68-4ac4-8be5-9f1e61b83215","Type":"ContainerStarted","Data":"1e55a1d6020abc51290e293cf035ee8f0ba1df9db5995675c89acea615844412"}
Sep 17 16:18:05.078963 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:18:05.078604    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7cbh5" event={"ID":"13e8fa4e-9d68-4ac4-8be5-9f1e61b83215","Type":"ContainerStarted","Data":"cf4579ee526781192e6e5a292e6327f62ca4cdd7ac50375c663377567b94405d"}
Sep 17 16:18:05.078963 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:18:05.078820    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7cbh5"
Sep 17 16:18:05.078963 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:18:05.078866    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7cbh5"
Sep 17 16:18:05.110440 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:18:05.110363    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7cbh5" podStartSLOduration=2.95481435 podStartE2EDuration="8.110344529s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.155530179s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:17:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:18:05.109357137 +0000 UTC m=+1440.879688948" watchObservedRunningTime="2026-09-17 16:18:05.110344529 +0000 UTC m=+1440.880676327"
Sep 17 16:18:11.090152 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:18:11.090122    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7cbh5"
Sep 17 16:19:41.543441 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:41.543384    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7cbh5"
Sep 17 16:19:47.679673 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:47.679563    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2b47p_a1850a4d-4d01-4a71-8636-b5918f1426c2/ovn-acl-logging/0.log"
Sep 17 16:19:47.681107 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:47.681088    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2b47p_a1850a4d-4d01-4a71-8636-b5918f1426c2/ovn-acl-logging/0.log"
Sep 17 16:19:52.549603 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:52.549560    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7cbh5" podUID="13e8fa4e-9d68-4ac4-8be5-9f1e61b83215" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.37:8080/v2/models/isvc-autogluon-v2-variants/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 17 16:19:54.564856 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:54.564821    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7cbh5"]
Sep 17 16:19:54.565333 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:54.565077    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7cbh5" podUID="13e8fa4e-9d68-4ac4-8be5-9f1e61b83215" containerName="kserve-container" containerID="cri-o://cf4579ee526781192e6e5a292e6327f62ca4cdd7ac50375c663377567b94405d" gracePeriod=30
Sep 17 16:19:54.565333 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:54.565140    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7cbh5" podUID="13e8fa4e-9d68-4ac4-8be5-9f1e61b83215" containerName="kube-rbac-proxy" containerID="cri-o://1e55a1d6020abc51290e293cf035ee8f0ba1df9db5995675c89acea615844412" gracePeriod=30
Sep 17 16:19:55.593789 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:55.593550    2585 generic.go:356] "Generic (PLEG): container finished" podID="13e8fa4e-9d68-4ac4-8be5-9f1e61b83215" containerID="1e55a1d6020abc51290e293cf035ee8f0ba1df9db5995675c89acea615844412" exitCode=2
Sep 17 16:19:55.593789 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:55.593703    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7cbh5" event={"ID":"13e8fa4e-9d68-4ac4-8be5-9f1e61b83215","Type":"ContainerDied","Data":"1e55a1d6020abc51290e293cf035ee8f0ba1df9db5995675c89acea615844412"}
Sep 17 16:19:56.026318 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:56.026293    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7cbh5"
Sep 17 16:19:56.105175 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:56.105143    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13e8fa4e-9d68-4ac4-8be5-9f1e61b83215-cabundle-cert\") pod \"13e8fa4e-9d68-4ac4-8be5-9f1e61b83215\" (UID: \"13e8fa4e-9d68-4ac4-8be5-9f1e61b83215\") "
Sep 17 16:19:56.105336 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:56.105186    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mqg5q\" (UniqueName: \"kubernetes.io/projected/13e8fa4e-9d68-4ac4-8be5-9f1e61b83215-kube-api-access-mqg5q\") pod \"13e8fa4e-9d68-4ac4-8be5-9f1e61b83215\" (UID: \"13e8fa4e-9d68-4ac4-8be5-9f1e61b83215\") "
Sep 17 16:19:56.105336 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:56.105222    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13e8fa4e-9d68-4ac4-8be5-9f1e61b83215-kserve-provision-location\") pod \"13e8fa4e-9d68-4ac4-8be5-9f1e61b83215\" (UID: \"13e8fa4e-9d68-4ac4-8be5-9f1e61b83215\") "
Sep 17 16:19:56.105336 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:56.105261    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13e8fa4e-9d68-4ac4-8be5-9f1e61b83215-kube-rbac-proxy-sar-config\") pod \"13e8fa4e-9d68-4ac4-8be5-9f1e61b83215\" (UID: \"13e8fa4e-9d68-4ac4-8be5-9f1e61b83215\") "
Sep 17 16:19:56.105336 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:56.105290    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13e8fa4e-9d68-4ac4-8be5-9f1e61b83215-proxy-tls\") pod \"13e8fa4e-9d68-4ac4-8be5-9f1e61b83215\" (UID: \"13e8fa4e-9d68-4ac4-8be5-9f1e61b83215\") "
Sep 17 16:19:56.105758 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:56.105606    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13e8fa4e-9d68-4ac4-8be5-9f1e61b83215-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "13e8fa4e-9d68-4ac4-8be5-9f1e61b83215" (UID: "13e8fa4e-9d68-4ac4-8be5-9f1e61b83215"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:19:56.105758 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:56.105623    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13e8fa4e-9d68-4ac4-8be5-9f1e61b83215-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "13e8fa4e-9d68-4ac4-8be5-9f1e61b83215" (UID: "13e8fa4e-9d68-4ac4-8be5-9f1e61b83215"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:19:56.107432 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:56.107388    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13e8fa4e-9d68-4ac4-8be5-9f1e61b83215-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "13e8fa4e-9d68-4ac4-8be5-9f1e61b83215" (UID: "13e8fa4e-9d68-4ac4-8be5-9f1e61b83215"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:19:56.107534 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:56.107482    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13e8fa4e-9d68-4ac4-8be5-9f1e61b83215-kube-api-access-mqg5q" (OuterVolumeSpecName: "kube-api-access-mqg5q") pod "13e8fa4e-9d68-4ac4-8be5-9f1e61b83215" (UID: "13e8fa4e-9d68-4ac4-8be5-9f1e61b83215"). InnerVolumeSpecName "kube-api-access-mqg5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:19:56.110486 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:56.110461    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13e8fa4e-9d68-4ac4-8be5-9f1e61b83215-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "13e8fa4e-9d68-4ac4-8be5-9f1e61b83215" (UID: "13e8fa4e-9d68-4ac4-8be5-9f1e61b83215"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:19:56.206881 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:56.206847    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13e8fa4e-9d68-4ac4-8be5-9f1e61b83215-cabundle-cert\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:19:56.206881 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:56.206883    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqg5q\" (UniqueName: \"kubernetes.io/projected/13e8fa4e-9d68-4ac4-8be5-9f1e61b83215-kube-api-access-mqg5q\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:19:56.207077 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:56.206895    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13e8fa4e-9d68-4ac4-8be5-9f1e61b83215-kserve-provision-location\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:19:56.207077 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:56.206908    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13e8fa4e-9d68-4ac4-8be5-9f1e61b83215-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:19:56.207077 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:56.206920    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13e8fa4e-9d68-4ac4-8be5-9f1e61b83215-proxy-tls\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:19:56.314438 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:56.314389    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd"]
Sep 17 16:19:56.314787 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:56.314772    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13e8fa4e-9d68-4ac4-8be5-9f1e61b83215" containerName="storage-initializer"
Sep 17 16:19:56.314787 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:56.314788    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="13e8fa4e-9d68-4ac4-8be5-9f1e61b83215" containerName="storage-initializer"
Sep 17 16:19:56.314902 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:56.314803    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13e8fa4e-9d68-4ac4-8be5-9f1e61b83215" containerName="kube-rbac-proxy"
Sep 17 16:19:56.314902 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:56.314809    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="13e8fa4e-9d68-4ac4-8be5-9f1e61b83215" containerName="kube-rbac-proxy"
Sep 17 16:19:56.314902 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:56.314818    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13e8fa4e-9d68-4ac4-8be5-9f1e61b83215" containerName="kserve-container"
Sep 17 16:19:56.314902 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:56.314824    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="13e8fa4e-9d68-4ac4-8be5-9f1e61b83215" containerName="kserve-container"
Sep 17 16:19:56.314902 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:56.314901    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="13e8fa4e-9d68-4ac4-8be5-9f1e61b83215" containerName="kube-rbac-proxy"
Sep 17 16:19:56.315064 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:56.314913    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="13e8fa4e-9d68-4ac4-8be5-9f1e61b83215" containerName="kserve-container"
Sep 17 16:19:56.319855 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:56.319831    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd"
Sep 17 16:19:56.322673 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:56.322644    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 17 16:19:56.322822 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:56.322805    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
Sep 17 16:19:56.330357 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:56.330331    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd"]
Sep 17 16:19:56.409267 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:56.409229    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rphd2\" (UniqueName: \"kubernetes.io/projected/6e134bad-b137-4571-b441-9f00e9152e99-kube-api-access-rphd2\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd\" (UID: \"6e134bad-b137-4571-b441-9f00e9152e99\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd"
Sep 17 16:19:56.409267 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:56.409268    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e134bad-b137-4571-b441-9f00e9152e99-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd\" (UID: \"6e134bad-b137-4571-b441-9f00e9152e99\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd"
Sep 17 16:19:56.409529 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:56.409304    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e134bad-b137-4571-b441-9f00e9152e99-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd\" (UID: \"6e134bad-b137-4571-b441-9f00e9152e99\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd"
Sep 17 16:19:56.409529 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:56.409363    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e134bad-b137-4571-b441-9f00e9152e99-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd\" (UID: \"6e134bad-b137-4571-b441-9f00e9152e99\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd"
Sep 17 16:19:56.409529 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:56.409392    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e134bad-b137-4571-b441-9f00e9152e99-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd\" (UID: \"6e134bad-b137-4571-b441-9f00e9152e99\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd"
Sep 17 16:19:56.509854 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:56.509744    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rphd2\" (UniqueName: \"kubernetes.io/projected/6e134bad-b137-4571-b441-9f00e9152e99-kube-api-access-rphd2\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd\" (UID: \"6e134bad-b137-4571-b441-9f00e9152e99\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd"
Sep 17 16:19:56.509854 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:56.509779    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e134bad-b137-4571-b441-9f00e9152e99-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd\" (UID: \"6e134bad-b137-4571-b441-9f00e9152e99\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd"
Sep 17 16:19:56.509854 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:56.509812    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e134bad-b137-4571-b441-9f00e9152e99-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd\" (UID: \"6e134bad-b137-4571-b441-9f00e9152e99\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd"
Sep 17 16:19:56.509854 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:56.509839    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e134bad-b137-4571-b441-9f00e9152e99-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd\" (UID: \"6e134bad-b137-4571-b441-9f00e9152e99\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd"
Sep 17 16:19:56.510196 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:56.509861    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e134bad-b137-4571-b441-9f00e9152e99-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd\" (UID: \"6e134bad-b137-4571-b441-9f00e9152e99\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd"
Sep 17 16:19:56.510249 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:19:56.510191    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-serving-cert: secret "sklearn-v2-mlserver-predictor-serving-cert" not found
Sep 17 16:19:56.510249 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:56.510199    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e134bad-b137-4571-b441-9f00e9152e99-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd\" (UID: \"6e134bad-b137-4571-b441-9f00e9152e99\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd"
Sep 17 16:19:56.510410 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:19:56.510262    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e134bad-b137-4571-b441-9f00e9152e99-proxy-tls podName:6e134bad-b137-4571-b441-9f00e9152e99 nodeName:}" failed. No retries permitted until 2026-09-17 16:19:57.010244237 +0000 UTC m=+1552.780576014 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6e134bad-b137-4571-b441-9f00e9152e99-proxy-tls") pod "sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd" (UID: "6e134bad-b137-4571-b441-9f00e9152e99") : secret "sklearn-v2-mlserver-predictor-serving-cert" not found
Sep 17 16:19:56.510523 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:56.510504    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e134bad-b137-4571-b441-9f00e9152e99-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd\" (UID: \"6e134bad-b137-4571-b441-9f00e9152e99\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd"
Sep 17 16:19:56.510612 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:56.510594    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e134bad-b137-4571-b441-9f00e9152e99-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd\" (UID: \"6e134bad-b137-4571-b441-9f00e9152e99\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd"
Sep 17 16:19:56.518943 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:56.518910    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rphd2\" (UniqueName: \"kubernetes.io/projected/6e134bad-b137-4571-b441-9f00e9152e99-kube-api-access-rphd2\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd\" (UID: \"6e134bad-b137-4571-b441-9f00e9152e99\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd"
Sep 17 16:19:56.600335 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:56.600295    2585 generic.go:356] "Generic (PLEG): container finished" podID="13e8fa4e-9d68-4ac4-8be5-9f1e61b83215" containerID="cf4579ee526781192e6e5a292e6327f62ca4cdd7ac50375c663377567b94405d" exitCode=0
Sep 17 16:19:56.600803 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:56.600362    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7cbh5"
Sep 17 16:19:56.600803 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:56.600370    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7cbh5" event={"ID":"13e8fa4e-9d68-4ac4-8be5-9f1e61b83215","Type":"ContainerDied","Data":"cf4579ee526781192e6e5a292e6327f62ca4cdd7ac50375c663377567b94405d"}
Sep 17 16:19:56.600803 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:56.600459    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7cbh5" event={"ID":"13e8fa4e-9d68-4ac4-8be5-9f1e61b83215","Type":"ContainerDied","Data":"1f5fcb1d3cc55a9067a0644c65f71bc09c8140720bf66304a08acee9266d4c3c"}
Sep 17 16:19:56.600803 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:56.600487    2585 scope.go:117] "RemoveContainer" containerID="1e55a1d6020abc51290e293cf035ee8f0ba1df9db5995675c89acea615844412"
Sep 17 16:19:56.609525 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:56.609501    2585 scope.go:117] "RemoveContainer" containerID="cf4579ee526781192e6e5a292e6327f62ca4cdd7ac50375c663377567b94405d"
Sep 17 16:19:56.618467 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:56.618398    2585 scope.go:117] "RemoveContainer" containerID="17019c169457850fb2a3680221d8acee156012ad611c8587305a4befe6a20651"
Sep 17 16:19:56.619574 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:56.619484    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7cbh5"]
Sep 17 16:19:56.623612 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:56.623589    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-7cbh5"]
Sep 17 16:19:56.627518 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:56.627496    2585 scope.go:117] "RemoveContainer" containerID="1e55a1d6020abc51290e293cf035ee8f0ba1df9db5995675c89acea615844412"
Sep 17 16:19:56.627860 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:19:56.627841    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e55a1d6020abc51290e293cf035ee8f0ba1df9db5995675c89acea615844412\": container with ID starting with 1e55a1d6020abc51290e293cf035ee8f0ba1df9db5995675c89acea615844412 not found: ID does not exist" containerID="1e55a1d6020abc51290e293cf035ee8f0ba1df9db5995675c89acea615844412"
Sep 17 16:19:56.627907 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:56.627870    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e55a1d6020abc51290e293cf035ee8f0ba1df9db5995675c89acea615844412"} err="failed to get container status \"1e55a1d6020abc51290e293cf035ee8f0ba1df9db5995675c89acea615844412\": rpc error: code = NotFound desc = could not find container \"1e55a1d6020abc51290e293cf035ee8f0ba1df9db5995675c89acea615844412\": container with ID starting with 1e55a1d6020abc51290e293cf035ee8f0ba1df9db5995675c89acea615844412 not found: ID does not exist"
Sep 17 16:19:56.627907 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:56.627889    2585 scope.go:117] "RemoveContainer" containerID="cf4579ee526781192e6e5a292e6327f62ca4cdd7ac50375c663377567b94405d"
Sep 17 16:19:56.628155 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:19:56.628132    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf4579ee526781192e6e5a292e6327f62ca4cdd7ac50375c663377567b94405d\": container with ID starting with cf4579ee526781192e6e5a292e6327f62ca4cdd7ac50375c663377567b94405d not found: ID does not exist" containerID="cf4579ee526781192e6e5a292e6327f62ca4cdd7ac50375c663377567b94405d"
Sep 17 16:19:56.628202 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:56.628161    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf4579ee526781192e6e5a292e6327f62ca4cdd7ac50375c663377567b94405d"} err="failed to get container status \"cf4579ee526781192e6e5a292e6327f62ca4cdd7ac50375c663377567b94405d\": rpc error: code = NotFound desc = could not find container \"cf4579ee526781192e6e5a292e6327f62ca4cdd7ac50375c663377567b94405d\": container with ID starting with cf4579ee526781192e6e5a292e6327f62ca4cdd7ac50375c663377567b94405d not found: ID does not exist"
Sep 17 16:19:56.628202 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:56.628178    2585 scope.go:117] "RemoveContainer" containerID="17019c169457850fb2a3680221d8acee156012ad611c8587305a4befe6a20651"
Sep 17 16:19:56.628481 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:19:56.628463    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17019c169457850fb2a3680221d8acee156012ad611c8587305a4befe6a20651\": container with ID starting with 17019c169457850fb2a3680221d8acee156012ad611c8587305a4befe6a20651 not found: ID does not exist" containerID="17019c169457850fb2a3680221d8acee156012ad611c8587305a4befe6a20651"
Sep 17 16:19:56.628526 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:56.628486    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17019c169457850fb2a3680221d8acee156012ad611c8587305a4befe6a20651"} err="failed to get container status \"17019c169457850fb2a3680221d8acee156012ad611c8587305a4befe6a20651\": rpc error: code = NotFound desc = could not find container \"17019c169457850fb2a3680221d8acee156012ad611c8587305a4befe6a20651\": container with ID starting with 17019c169457850fb2a3680221d8acee156012ad611c8587305a4befe6a20651 not found: ID does not exist"
Sep 17 16:19:57.015563 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:57.015522    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e134bad-b137-4571-b441-9f00e9152e99-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd\" (UID: \"6e134bad-b137-4571-b441-9f00e9152e99\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd"
Sep 17 16:19:57.018014 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:57.017988    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e134bad-b137-4571-b441-9f00e9152e99-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd\" (UID: \"6e134bad-b137-4571-b441-9f00e9152e99\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd"
Sep 17 16:19:57.231104 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:57.231066    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd"
Sep 17 16:19:57.507720 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:57.507688    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd"]
Sep 17 16:19:57.510764 ip-10-0-135-43 kubenswrapper[2585]: W0917 16:19:57.510737    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e134bad_b137_4571_b441_9f00e9152e99.slice/crio-f7bb76414c0c6f9d3d2738907e43a3e2549068379aae471e60c3cf1d736fca7b WatchSource:0}: Error finding container f7bb76414c0c6f9d3d2738907e43a3e2549068379aae471e60c3cf1d736fca7b: Status 404 returned error can't find the container with id f7bb76414c0c6f9d3d2738907e43a3e2549068379aae471e60c3cf1d736fca7b
Sep 17 16:19:57.608390 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:57.608352    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd" event={"ID":"6e134bad-b137-4571-b441-9f00e9152e99","Type":"ContainerStarted","Data":"12b6f66495c58bdfcf0a7647c3ceb28337f3f91edcaa9c9b1330d4e3351f19e3"}
Sep 17 16:19:57.608782 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:57.608411    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd" event={"ID":"6e134bad-b137-4571-b441-9f00e9152e99","Type":"ContainerStarted","Data":"f7bb76414c0c6f9d3d2738907e43a3e2549068379aae471e60c3cf1d736fca7b"}
Sep 17 16:19:58.513156 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:58.513119    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13e8fa4e-9d68-4ac4-8be5-9f1e61b83215" path="/var/lib/kubelet/pods/13e8fa4e-9d68-4ac4-8be5-9f1e61b83215/volumes"
Sep 17 16:19:59.730404 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:59.730364    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z22v8"]
Sep 17 16:19:59.734278 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:59.734251    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z22v8"
Sep 17 16:19:59.738050 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:59.738027    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Sep 17 16:19:59.739118 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:59.739097    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Sep 17 16:19:59.753541 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:59.753508    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z22v8"]
Sep 17 16:19:59.838712 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:59.838669    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/04d20574-15c0-48e5-b35f-2c32c6d3f6e7-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-z22v8\" (UID: \"04d20574-15c0-48e5-b35f-2c32c6d3f6e7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z22v8"
Sep 17 16:19:59.838712 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:59.838719    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5kdw\" (UniqueName: \"kubernetes.io/projected/04d20574-15c0-48e5-b35f-2c32c6d3f6e7-kube-api-access-g5kdw\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-z22v8\" (UID: \"04d20574-15c0-48e5-b35f-2c32c6d3f6e7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z22v8"
Sep 17 16:19:59.838931 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:59.838745    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04d20574-15c0-48e5-b35f-2c32c6d3f6e7-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-z22v8\" (UID: \"04d20574-15c0-48e5-b35f-2c32c6d3f6e7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z22v8"
Sep 17 16:19:59.838931 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:59.838769    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/04d20574-15c0-48e5-b35f-2c32c6d3f6e7-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-z22v8\" (UID: \"04d20574-15c0-48e5-b35f-2c32c6d3f6e7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z22v8"
Sep 17 16:19:59.838931 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:59.838865    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04d20574-15c0-48e5-b35f-2c32c6d3f6e7-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-z22v8\" (UID: \"04d20574-15c0-48e5-b35f-2c32c6d3f6e7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z22v8"
Sep 17 16:19:59.939767 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:59.939722    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5kdw\" (UniqueName: \"kubernetes.io/projected/04d20574-15c0-48e5-b35f-2c32c6d3f6e7-kube-api-access-g5kdw\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-z22v8\" (UID: \"04d20574-15c0-48e5-b35f-2c32c6d3f6e7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z22v8"
Sep 17 16:19:59.939767 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:59.939765    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04d20574-15c0-48e5-b35f-2c32c6d3f6e7-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-z22v8\" (UID: \"04d20574-15c0-48e5-b35f-2c32c6d3f6e7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z22v8"
Sep 17 16:19:59.940023 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:59.939784    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/04d20574-15c0-48e5-b35f-2c32c6d3f6e7-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-z22v8\" (UID: \"04d20574-15c0-48e5-b35f-2c32c6d3f6e7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z22v8"
Sep 17 16:19:59.940023 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:59.939841    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04d20574-15c0-48e5-b35f-2c32c6d3f6e7-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-z22v8\" (UID: \"04d20574-15c0-48e5-b35f-2c32c6d3f6e7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z22v8"
Sep 17 16:19:59.940023 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:59.939873    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/04d20574-15c0-48e5-b35f-2c32c6d3f6e7-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-z22v8\" (UID: \"04d20574-15c0-48e5-b35f-2c32c6d3f6e7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z22v8"
Sep 17 16:19:59.940355 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:59.940185    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04d20574-15c0-48e5-b35f-2c32c6d3f6e7-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-z22v8\" (UID: \"04d20574-15c0-48e5-b35f-2c32c6d3f6e7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z22v8"
Sep 17 16:19:59.940555 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:59.940536    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/04d20574-15c0-48e5-b35f-2c32c6d3f6e7-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-z22v8\" (UID: \"04d20574-15c0-48e5-b35f-2c32c6d3f6e7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z22v8"
Sep 17 16:19:59.940630 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:59.940587    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/04d20574-15c0-48e5-b35f-2c32c6d3f6e7-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-z22v8\" (UID: \"04d20574-15c0-48e5-b35f-2c32c6d3f6e7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z22v8"
Sep 17 16:19:59.942501 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:59.942476    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04d20574-15c0-48e5-b35f-2c32c6d3f6e7-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-z22v8\" (UID: \"04d20574-15c0-48e5-b35f-2c32c6d3f6e7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z22v8"
Sep 17 16:19:59.949370 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:19:59.949347    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5kdw\" (UniqueName: \"kubernetes.io/projected/04d20574-15c0-48e5-b35f-2c32c6d3f6e7-kube-api-access-g5kdw\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-z22v8\" (UID: \"04d20574-15c0-48e5-b35f-2c32c6d3f6e7\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z22v8"
Sep 17 16:20:00.045454 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:20:00.045323    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z22v8"
Sep 17 16:20:00.325833 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:20:00.325767    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z22v8"]
Sep 17 16:20:00.327980 ip-10-0-135-43 kubenswrapper[2585]: W0917 16:20:00.327950    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04d20574_15c0_48e5_b35f_2c32c6d3f6e7.slice/crio-4305fafe8799d40dc325d64d906cdcd4c173d7c1058b1a4da5dd53d0ad543d0f WatchSource:0}: Error finding container 4305fafe8799d40dc325d64d906cdcd4c173d7c1058b1a4da5dd53d0ad543d0f: Status 404 returned error can't find the container with id 4305fafe8799d40dc325d64d906cdcd4c173d7c1058b1a4da5dd53d0ad543d0f
Sep 17 16:20:00.624757 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:20:00.624668    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z22v8" event={"ID":"04d20574-15c0-48e5-b35f-2c32c6d3f6e7","Type":"ContainerStarted","Data":"1716d6f0d9af77bbbd4a5783980e105c0320a1e4e737216f26b5328c96761d91"}
Sep 17 16:20:00.624757 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:20:00.624729    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z22v8" event={"ID":"04d20574-15c0-48e5-b35f-2c32c6d3f6e7","Type":"ContainerStarted","Data":"4305fafe8799d40dc325d64d906cdcd4c173d7c1058b1a4da5dd53d0ad543d0f"}
Sep 17 16:20:01.630767 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:20:01.630732    2585 generic.go:356] "Generic (PLEG): container finished" podID="6e134bad-b137-4571-b441-9f00e9152e99" containerID="12b6f66495c58bdfcf0a7647c3ceb28337f3f91edcaa9c9b1330d4e3351f19e3" exitCode=0
Sep 17 16:20:01.631128 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:20:01.630773    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd" event={"ID":"6e134bad-b137-4571-b441-9f00e9152e99","Type":"ContainerDied","Data":"12b6f66495c58bdfcf0a7647c3ceb28337f3f91edcaa9c9b1330d4e3351f19e3"}
Sep 17 16:20:01.631989 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:20:01.631975    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 16:20:08.692507 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:20:08.692467    2585 generic.go:356] "Generic (PLEG): container finished" podID="04d20574-15c0-48e5-b35f-2c32c6d3f6e7" containerID="1716d6f0d9af77bbbd4a5783980e105c0320a1e4e737216f26b5328c96761d91" exitCode=0
Sep 17 16:20:08.692967 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:20:08.692554    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z22v8" event={"ID":"04d20574-15c0-48e5-b35f-2c32c6d3f6e7","Type":"ContainerDied","Data":"1716d6f0d9af77bbbd4a5783980e105c0320a1e4e737216f26b5328c96761d91"}
Sep 17 16:20:16.738973 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:20:16.738924    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z22v8" event={"ID":"04d20574-15c0-48e5-b35f-2c32c6d3f6e7","Type":"ContainerStarted","Data":"dca34b4920a012f5915b59ce23cbe1ed739a9eba8828508473f8035e6ce17321"}
Sep 17 16:20:16.739406 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:20:16.738994    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z22v8" event={"ID":"04d20574-15c0-48e5-b35f-2c32c6d3f6e7","Type":"ContainerStarted","Data":"e24e9d2e7323d1cd695eda477b8b06680454399e62630b8e618aedc222a89978"}
Sep 17 16:20:16.739406 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:20:16.739211    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z22v8"
Sep 17 16:20:16.739406 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:20:16.739260    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z22v8"
Sep 17 16:20:16.775869 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:20:16.775801    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z22v8" podStartSLOduration=12.996422226 podStartE2EDuration="17.775778746s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.77935652s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:19:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:20:16.773099496 +0000 UTC m=+1572.543431294" watchObservedRunningTime="2026-09-17 16:20:16.775778746 +0000 UTC m=+1572.546110546"
Sep 17 16:20:22.753067 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:20:22.753015    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z22v8"
Sep 17 16:21:49.548554 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:21:49.548513    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z22v8"
Sep 17 16:22:07.805871 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:22:07.805831    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z22v8"]
Sep 17 16:22:07.806468 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:22:07.806102    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z22v8" podUID="04d20574-15c0-48e5-b35f-2c32c6d3f6e7" containerName="kserve-container" containerID="cri-o://e24e9d2e7323d1cd695eda477b8b06680454399e62630b8e618aedc222a89978" gracePeriod=30
Sep 17 16:22:07.806468 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:22:07.806148    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z22v8" podUID="04d20574-15c0-48e5-b35f-2c32c6d3f6e7" containerName="kube-rbac-proxy" containerID="cri-o://dca34b4920a012f5915b59ce23cbe1ed739a9eba8828508473f8035e6ce17321" gracePeriod=30
Sep 17 16:22:09.507803 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:22:09.507751    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z22v8" podUID="04d20574-15c0-48e5-b35f-2c32c6d3f6e7" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.39:8080/v2/models/isvc-autogluon-v2-noslash/ready\": dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 17 16:22:09.939609 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:22:09.939582    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z22v8"
Sep 17 16:22:10.032119 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:22:10.032036    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04d20574-15c0-48e5-b35f-2c32c6d3f6e7-proxy-tls\") pod \"04d20574-15c0-48e5-b35f-2c32c6d3f6e7\" (UID: \"04d20574-15c0-48e5-b35f-2c32c6d3f6e7\") "
Sep 17 16:22:10.032119 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:22:10.032091    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04d20574-15c0-48e5-b35f-2c32c6d3f6e7-kserve-provision-location\") pod \"04d20574-15c0-48e5-b35f-2c32c6d3f6e7\" (UID: \"04d20574-15c0-48e5-b35f-2c32c6d3f6e7\") "
Sep 17 16:22:10.032119 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:22:10.032115    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/04d20574-15c0-48e5-b35f-2c32c6d3f6e7-kube-rbac-proxy-sar-config\") pod \"04d20574-15c0-48e5-b35f-2c32c6d3f6e7\" (UID: \"04d20574-15c0-48e5-b35f-2c32c6d3f6e7\") "
Sep 17 16:22:10.032408 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:22:10.032226    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/04d20574-15c0-48e5-b35f-2c32c6d3f6e7-cabundle-cert\") pod \"04d20574-15c0-48e5-b35f-2c32c6d3f6e7\" (UID: \"04d20574-15c0-48e5-b35f-2c32c6d3f6e7\") "
Sep 17 16:22:10.032408 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:22:10.032272    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g5kdw\" (UniqueName: \"kubernetes.io/projected/04d20574-15c0-48e5-b35f-2c32c6d3f6e7-kube-api-access-g5kdw\") pod \"04d20574-15c0-48e5-b35f-2c32c6d3f6e7\" (UID: \"04d20574-15c0-48e5-b35f-2c32c6d3f6e7\") "
Sep 17 16:22:10.032558 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:22:10.032506    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04d20574-15c0-48e5-b35f-2c32c6d3f6e7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "04d20574-15c0-48e5-b35f-2c32c6d3f6e7" (UID: "04d20574-15c0-48e5-b35f-2c32c6d3f6e7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:22:10.032635 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:22:10.032620    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/04d20574-15c0-48e5-b35f-2c32c6d3f6e7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:22:10.032687 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:22:10.032625    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04d20574-15c0-48e5-b35f-2c32c6d3f6e7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "04d20574-15c0-48e5-b35f-2c32c6d3f6e7" (UID: "04d20574-15c0-48e5-b35f-2c32c6d3f6e7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:22:10.034531 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:22:10.034499    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04d20574-15c0-48e5-b35f-2c32c6d3f6e7-kube-api-access-g5kdw" (OuterVolumeSpecName: "kube-api-access-g5kdw") pod "04d20574-15c0-48e5-b35f-2c32c6d3f6e7" (UID: "04d20574-15c0-48e5-b35f-2c32c6d3f6e7"). InnerVolumeSpecName "kube-api-access-g5kdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:22:10.034731 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:22:10.034708    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04d20574-15c0-48e5-b35f-2c32c6d3f6e7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "04d20574-15c0-48e5-b35f-2c32c6d3f6e7" (UID: "04d20574-15c0-48e5-b35f-2c32c6d3f6e7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:22:10.036168 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:22:10.036148    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04d20574-15c0-48e5-b35f-2c32c6d3f6e7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "04d20574-15c0-48e5-b35f-2c32c6d3f6e7" (UID: "04d20574-15c0-48e5-b35f-2c32c6d3f6e7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:22:10.133435 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:22:10.133384    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/04d20574-15c0-48e5-b35f-2c32c6d3f6e7-cabundle-cert\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:22:10.133435 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:22:10.133431    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5kdw\" (UniqueName: \"kubernetes.io/projected/04d20574-15c0-48e5-b35f-2c32c6d3f6e7-kube-api-access-g5kdw\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:22:10.133632 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:22:10.133447    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/04d20574-15c0-48e5-b35f-2c32c6d3f6e7-proxy-tls\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:22:10.133632 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:22:10.133460    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04d20574-15c0-48e5-b35f-2c32c6d3f6e7-kserve-provision-location\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:22:10.349921 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:22:10.349834    2585 generic.go:356] "Generic (PLEG): container finished" podID="04d20574-15c0-48e5-b35f-2c32c6d3f6e7" containerID="dca34b4920a012f5915b59ce23cbe1ed739a9eba8828508473f8035e6ce17321" exitCode=2
Sep 17 16:22:10.349921 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:22:10.349857    2585 generic.go:356] "Generic (PLEG): container finished" podID="04d20574-15c0-48e5-b35f-2c32c6d3f6e7" containerID="e24e9d2e7323d1cd695eda477b8b06680454399e62630b8e618aedc222a89978" exitCode=0
Sep 17 16:22:10.349921 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:22:10.349908    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z22v8"
Sep 17 16:22:10.350194 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:22:10.349912    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z22v8" event={"ID":"04d20574-15c0-48e5-b35f-2c32c6d3f6e7","Type":"ContainerDied","Data":"dca34b4920a012f5915b59ce23cbe1ed739a9eba8828508473f8035e6ce17321"}
Sep 17 16:22:10.350194 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:22:10.350040    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z22v8" event={"ID":"04d20574-15c0-48e5-b35f-2c32c6d3f6e7","Type":"ContainerDied","Data":"e24e9d2e7323d1cd695eda477b8b06680454399e62630b8e618aedc222a89978"}
Sep 17 16:22:10.350194 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:22:10.350066    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z22v8" event={"ID":"04d20574-15c0-48e5-b35f-2c32c6d3f6e7","Type":"ContainerDied","Data":"4305fafe8799d40dc325d64d906cdcd4c173d7c1058b1a4da5dd53d0ad543d0f"}
Sep 17 16:22:10.350194 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:22:10.350090    2585 scope.go:117] "RemoveContainer" containerID="dca34b4920a012f5915b59ce23cbe1ed739a9eba8828508473f8035e6ce17321"
Sep 17 16:22:10.358975 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:22:10.358957    2585 scope.go:117] "RemoveContainer" containerID="e24e9d2e7323d1cd695eda477b8b06680454399e62630b8e618aedc222a89978"
Sep 17 16:22:10.368245 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:22:10.368221    2585 scope.go:117] "RemoveContainer" containerID="1716d6f0d9af77bbbd4a5783980e105c0320a1e4e737216f26b5328c96761d91"
Sep 17 16:22:10.376703 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:22:10.376678    2585 scope.go:117] "RemoveContainer" containerID="dca34b4920a012f5915b59ce23cbe1ed739a9eba8828508473f8035e6ce17321"
Sep 17 16:22:10.377067 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:22:10.377047    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dca34b4920a012f5915b59ce23cbe1ed739a9eba8828508473f8035e6ce17321\": container with ID starting with dca34b4920a012f5915b59ce23cbe1ed739a9eba8828508473f8035e6ce17321 not found: ID does not exist" containerID="dca34b4920a012f5915b59ce23cbe1ed739a9eba8828508473f8035e6ce17321"
Sep 17 16:22:10.377156 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:22:10.377092    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dca34b4920a012f5915b59ce23cbe1ed739a9eba8828508473f8035e6ce17321"} err="failed to get container status \"dca34b4920a012f5915b59ce23cbe1ed739a9eba8828508473f8035e6ce17321\": rpc error: code = NotFound desc = could not find container \"dca34b4920a012f5915b59ce23cbe1ed739a9eba8828508473f8035e6ce17321\": container with ID starting with dca34b4920a012f5915b59ce23cbe1ed739a9eba8828508473f8035e6ce17321 not found: ID does not exist"
Sep 17 16:22:10.377156 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:22:10.377120    2585 scope.go:117] "RemoveContainer" containerID="e24e9d2e7323d1cd695eda477b8b06680454399e62630b8e618aedc222a89978"
Sep 17 16:22:10.377481 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:22:10.377457    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e24e9d2e7323d1cd695eda477b8b06680454399e62630b8e618aedc222a89978\": container with ID starting with e24e9d2e7323d1cd695eda477b8b06680454399e62630b8e618aedc222a89978 not found: ID does not exist" containerID="e24e9d2e7323d1cd695eda477b8b06680454399e62630b8e618aedc222a89978"
Sep 17 16:22:10.377584 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:22:10.377488    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e24e9d2e7323d1cd695eda477b8b06680454399e62630b8e618aedc222a89978"} err="failed to get container status \"e24e9d2e7323d1cd695eda477b8b06680454399e62630b8e618aedc222a89978\": rpc error: code = NotFound desc = could not find container \"e24e9d2e7323d1cd695eda477b8b06680454399e62630b8e618aedc222a89978\": container with ID starting with e24e9d2e7323d1cd695eda477b8b06680454399e62630b8e618aedc222a89978 not found: ID does not exist"
Sep 17 16:22:10.377584 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:22:10.377506    2585 scope.go:117] "RemoveContainer" containerID="1716d6f0d9af77bbbd4a5783980e105c0320a1e4e737216f26b5328c96761d91"
Sep 17 16:22:10.377804 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:22:10.377783    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1716d6f0d9af77bbbd4a5783980e105c0320a1e4e737216f26b5328c96761d91\": container with ID starting with 1716d6f0d9af77bbbd4a5783980e105c0320a1e4e737216f26b5328c96761d91 not found: ID does not exist" containerID="1716d6f0d9af77bbbd4a5783980e105c0320a1e4e737216f26b5328c96761d91"
Sep 17 16:22:10.377865 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:22:10.377809    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1716d6f0d9af77bbbd4a5783980e105c0320a1e4e737216f26b5328c96761d91"} err="failed to get container status \"1716d6f0d9af77bbbd4a5783980e105c0320a1e4e737216f26b5328c96761d91\": rpc error: code = NotFound desc = could not find container \"1716d6f0d9af77bbbd4a5783980e105c0320a1e4e737216f26b5328c96761d91\": container with ID starting with 1716d6f0d9af77bbbd4a5783980e105c0320a1e4e737216f26b5328c96761d91 not found: ID does not exist"
Sep 17 16:22:10.377865 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:22:10.377824    2585 scope.go:117] "RemoveContainer" containerID="dca34b4920a012f5915b59ce23cbe1ed739a9eba8828508473f8035e6ce17321"
Sep 17 16:22:10.378076 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:22:10.378057    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dca34b4920a012f5915b59ce23cbe1ed739a9eba8828508473f8035e6ce17321"} err="failed to get container status \"dca34b4920a012f5915b59ce23cbe1ed739a9eba8828508473f8035e6ce17321\": rpc error: code = NotFound desc = could not find container \"dca34b4920a012f5915b59ce23cbe1ed739a9eba8828508473f8035e6ce17321\": container with ID starting with dca34b4920a012f5915b59ce23cbe1ed739a9eba8828508473f8035e6ce17321 not found: ID does not exist"
Sep 17 16:22:10.378122 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:22:10.378076    2585 scope.go:117] "RemoveContainer" containerID="e24e9d2e7323d1cd695eda477b8b06680454399e62630b8e618aedc222a89978"
Sep 17 16:22:10.378324 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:22:10.378308    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e24e9d2e7323d1cd695eda477b8b06680454399e62630b8e618aedc222a89978"} err="failed to get container status \"e24e9d2e7323d1cd695eda477b8b06680454399e62630b8e618aedc222a89978\": rpc error: code = NotFound desc = could not find container \"e24e9d2e7323d1cd695eda477b8b06680454399e62630b8e618aedc222a89978\": container with ID starting with e24e9d2e7323d1cd695eda477b8b06680454399e62630b8e618aedc222a89978 not found: ID does not exist"
Sep 17 16:22:10.378368 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:22:10.378324    2585 scope.go:117] "RemoveContainer" containerID="1716d6f0d9af77bbbd4a5783980e105c0320a1e4e737216f26b5328c96761d91"
Sep 17 16:22:10.378604 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:22:10.378583    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1716d6f0d9af77bbbd4a5783980e105c0320a1e4e737216f26b5328c96761d91"} err="failed to get container status \"1716d6f0d9af77bbbd4a5783980e105c0320a1e4e737216f26b5328c96761d91\": rpc error: code = NotFound desc = could not find container \"1716d6f0d9af77bbbd4a5783980e105c0320a1e4e737216f26b5328c96761d91\": container with ID starting with 1716d6f0d9af77bbbd4a5783980e105c0320a1e4e737216f26b5328c96761d91 not found: ID does not exist"
Sep 17 16:22:10.387733 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:22:10.387705    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z22v8"]
Sep 17 16:22:10.405593 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:22:10.405562    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-z22v8"]
Sep 17 16:22:10.513289 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:22:10.513269    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04d20574-15c0-48e5-b35f-2c32c6d3f6e7" path="/var/lib/kubelet/pods/04d20574-15c0-48e5-b35f-2c32c6d3f6e7/volumes"
Sep 17 16:22:11.356484 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:22:11.356447    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd" event={"ID":"6e134bad-b137-4571-b441-9f00e9152e99","Type":"ContainerStarted","Data":"6ffccffc488093442636aa1a3080455961c8afd7bbf8af4eb2583474ba26b5b2"}
Sep 17 16:22:11.356693 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:22:11.356519    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd" event={"ID":"6e134bad-b137-4571-b441-9f00e9152e99","Type":"ContainerStarted","Data":"4cd5b179a95652f5c64115fd8a0da033fdd30162db59a499372ccfe3fddeee68"}
Sep 17 16:22:11.356693 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:22:11.356555    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd"
Sep 17 16:22:11.356693 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:22:11.356585    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd"
Sep 17 16:22:11.393404 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:22:11.393351    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd" podStartSLOduration=2.674671118 podStartE2EDuration="2m15.393334896s" totalImagesPullingTime="2m8.852180895s" totalInitContainerRuntime="3.866482883s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:19:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:22:11.393248353 +0000 UTC m=+1687.163580163" watchObservedRunningTime="2026-09-17 16:22:11.393334896 +0000 UTC m=+1687.163666693"
Sep 17 16:22:17.414744 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:22:17.414657    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd"
Sep 17 16:22:47.418854 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:22:47.418806    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd" podUID="6e134bad-b137-4571-b441-9f00e9152e99" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 17 16:22:57.417950 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:22:57.417916    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd"
Sep 17 16:23:06.552947 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:06.552901    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd"]
Sep 17 16:23:06.553364 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:06.553141    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd" podUID="6e134bad-b137-4571-b441-9f00e9152e99" containerName="kserve-container" containerID="cri-o://4cd5b179a95652f5c64115fd8a0da033fdd30162db59a499372ccfe3fddeee68" gracePeriod=30
Sep 17 16:23:06.553364 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:06.553189    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd" podUID="6e134bad-b137-4571-b441-9f00e9152e99" containerName="kube-rbac-proxy" containerID="cri-o://6ffccffc488093442636aa1a3080455961c8afd7bbf8af4eb2583474ba26b5b2" gracePeriod=30
Sep 17 16:23:07.363028 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:07.362988    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd" podUID="6e134bad-b137-4571-b441-9f00e9152e99" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Sep 17 16:23:07.616901 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:07.616808    2585 generic.go:356] "Generic (PLEG): container finished" podID="6e134bad-b137-4571-b441-9f00e9152e99" containerID="6ffccffc488093442636aa1a3080455961c8afd7bbf8af4eb2583474ba26b5b2" exitCode=2
Sep 17 16:23:07.616901 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:07.616869    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd" event={"ID":"6e134bad-b137-4571-b441-9f00e9152e99","Type":"ContainerDied","Data":"6ffccffc488093442636aa1a3080455961c8afd7bbf8af4eb2583474ba26b5b2"}
Sep 17 16:23:08.457643 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:08.457602    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd" podUID="6e134bad-b137-4571-b441-9f00e9152e99" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.38:8080/v2/models/sklearn-v2-mlserver/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 17 16:23:12.362779 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:12.362735    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd" podUID="6e134bad-b137-4571-b441-9f00e9152e99" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Sep 17 16:23:14.089935 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:14.089907    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd"
Sep 17 16:23:14.179575 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:14.179488    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rphd2\" (UniqueName: \"kubernetes.io/projected/6e134bad-b137-4571-b441-9f00e9152e99-kube-api-access-rphd2\") pod \"6e134bad-b137-4571-b441-9f00e9152e99\" (UID: \"6e134bad-b137-4571-b441-9f00e9152e99\") "
Sep 17 16:23:14.179575 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:14.179529    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e134bad-b137-4571-b441-9f00e9152e99-kserve-provision-location\") pod \"6e134bad-b137-4571-b441-9f00e9152e99\" (UID: \"6e134bad-b137-4571-b441-9f00e9152e99\") "
Sep 17 16:23:14.179575 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:14.179566    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e134bad-b137-4571-b441-9f00e9152e99-proxy-tls\") pod \"6e134bad-b137-4571-b441-9f00e9152e99\" (UID: \"6e134bad-b137-4571-b441-9f00e9152e99\") "
Sep 17 16:23:14.179839 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:14.179644    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e134bad-b137-4571-b441-9f00e9152e99-cabundle-cert\") pod \"6e134bad-b137-4571-b441-9f00e9152e99\" (UID: \"6e134bad-b137-4571-b441-9f00e9152e99\") "
Sep 17 16:23:14.179839 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:14.179689    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e134bad-b137-4571-b441-9f00e9152e99-kube-rbac-proxy-sar-config\") pod \"6e134bad-b137-4571-b441-9f00e9152e99\" (UID: \"6e134bad-b137-4571-b441-9f00e9152e99\") "
Sep 17 16:23:14.179956 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:14.179896    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e134bad-b137-4571-b441-9f00e9152e99-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6e134bad-b137-4571-b441-9f00e9152e99" (UID: "6e134bad-b137-4571-b441-9f00e9152e99"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:23:14.180021 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:14.179987    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e134bad-b137-4571-b441-9f00e9152e99-kserve-provision-location\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:23:14.180075 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:14.180012    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e134bad-b137-4571-b441-9f00e9152e99-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6e134bad-b137-4571-b441-9f00e9152e99" (UID: "6e134bad-b137-4571-b441-9f00e9152e99"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:23:14.180131 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:14.180104    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e134bad-b137-4571-b441-9f00e9152e99-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6e134bad-b137-4571-b441-9f00e9152e99" (UID: "6e134bad-b137-4571-b441-9f00e9152e99"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:23:14.181778 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:14.181744    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e134bad-b137-4571-b441-9f00e9152e99-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6e134bad-b137-4571-b441-9f00e9152e99" (UID: "6e134bad-b137-4571-b441-9f00e9152e99"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:23:14.181931 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:14.181830    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e134bad-b137-4571-b441-9f00e9152e99-kube-api-access-rphd2" (OuterVolumeSpecName: "kube-api-access-rphd2") pod "6e134bad-b137-4571-b441-9f00e9152e99" (UID: "6e134bad-b137-4571-b441-9f00e9152e99"). InnerVolumeSpecName "kube-api-access-rphd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:23:14.280867 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:14.280826    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e134bad-b137-4571-b441-9f00e9152e99-cabundle-cert\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:23:14.280867 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:14.280859    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e134bad-b137-4571-b441-9f00e9152e99-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:23:14.280867 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:14.280869    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rphd2\" (UniqueName: \"kubernetes.io/projected/6e134bad-b137-4571-b441-9f00e9152e99-kube-api-access-rphd2\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:23:14.280867 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:14.280878    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e134bad-b137-4571-b441-9f00e9152e99-proxy-tls\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:23:14.649732 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:14.649696    2585 generic.go:356] "Generic (PLEG): container finished" podID="6e134bad-b137-4571-b441-9f00e9152e99" containerID="4cd5b179a95652f5c64115fd8a0da033fdd30162db59a499372ccfe3fddeee68" exitCode=0
Sep 17 16:23:14.649943 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:14.649751    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd"
Sep 17 16:23:14.649943 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:14.649784    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd" event={"ID":"6e134bad-b137-4571-b441-9f00e9152e99","Type":"ContainerDied","Data":"4cd5b179a95652f5c64115fd8a0da033fdd30162db59a499372ccfe3fddeee68"}
Sep 17 16:23:14.649943 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:14.649855    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd" event={"ID":"6e134bad-b137-4571-b441-9f00e9152e99","Type":"ContainerDied","Data":"f7bb76414c0c6f9d3d2738907e43a3e2549068379aae471e60c3cf1d736fca7b"}
Sep 17 16:23:14.649943 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:14.649898    2585 scope.go:117] "RemoveContainer" containerID="6ffccffc488093442636aa1a3080455961c8afd7bbf8af4eb2583474ba26b5b2"
Sep 17 16:23:14.658442 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:14.658409    2585 scope.go:117] "RemoveContainer" containerID="4cd5b179a95652f5c64115fd8a0da033fdd30162db59a499372ccfe3fddeee68"
Sep 17 16:23:14.666831 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:14.666809    2585 scope.go:117] "RemoveContainer" containerID="12b6f66495c58bdfcf0a7647c3ceb28337f3f91edcaa9c9b1330d4e3351f19e3"
Sep 17 16:23:14.670739 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:14.670713    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd"]
Sep 17 16:23:14.676005 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:14.675830    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-b2pfd"]
Sep 17 16:23:14.679995 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:14.679970    2585 scope.go:117] "RemoveContainer" containerID="6ffccffc488093442636aa1a3080455961c8afd7bbf8af4eb2583474ba26b5b2"
Sep 17 16:23:14.680325 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:23:14.680301    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ffccffc488093442636aa1a3080455961c8afd7bbf8af4eb2583474ba26b5b2\": container with ID starting with 6ffccffc488093442636aa1a3080455961c8afd7bbf8af4eb2583474ba26b5b2 not found: ID does not exist" containerID="6ffccffc488093442636aa1a3080455961c8afd7bbf8af4eb2583474ba26b5b2"
Sep 17 16:23:14.680381 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:14.680333    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ffccffc488093442636aa1a3080455961c8afd7bbf8af4eb2583474ba26b5b2"} err="failed to get container status \"6ffccffc488093442636aa1a3080455961c8afd7bbf8af4eb2583474ba26b5b2\": rpc error: code = NotFound desc = could not find container \"6ffccffc488093442636aa1a3080455961c8afd7bbf8af4eb2583474ba26b5b2\": container with ID starting with 6ffccffc488093442636aa1a3080455961c8afd7bbf8af4eb2583474ba26b5b2 not found: ID does not exist"
Sep 17 16:23:14.680381 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:14.680352    2585 scope.go:117] "RemoveContainer" containerID="4cd5b179a95652f5c64115fd8a0da033fdd30162db59a499372ccfe3fddeee68"
Sep 17 16:23:14.680635 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:23:14.680619    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cd5b179a95652f5c64115fd8a0da033fdd30162db59a499372ccfe3fddeee68\": container with ID starting with 4cd5b179a95652f5c64115fd8a0da033fdd30162db59a499372ccfe3fddeee68 not found: ID does not exist" containerID="4cd5b179a95652f5c64115fd8a0da033fdd30162db59a499372ccfe3fddeee68"
Sep 17 16:23:14.680677 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:14.680638    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cd5b179a95652f5c64115fd8a0da033fdd30162db59a499372ccfe3fddeee68"} err="failed to get container status \"4cd5b179a95652f5c64115fd8a0da033fdd30162db59a499372ccfe3fddeee68\": rpc error: code = NotFound desc = could not find container \"4cd5b179a95652f5c64115fd8a0da033fdd30162db59a499372ccfe3fddeee68\": container with ID starting with 4cd5b179a95652f5c64115fd8a0da033fdd30162db59a499372ccfe3fddeee68 not found: ID does not exist"
Sep 17 16:23:14.680677 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:14.680652    2585 scope.go:117] "RemoveContainer" containerID="12b6f66495c58bdfcf0a7647c3ceb28337f3f91edcaa9c9b1330d4e3351f19e3"
Sep 17 16:23:14.680852 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:23:14.680840    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12b6f66495c58bdfcf0a7647c3ceb28337f3f91edcaa9c9b1330d4e3351f19e3\": container with ID starting with 12b6f66495c58bdfcf0a7647c3ceb28337f3f91edcaa9c9b1330d4e3351f19e3 not found: ID does not exist" containerID="12b6f66495c58bdfcf0a7647c3ceb28337f3f91edcaa9c9b1330d4e3351f19e3"
Sep 17 16:23:14.680889 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:14.680854    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12b6f66495c58bdfcf0a7647c3ceb28337f3f91edcaa9c9b1330d4e3351f19e3"} err="failed to get container status \"12b6f66495c58bdfcf0a7647c3ceb28337f3f91edcaa9c9b1330d4e3351f19e3\": rpc error: code = NotFound desc = could not find container \"12b6f66495c58bdfcf0a7647c3ceb28337f3f91edcaa9c9b1330d4e3351f19e3\": container with ID starting with 12b6f66495c58bdfcf0a7647c3ceb28337f3f91edcaa9c9b1330d4e3351f19e3 not found: ID does not exist"
Sep 17 16:23:16.512643 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:16.512610    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e134bad-b137-4571-b441-9f00e9152e99" path="/var/lib/kubelet/pods/6e134bad-b137-4571-b441-9f00e9152e99/volumes"
Sep 17 16:23:59.143458 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:59.143405    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-ztcck"]
Sep 17 16:23:59.143879 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:59.143776    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e134bad-b137-4571-b441-9f00e9152e99" containerName="kserve-container"
Sep 17 16:23:59.143879 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:59.143789    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e134bad-b137-4571-b441-9f00e9152e99" containerName="kserve-container"
Sep 17 16:23:59.143879 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:59.143807    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e134bad-b137-4571-b441-9f00e9152e99" containerName="storage-initializer"
Sep 17 16:23:59.143879 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:59.143813    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e134bad-b137-4571-b441-9f00e9152e99" containerName="storage-initializer"
Sep 17 16:23:59.143879 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:59.143821    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e134bad-b137-4571-b441-9f00e9152e99" containerName="kube-rbac-proxy"
Sep 17 16:23:59.143879 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:59.143828    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e134bad-b137-4571-b441-9f00e9152e99" containerName="kube-rbac-proxy"
Sep 17 16:23:59.143879 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:59.143844    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04d20574-15c0-48e5-b35f-2c32c6d3f6e7" containerName="storage-initializer"
Sep 17 16:23:59.143879 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:59.143852    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="04d20574-15c0-48e5-b35f-2c32c6d3f6e7" containerName="storage-initializer"
Sep 17 16:23:59.143879 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:59.143859    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04d20574-15c0-48e5-b35f-2c32c6d3f6e7" containerName="kserve-container"
Sep 17 16:23:59.143879 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:59.143865    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="04d20574-15c0-48e5-b35f-2c32c6d3f6e7" containerName="kserve-container"
Sep 17 16:23:59.143879 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:59.143874    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04d20574-15c0-48e5-b35f-2c32c6d3f6e7" containerName="kube-rbac-proxy"
Sep 17 16:23:59.143879 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:59.143879    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="04d20574-15c0-48e5-b35f-2c32c6d3f6e7" containerName="kube-rbac-proxy"
Sep 17 16:23:59.144243 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:59.143933    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="04d20574-15c0-48e5-b35f-2c32c6d3f6e7" containerName="kube-rbac-proxy"
Sep 17 16:23:59.144243 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:59.143942    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e134bad-b137-4571-b441-9f00e9152e99" containerName="kserve-container"
Sep 17 16:23:59.144243 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:59.143949    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e134bad-b137-4571-b441-9f00e9152e99" containerName="kube-rbac-proxy"
Sep 17 16:23:59.144243 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:59.143956    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="04d20574-15c0-48e5-b35f-2c32c6d3f6e7" containerName="kserve-container"
Sep 17 16:23:59.146187 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:59.146166    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-ztcck"
Sep 17 16:23:59.150029 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:59.149995    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
Sep 17 16:23:59.150342 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:59.150321    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xkc4g\""
Sep 17 16:23:59.150622 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:59.150604    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 16:23:59.150866 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:59.150847    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 17 16:23:59.151069 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:59.151048    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 16:23:59.151226 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:59.150986    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 16:23:59.153090 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:59.153065    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:23:59.159791 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:59.159763    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-ztcck"]
Sep 17 16:23:59.268551 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:59.268508    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/170e3bb9-bd00-4859-b4fa-78ea24fa098c-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-ztcck\" (UID: \"170e3bb9-bd00-4859-b4fa-78ea24fa098c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-ztcck"
Sep 17 16:23:59.268551 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:59.268553    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/170e3bb9-bd00-4859-b4fa-78ea24fa098c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-ztcck\" (UID: \"170e3bb9-bd00-4859-b4fa-78ea24fa098c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-ztcck"
Sep 17 16:23:59.268774 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:59.268644    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cck95\" (UniqueName: \"kubernetes.io/projected/170e3bb9-bd00-4859-b4fa-78ea24fa098c-kube-api-access-cck95\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-ztcck\" (UID: \"170e3bb9-bd00-4859-b4fa-78ea24fa098c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-ztcck"
Sep 17 16:23:59.268774 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:59.268677    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/170e3bb9-bd00-4859-b4fa-78ea24fa098c-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-ztcck\" (UID: \"170e3bb9-bd00-4859-b4fa-78ea24fa098c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-ztcck"
Sep 17 16:23:59.268774 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:59.268708    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/170e3bb9-bd00-4859-b4fa-78ea24fa098c-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-ztcck\" (UID: \"170e3bb9-bd00-4859-b4fa-78ea24fa098c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-ztcck"
Sep 17 16:23:59.370010 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:59.369968    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/170e3bb9-bd00-4859-b4fa-78ea24fa098c-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-ztcck\" (UID: \"170e3bb9-bd00-4859-b4fa-78ea24fa098c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-ztcck"
Sep 17 16:23:59.370010 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:59.370011    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/170e3bb9-bd00-4859-b4fa-78ea24fa098c-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-ztcck\" (UID: \"170e3bb9-bd00-4859-b4fa-78ea24fa098c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-ztcck"
Sep 17 16:23:59.370236 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:59.370037    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/170e3bb9-bd00-4859-b4fa-78ea24fa098c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-ztcck\" (UID: \"170e3bb9-bd00-4859-b4fa-78ea24fa098c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-ztcck"
Sep 17 16:23:59.370236 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:59.370104    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cck95\" (UniqueName: \"kubernetes.io/projected/170e3bb9-bd00-4859-b4fa-78ea24fa098c-kube-api-access-cck95\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-ztcck\" (UID: \"170e3bb9-bd00-4859-b4fa-78ea24fa098c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-ztcck"
Sep 17 16:23:59.370236 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:23:59.370133    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-serving-cert: secret "isvc-sklearn-v2-runtime-predictor-serving-cert" not found
Sep 17 16:23:59.370236 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:59.370150    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/170e3bb9-bd00-4859-b4fa-78ea24fa098c-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-ztcck\" (UID: \"170e3bb9-bd00-4859-b4fa-78ea24fa098c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-ztcck"
Sep 17 16:23:59.370236 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:23:59.370219    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/170e3bb9-bd00-4859-b4fa-78ea24fa098c-proxy-tls podName:170e3bb9-bd00-4859-b4fa-78ea24fa098c nodeName:}" failed. No retries permitted until 2026-09-17 16:23:59.87019755 +0000 UTC m=+1795.640529344 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/170e3bb9-bd00-4859-b4fa-78ea24fa098c-proxy-tls") pod "isvc-sklearn-v2-runtime-predictor-5f69c7d498-ztcck" (UID: "170e3bb9-bd00-4859-b4fa-78ea24fa098c") : secret "isvc-sklearn-v2-runtime-predictor-serving-cert" not found
Sep 17 16:23:59.370616 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:59.370593    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/170e3bb9-bd00-4859-b4fa-78ea24fa098c-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-ztcck\" (UID: \"170e3bb9-bd00-4859-b4fa-78ea24fa098c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-ztcck"
Sep 17 16:23:59.370746 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:59.370727    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/170e3bb9-bd00-4859-b4fa-78ea24fa098c-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-ztcck\" (UID: \"170e3bb9-bd00-4859-b4fa-78ea24fa098c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-ztcck"
Sep 17 16:23:59.370816 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:59.370797    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/170e3bb9-bd00-4859-b4fa-78ea24fa098c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-ztcck\" (UID: \"170e3bb9-bd00-4859-b4fa-78ea24fa098c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-ztcck"
Sep 17 16:23:59.379128 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:59.379100    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cck95\" (UniqueName: \"kubernetes.io/projected/170e3bb9-bd00-4859-b4fa-78ea24fa098c-kube-api-access-cck95\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-ztcck\" (UID: \"170e3bb9-bd00-4859-b4fa-78ea24fa098c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-ztcck"
Sep 17 16:23:59.874673 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:59.874563    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/170e3bb9-bd00-4859-b4fa-78ea24fa098c-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-ztcck\" (UID: \"170e3bb9-bd00-4859-b4fa-78ea24fa098c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-ztcck"
Sep 17 16:23:59.877169 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:23:59.877145    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/170e3bb9-bd00-4859-b4fa-78ea24fa098c-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-ztcck\" (UID: \"170e3bb9-bd00-4859-b4fa-78ea24fa098c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-ztcck"
Sep 17 16:24:00.061709 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:24:00.061663    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-ztcck"
Sep 17 16:24:00.345046 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:24:00.345008    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-ztcck"]
Sep 17 16:24:00.348603 ip-10-0-135-43 kubenswrapper[2585]: W0917 16:24:00.348570    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod170e3bb9_bd00_4859_b4fa_78ea24fa098c.slice/crio-09887c3fe06e7b79e1ddab889b179b8984f9c23d14f73a62b0d2a0cee858d5ac WatchSource:0}: Error finding container 09887c3fe06e7b79e1ddab889b179b8984f9c23d14f73a62b0d2a0cee858d5ac: Status 404 returned error can't find the container with id 09887c3fe06e7b79e1ddab889b179b8984f9c23d14f73a62b0d2a0cee858d5ac
Sep 17 16:24:00.857133 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:24:00.857097    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-ztcck" event={"ID":"170e3bb9-bd00-4859-b4fa-78ea24fa098c","Type":"ContainerStarted","Data":"91272ba50b0c55d3a17b5d918395cf2185f75268c2fe0218b6c762ba6f3efa61"}
Sep 17 16:24:00.857289 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:24:00.857161    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-ztcck" event={"ID":"170e3bb9-bd00-4859-b4fa-78ea24fa098c","Type":"ContainerStarted","Data":"09887c3fe06e7b79e1ddab889b179b8984f9c23d14f73a62b0d2a0cee858d5ac"}
Sep 17 16:24:04.876847 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:24:04.876759    2585 generic.go:356] "Generic (PLEG): container finished" podID="170e3bb9-bd00-4859-b4fa-78ea24fa098c" containerID="91272ba50b0c55d3a17b5d918395cf2185f75268c2fe0218b6c762ba6f3efa61" exitCode=0
Sep 17 16:24:04.876847 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:24:04.876829    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-ztcck" event={"ID":"170e3bb9-bd00-4859-b4fa-78ea24fa098c","Type":"ContainerDied","Data":"91272ba50b0c55d3a17b5d918395cf2185f75268c2fe0218b6c762ba6f3efa61"}
Sep 17 16:24:05.883435 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:24:05.883390    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-ztcck" event={"ID":"170e3bb9-bd00-4859-b4fa-78ea24fa098c","Type":"ContainerStarted","Data":"ff19ae604f28b2ecb3b5cd6caa8c7e10b679edad46afd2e7ba1b2456891817c3"}
Sep 17 16:24:05.883824 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:24:05.883463    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-ztcck" event={"ID":"170e3bb9-bd00-4859-b4fa-78ea24fa098c","Type":"ContainerStarted","Data":"4385f1d84e777ae3c36c379a4a5b2c474490e1c69e84b8d796c1dd2ce8647417"}
Sep 17 16:24:05.883824 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:24:05.883701    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-ztcck"
Sep 17 16:24:05.907372 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:24:05.907318    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-ztcck" podStartSLOduration=2.723981554 podStartE2EDuration="6.907301687s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.183320133s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:23:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:24:05.905686339 +0000 UTC m=+1801.676018138" watchObservedRunningTime="2026-09-17 16:24:05.907301687 +0000 UTC m=+1801.677633485"
Sep 17 16:24:06.888090 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:24:06.888058    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-ztcck"
Sep 17 16:24:12.898154 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:24:12.898123    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-ztcck"
Sep 17 16:24:42.914280 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:24:42.914239    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-ztcck" podUID="170e3bb9-bd00-4859-b4fa-78ea24fa098c" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 17 16:24:47.709172 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:24:47.709141    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2b47p_a1850a4d-4d01-4a71-8636-b5918f1426c2/ovn-acl-logging/0.log"
Sep 17 16:24:47.712054 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:24:47.712034    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2b47p_a1850a4d-4d01-4a71-8636-b5918f1426c2/ovn-acl-logging/0.log"
Sep 17 16:24:52.900722 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:24:52.900692    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-ztcck"
Sep 17 16:24:59.203850 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:24:59.203812    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-ztcck"]
Sep 17 16:24:59.204292 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:24:59.204039    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-ztcck" podUID="170e3bb9-bd00-4859-b4fa-78ea24fa098c" containerName="kserve-container" containerID="cri-o://4385f1d84e777ae3c36c379a4a5b2c474490e1c69e84b8d796c1dd2ce8647417" gracePeriod=30
Sep 17 16:24:59.204292 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:24:59.204077    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-ztcck" podUID="170e3bb9-bd00-4859-b4fa-78ea24fa098c" containerName="kube-rbac-proxy" containerID="cri-o://ff19ae604f28b2ecb3b5cd6caa8c7e10b679edad46afd2e7ba1b2456891817c3" gracePeriod=30
Sep 17 16:25:00.120398 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:00.120361    2585 generic.go:356] "Generic (PLEG): container finished" podID="170e3bb9-bd00-4859-b4fa-78ea24fa098c" containerID="ff19ae604f28b2ecb3b5cd6caa8c7e10b679edad46afd2e7ba1b2456891817c3" exitCode=2
Sep 17 16:25:00.120588 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:00.120455    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-ztcck" event={"ID":"170e3bb9-bd00-4859-b4fa-78ea24fa098c","Type":"ContainerDied","Data":"ff19ae604f28b2ecb3b5cd6caa8c7e10b679edad46afd2e7ba1b2456891817c3"}
Sep 17 16:25:02.893270 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:02.893222    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-ztcck" podUID="170e3bb9-bd00-4859-b4fa-78ea24fa098c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused"
Sep 17 16:25:02.898705 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:02.898673    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-ztcck" podUID="170e3bb9-bd00-4859-b4fa-78ea24fa098c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.40:8080/v2/models/isvc-sklearn-v2-runtime/ready\": dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 17 16:25:06.753929 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:06.753903    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-ztcck"
Sep 17 16:25:06.812373 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:06.812293    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/170e3bb9-bd00-4859-b4fa-78ea24fa098c-cabundle-cert\") pod \"170e3bb9-bd00-4859-b4fa-78ea24fa098c\" (UID: \"170e3bb9-bd00-4859-b4fa-78ea24fa098c\") "
Sep 17 16:25:06.812373 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:06.812356    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cck95\" (UniqueName: \"kubernetes.io/projected/170e3bb9-bd00-4859-b4fa-78ea24fa098c-kube-api-access-cck95\") pod \"170e3bb9-bd00-4859-b4fa-78ea24fa098c\" (UID: \"170e3bb9-bd00-4859-b4fa-78ea24fa098c\") "
Sep 17 16:25:06.812588 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:06.812388    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/170e3bb9-bd00-4859-b4fa-78ea24fa098c-kube-rbac-proxy-sar-config\") pod \"170e3bb9-bd00-4859-b4fa-78ea24fa098c\" (UID: \"170e3bb9-bd00-4859-b4fa-78ea24fa098c\") "
Sep 17 16:25:06.812588 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:06.812440    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/170e3bb9-bd00-4859-b4fa-78ea24fa098c-proxy-tls\") pod \"170e3bb9-bd00-4859-b4fa-78ea24fa098c\" (UID: \"170e3bb9-bd00-4859-b4fa-78ea24fa098c\") "
Sep 17 16:25:06.812588 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:06.812480    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/170e3bb9-bd00-4859-b4fa-78ea24fa098c-kserve-provision-location\") pod \"170e3bb9-bd00-4859-b4fa-78ea24fa098c\" (UID: \"170e3bb9-bd00-4859-b4fa-78ea24fa098c\") "
Sep 17 16:25:06.812753 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:06.812722    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/170e3bb9-bd00-4859-b4fa-78ea24fa098c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "170e3bb9-bd00-4859-b4fa-78ea24fa098c" (UID: "170e3bb9-bd00-4859-b4fa-78ea24fa098c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:25:06.812881 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:06.812850    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/170e3bb9-bd00-4859-b4fa-78ea24fa098c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "170e3bb9-bd00-4859-b4fa-78ea24fa098c" (UID: "170e3bb9-bd00-4859-b4fa-78ea24fa098c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:25:06.812881 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:06.812861    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/170e3bb9-bd00-4859-b4fa-78ea24fa098c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "170e3bb9-bd00-4859-b4fa-78ea24fa098c" (UID: "170e3bb9-bd00-4859-b4fa-78ea24fa098c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:25:06.814584 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:06.814559    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/170e3bb9-bd00-4859-b4fa-78ea24fa098c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "170e3bb9-bd00-4859-b4fa-78ea24fa098c" (UID: "170e3bb9-bd00-4859-b4fa-78ea24fa098c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:25:06.814584 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:06.814569    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/170e3bb9-bd00-4859-b4fa-78ea24fa098c-kube-api-access-cck95" (OuterVolumeSpecName: "kube-api-access-cck95") pod "170e3bb9-bd00-4859-b4fa-78ea24fa098c" (UID: "170e3bb9-bd00-4859-b4fa-78ea24fa098c"). InnerVolumeSpecName "kube-api-access-cck95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:25:06.913503 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:06.913458    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cck95\" (UniqueName: \"kubernetes.io/projected/170e3bb9-bd00-4859-b4fa-78ea24fa098c-kube-api-access-cck95\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:25:06.913503 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:06.913493    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/170e3bb9-bd00-4859-b4fa-78ea24fa098c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:25:06.913503 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:06.913505    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/170e3bb9-bd00-4859-b4fa-78ea24fa098c-proxy-tls\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:25:06.913503 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:06.913515    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/170e3bb9-bd00-4859-b4fa-78ea24fa098c-kserve-provision-location\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:25:06.913777 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:06.913524    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/170e3bb9-bd00-4859-b4fa-78ea24fa098c-cabundle-cert\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:25:07.153192 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:07.153103    2585 generic.go:356] "Generic (PLEG): container finished" podID="170e3bb9-bd00-4859-b4fa-78ea24fa098c" containerID="4385f1d84e777ae3c36c379a4a5b2c474490e1c69e84b8d796c1dd2ce8647417" exitCode=0
Sep 17 16:25:07.153192 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:07.153160    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-ztcck"
Sep 17 16:25:07.153394 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:07.153183    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-ztcck" event={"ID":"170e3bb9-bd00-4859-b4fa-78ea24fa098c","Type":"ContainerDied","Data":"4385f1d84e777ae3c36c379a4a5b2c474490e1c69e84b8d796c1dd2ce8647417"}
Sep 17 16:25:07.153394 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:07.153277    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-ztcck" event={"ID":"170e3bb9-bd00-4859-b4fa-78ea24fa098c","Type":"ContainerDied","Data":"09887c3fe06e7b79e1ddab889b179b8984f9c23d14f73a62b0d2a0cee858d5ac"}
Sep 17 16:25:07.153394 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:07.153305    2585 scope.go:117] "RemoveContainer" containerID="ff19ae604f28b2ecb3b5cd6caa8c7e10b679edad46afd2e7ba1b2456891817c3"
Sep 17 16:25:07.162373 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:07.162356    2585 scope.go:117] "RemoveContainer" containerID="4385f1d84e777ae3c36c379a4a5b2c474490e1c69e84b8d796c1dd2ce8647417"
Sep 17 16:25:07.170236 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:07.170217    2585 scope.go:117] "RemoveContainer" containerID="91272ba50b0c55d3a17b5d918395cf2185f75268c2fe0218b6c762ba6f3efa61"
Sep 17 16:25:07.178350 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:07.178325    2585 scope.go:117] "RemoveContainer" containerID="ff19ae604f28b2ecb3b5cd6caa8c7e10b679edad46afd2e7ba1b2456891817c3"
Sep 17 16:25:07.178720 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:25:07.178685    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff19ae604f28b2ecb3b5cd6caa8c7e10b679edad46afd2e7ba1b2456891817c3\": container with ID starting with ff19ae604f28b2ecb3b5cd6caa8c7e10b679edad46afd2e7ba1b2456891817c3 not found: ID does not exist" containerID="ff19ae604f28b2ecb3b5cd6caa8c7e10b679edad46afd2e7ba1b2456891817c3"
Sep 17 16:25:07.178839 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:07.178718    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff19ae604f28b2ecb3b5cd6caa8c7e10b679edad46afd2e7ba1b2456891817c3"} err="failed to get container status \"ff19ae604f28b2ecb3b5cd6caa8c7e10b679edad46afd2e7ba1b2456891817c3\": rpc error: code = NotFound desc = could not find container \"ff19ae604f28b2ecb3b5cd6caa8c7e10b679edad46afd2e7ba1b2456891817c3\": container with ID starting with ff19ae604f28b2ecb3b5cd6caa8c7e10b679edad46afd2e7ba1b2456891817c3 not found: ID does not exist"
Sep 17 16:25:07.178839 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:07.178743    2585 scope.go:117] "RemoveContainer" containerID="4385f1d84e777ae3c36c379a4a5b2c474490e1c69e84b8d796c1dd2ce8647417"
Sep 17 16:25:07.179123 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:25:07.179101    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4385f1d84e777ae3c36c379a4a5b2c474490e1c69e84b8d796c1dd2ce8647417\": container with ID starting with 4385f1d84e777ae3c36c379a4a5b2c474490e1c69e84b8d796c1dd2ce8647417 not found: ID does not exist" containerID="4385f1d84e777ae3c36c379a4a5b2c474490e1c69e84b8d796c1dd2ce8647417"
Sep 17 16:25:07.179198 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:07.179134    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4385f1d84e777ae3c36c379a4a5b2c474490e1c69e84b8d796c1dd2ce8647417"} err="failed to get container status \"4385f1d84e777ae3c36c379a4a5b2c474490e1c69e84b8d796c1dd2ce8647417\": rpc error: code = NotFound desc = could not find container \"4385f1d84e777ae3c36c379a4a5b2c474490e1c69e84b8d796c1dd2ce8647417\": container with ID starting with 4385f1d84e777ae3c36c379a4a5b2c474490e1c69e84b8d796c1dd2ce8647417 not found: ID does not exist"
Sep 17 16:25:07.179198 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:07.179152    2585 scope.go:117] "RemoveContainer" containerID="91272ba50b0c55d3a17b5d918395cf2185f75268c2fe0218b6c762ba6f3efa61"
Sep 17 16:25:07.179514 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:25:07.179469    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91272ba50b0c55d3a17b5d918395cf2185f75268c2fe0218b6c762ba6f3efa61\": container with ID starting with 91272ba50b0c55d3a17b5d918395cf2185f75268c2fe0218b6c762ba6f3efa61 not found: ID does not exist" containerID="91272ba50b0c55d3a17b5d918395cf2185f75268c2fe0218b6c762ba6f3efa61"
Sep 17 16:25:07.179609 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:07.179511    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91272ba50b0c55d3a17b5d918395cf2185f75268c2fe0218b6c762ba6f3efa61"} err="failed to get container status \"91272ba50b0c55d3a17b5d918395cf2185f75268c2fe0218b6c762ba6f3efa61\": rpc error: code = NotFound desc = could not find container \"91272ba50b0c55d3a17b5d918395cf2185f75268c2fe0218b6c762ba6f3efa61\": container with ID starting with 91272ba50b0c55d3a17b5d918395cf2185f75268c2fe0218b6c762ba6f3efa61 not found: ID does not exist"
Sep 17 16:25:07.179609 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:07.179546    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-ztcck"]
Sep 17 16:25:07.184259 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:07.184236    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-ztcck"]
Sep 17 16:25:08.512378 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:08.512344    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="170e3bb9-bd00-4859-b4fa-78ea24fa098c" path="/var/lib/kubelet/pods/170e3bb9-bd00-4859-b4fa-78ea24fa098c/volumes"
Sep 17 16:25:11.615857 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:11.615822    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-9p2f4"]
Sep 17 16:25:11.616246 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:11.616169    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="170e3bb9-bd00-4859-b4fa-78ea24fa098c" containerName="storage-initializer"
Sep 17 16:25:11.616246 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:11.616180    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="170e3bb9-bd00-4859-b4fa-78ea24fa098c" containerName="storage-initializer"
Sep 17 16:25:11.616246 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:11.616191    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="170e3bb9-bd00-4859-b4fa-78ea24fa098c" containerName="kserve-container"
Sep 17 16:25:11.616246 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:11.616196    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="170e3bb9-bd00-4859-b4fa-78ea24fa098c" containerName="kserve-container"
Sep 17 16:25:11.616246 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:11.616212    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="170e3bb9-bd00-4859-b4fa-78ea24fa098c" containerName="kube-rbac-proxy"
Sep 17 16:25:11.616246 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:11.616218    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="170e3bb9-bd00-4859-b4fa-78ea24fa098c" containerName="kube-rbac-proxy"
Sep 17 16:25:11.616457 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:11.616262    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="170e3bb9-bd00-4859-b4fa-78ea24fa098c" containerName="kube-rbac-proxy"
Sep 17 16:25:11.616457 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:11.616271    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="170e3bb9-bd00-4859-b4fa-78ea24fa098c" containerName="kserve-container"
Sep 17 16:25:11.619475 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:11.619457    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-9p2f4"
Sep 17 16:25:11.624281 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:11.624255    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 16:25:11.624395 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:11.624300    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-kube-rbac-proxy-sar-config\""
Sep 17 16:25:11.624395 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:11.624336    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:25:11.624525 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:11.624398    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 16:25:11.625143 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:11.625120    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xkc4g\""
Sep 17 16:25:11.625368 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:11.625347    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-predictor-serving-cert\""
Sep 17 16:25:11.625501 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:11.625388    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 16:25:11.637238 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:11.637216    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-9p2f4"]
Sep 17 16:25:11.754305 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:11.754262    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19c940c4-3bbf-4ada-964b-809f49b05a82-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-9p2f4\" (UID: \"19c940c4-3bbf-4ada-964b-809f49b05a82\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-9p2f4"
Sep 17 16:25:11.754305 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:11.754310    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19c940c4-3bbf-4ada-964b-809f49b05a82-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-9p2f4\" (UID: \"19c940c4-3bbf-4ada-964b-809f49b05a82\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-9p2f4"
Sep 17 16:25:11.754601 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:11.754333    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19c940c4-3bbf-4ada-964b-809f49b05a82-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-9p2f4\" (UID: \"19c940c4-3bbf-4ada-964b-809f49b05a82\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-9p2f4"
Sep 17 16:25:11.754601 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:11.754461    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smnrp\" (UniqueName: \"kubernetes.io/projected/19c940c4-3bbf-4ada-964b-809f49b05a82-kube-api-access-smnrp\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-9p2f4\" (UID: \"19c940c4-3bbf-4ada-964b-809f49b05a82\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-9p2f4"
Sep 17 16:25:11.754601 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:11.754554    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19c940c4-3bbf-4ada-964b-809f49b05a82-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-9p2f4\" (UID: \"19c940c4-3bbf-4ada-964b-809f49b05a82\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-9p2f4"
Sep 17 16:25:11.855575 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:11.855524    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19c940c4-3bbf-4ada-964b-809f49b05a82-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-9p2f4\" (UID: \"19c940c4-3bbf-4ada-964b-809f49b05a82\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-9p2f4"
Sep 17 16:25:11.855575 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:11.855579    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19c940c4-3bbf-4ada-964b-809f49b05a82-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-9p2f4\" (UID: \"19c940c4-3bbf-4ada-964b-809f49b05a82\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-9p2f4"
Sep 17 16:25:11.855775 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:11.855603    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19c940c4-3bbf-4ada-964b-809f49b05a82-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-9p2f4\" (UID: \"19c940c4-3bbf-4ada-964b-809f49b05a82\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-9p2f4"
Sep 17 16:25:11.855775 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:11.855639    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19c940c4-3bbf-4ada-964b-809f49b05a82-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-9p2f4\" (UID: \"19c940c4-3bbf-4ada-964b-809f49b05a82\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-9p2f4"
Sep 17 16:25:11.855775 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:11.855686    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smnrp\" (UniqueName: \"kubernetes.io/projected/19c940c4-3bbf-4ada-964b-809f49b05a82-kube-api-access-smnrp\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-9p2f4\" (UID: \"19c940c4-3bbf-4ada-964b-809f49b05a82\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-9p2f4"
Sep 17 16:25:11.856119 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:11.856098    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19c940c4-3bbf-4ada-964b-809f49b05a82-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-9p2f4\" (UID: \"19c940c4-3bbf-4ada-964b-809f49b05a82\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-9p2f4"
Sep 17 16:25:11.856354 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:11.856319    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19c940c4-3bbf-4ada-964b-809f49b05a82-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-9p2f4\" (UID: \"19c940c4-3bbf-4ada-964b-809f49b05a82\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-9p2f4"
Sep 17 16:25:11.856511 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:11.856358    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19c940c4-3bbf-4ada-964b-809f49b05a82-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-9p2f4\" (UID: \"19c940c4-3bbf-4ada-964b-809f49b05a82\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-9p2f4"
Sep 17 16:25:11.858109 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:11.858090    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19c940c4-3bbf-4ada-964b-809f49b05a82-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-9p2f4\" (UID: \"19c940c4-3bbf-4ada-964b-809f49b05a82\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-9p2f4"
Sep 17 16:25:11.870613 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:11.870549    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smnrp\" (UniqueName: \"kubernetes.io/projected/19c940c4-3bbf-4ada-964b-809f49b05a82-kube-api-access-smnrp\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-9p2f4\" (UID: \"19c940c4-3bbf-4ada-964b-809f49b05a82\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-9p2f4"
Sep 17 16:25:11.930043 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:11.929999    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-9p2f4"
Sep 17 16:25:12.216138 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:12.216106    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-9p2f4"]
Sep 17 16:25:12.219845 ip-10-0-135-43 kubenswrapper[2585]: W0917 16:25:12.219816    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19c940c4_3bbf_4ada_964b_809f49b05a82.slice/crio-55e8312a50063a8b3cacd7b60a9d49cc92fccee8265cbe4579749dee75b78cb6 WatchSource:0}: Error finding container 55e8312a50063a8b3cacd7b60a9d49cc92fccee8265cbe4579749dee75b78cb6: Status 404 returned error can't find the container with id 55e8312a50063a8b3cacd7b60a9d49cc92fccee8265cbe4579749dee75b78cb6
Sep 17 16:25:13.180197 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:13.180162    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-9p2f4" event={"ID":"19c940c4-3bbf-4ada-964b-809f49b05a82","Type":"ContainerStarted","Data":"2c7443624989cc18225b0389bfc3c3feb076ab208bafa7539b5d848d848b3383"}
Sep 17 16:25:13.180702 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:13.180247    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-9p2f4" event={"ID":"19c940c4-3bbf-4ada-964b-809f49b05a82","Type":"ContainerStarted","Data":"55e8312a50063a8b3cacd7b60a9d49cc92fccee8265cbe4579749dee75b78cb6"}
Sep 17 16:25:22.221311 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:22.221227    2585 generic.go:356] "Generic (PLEG): container finished" podID="19c940c4-3bbf-4ada-964b-809f49b05a82" containerID="2c7443624989cc18225b0389bfc3c3feb076ab208bafa7539b5d848d848b3383" exitCode=0
Sep 17 16:25:22.221311 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:22.221296    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-9p2f4" event={"ID":"19c940c4-3bbf-4ada-964b-809f49b05a82","Type":"ContainerDied","Data":"2c7443624989cc18225b0389bfc3c3feb076ab208bafa7539b5d848d848b3383"}
Sep 17 16:25:23.229168 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:23.229125    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-9p2f4" event={"ID":"19c940c4-3bbf-4ada-964b-809f49b05a82","Type":"ContainerStarted","Data":"6bfe05664a9e70431fccc00e443133b77097e5edcec71ee7bafe91bf3bd221df"}
Sep 17 16:25:23.229600 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:23.229190    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-9p2f4" event={"ID":"19c940c4-3bbf-4ada-964b-809f49b05a82","Type":"ContainerStarted","Data":"b696aac3fd3018d362fe85e97842d4489ccf313b47ec40e194f2b9644f12599d"}
Sep 17 16:25:23.229600 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:23.229482    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-9p2f4"
Sep 17 16:25:23.229722 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:23.229643    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-9p2f4"
Sep 17 16:25:23.230943 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:23.230918    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-9p2f4" podUID="19c940c4-3bbf-4ada-964b-809f49b05a82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 17 16:25:24.234518 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:24.234473    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-9p2f4" podUID="19c940c4-3bbf-4ada-964b-809f49b05a82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 17 16:25:29.240619 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:29.240580    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-9p2f4"
Sep 17 16:25:29.241357 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:29.241317    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-9p2f4" podUID="19c940c4-3bbf-4ada-964b-809f49b05a82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 17 16:25:29.271384 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:29.271332    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-9p2f4" podStartSLOduration=8.577723971 podStartE2EDuration="18.271316562s" totalImagesPullingTime="0s" totalInitContainerRuntime="9.693592591s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:25:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:25:23.269701508 +0000 UTC m=+1879.040033304" watchObservedRunningTime="2026-09-17 16:25:29.271316562 +0000 UTC m=+1885.041648411"
Sep 17 16:25:39.242551 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:39.242520    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-9p2f4"
Sep 17 16:25:47.791727 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:47.791694    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-9p2f4"]
Sep 17 16:25:47.792289 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:47.791915    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-9p2f4" podUID="19c940c4-3bbf-4ada-964b-809f49b05a82" containerName="kserve-container" containerID="cri-o://b696aac3fd3018d362fe85e97842d4489ccf313b47ec40e194f2b9644f12599d" gracePeriod=30
Sep 17 16:25:47.792289 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:47.791965    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-9p2f4" podUID="19c940c4-3bbf-4ada-964b-809f49b05a82" containerName="kube-rbac-proxy" containerID="cri-o://6bfe05664a9e70431fccc00e443133b77097e5edcec71ee7bafe91bf3bd221df" gracePeriod=30
Sep 17 16:25:48.345358 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:48.345323    2585 generic.go:356] "Generic (PLEG): container finished" podID="19c940c4-3bbf-4ada-964b-809f49b05a82" containerID="6bfe05664a9e70431fccc00e443133b77097e5edcec71ee7bafe91bf3bd221df" exitCode=2
Sep 17 16:25:48.345559 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:48.345397    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-9p2f4" event={"ID":"19c940c4-3bbf-4ada-964b-809f49b05a82","Type":"ContainerDied","Data":"6bfe05664a9e70431fccc00e443133b77097e5edcec71ee7bafe91bf3bd221df"}
Sep 17 16:25:49.235173 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:49.235124    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-9p2f4" podUID="19c940c4-3bbf-4ada-964b-809f49b05a82" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Sep 17 16:25:49.241551 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:49.241509    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-9p2f4" podUID="19c940c4-3bbf-4ada-964b-809f49b05a82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 17 16:25:49.331888 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:49.331863    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-9p2f4"
Sep 17 16:25:49.351790 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:49.351708    2585 generic.go:356] "Generic (PLEG): container finished" podID="19c940c4-3bbf-4ada-964b-809f49b05a82" containerID="b696aac3fd3018d362fe85e97842d4489ccf313b47ec40e194f2b9644f12599d" exitCode=0
Sep 17 16:25:49.351790 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:49.351765    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-9p2f4"
Sep 17 16:25:49.351790 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:49.351779    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-9p2f4" event={"ID":"19c940c4-3bbf-4ada-964b-809f49b05a82","Type":"ContainerDied","Data":"b696aac3fd3018d362fe85e97842d4489ccf313b47ec40e194f2b9644f12599d"}
Sep 17 16:25:49.352090 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:49.351827    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-9p2f4" event={"ID":"19c940c4-3bbf-4ada-964b-809f49b05a82","Type":"ContainerDied","Data":"55e8312a50063a8b3cacd7b60a9d49cc92fccee8265cbe4579749dee75b78cb6"}
Sep 17 16:25:49.352090 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:49.351854    2585 scope.go:117] "RemoveContainer" containerID="6bfe05664a9e70431fccc00e443133b77097e5edcec71ee7bafe91bf3bd221df"
Sep 17 16:25:49.361523 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:49.361490    2585 scope.go:117] "RemoveContainer" containerID="b696aac3fd3018d362fe85e97842d4489ccf313b47ec40e194f2b9644f12599d"
Sep 17 16:25:49.369808 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:49.369782    2585 scope.go:117] "RemoveContainer" containerID="2c7443624989cc18225b0389bfc3c3feb076ab208bafa7539b5d848d848b3383"
Sep 17 16:25:49.375153 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:49.375132    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19c940c4-3bbf-4ada-964b-809f49b05a82-cabundle-cert\") pod \"19c940c4-3bbf-4ada-964b-809f49b05a82\" (UID: \"19c940c4-3bbf-4ada-964b-809f49b05a82\") "
Sep 17 16:25:49.375248 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:49.375166    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19c940c4-3bbf-4ada-964b-809f49b05a82-kserve-provision-location\") pod \"19c940c4-3bbf-4ada-964b-809f49b05a82\" (UID: \"19c940c4-3bbf-4ada-964b-809f49b05a82\") "
Sep 17 16:25:49.375248 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:49.375206    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-smnrp\" (UniqueName: \"kubernetes.io/projected/19c940c4-3bbf-4ada-964b-809f49b05a82-kube-api-access-smnrp\") pod \"19c940c4-3bbf-4ada-964b-809f49b05a82\" (UID: \"19c940c4-3bbf-4ada-964b-809f49b05a82\") "
Sep 17 16:25:49.375385 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:49.375362    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19c940c4-3bbf-4ada-964b-809f49b05a82-proxy-tls\") pod \"19c940c4-3bbf-4ada-964b-809f49b05a82\" (UID: \"19c940c4-3bbf-4ada-964b-809f49b05a82\") "
Sep 17 16:25:49.375569 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:49.375432    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19c940c4-3bbf-4ada-964b-809f49b05a82-kube-rbac-proxy-sar-config\") pod \"19c940c4-3bbf-4ada-964b-809f49b05a82\" (UID: \"19c940c4-3bbf-4ada-964b-809f49b05a82\") "
Sep 17 16:25:49.375671 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:49.375648    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19c940c4-3bbf-4ada-964b-809f49b05a82-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "19c940c4-3bbf-4ada-964b-809f49b05a82" (UID: "19c940c4-3bbf-4ada-964b-809f49b05a82"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:25:49.375952 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:49.375923    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19c940c4-3bbf-4ada-964b-809f49b05a82-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "19c940c4-3bbf-4ada-964b-809f49b05a82" (UID: "19c940c4-3bbf-4ada-964b-809f49b05a82"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:25:49.376302 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:49.376280    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19c940c4-3bbf-4ada-964b-809f49b05a82-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "19c940c4-3bbf-4ada-964b-809f49b05a82" (UID: "19c940c4-3bbf-4ada-964b-809f49b05a82"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:25:49.377443 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:49.377401    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19c940c4-3bbf-4ada-964b-809f49b05a82-kube-api-access-smnrp" (OuterVolumeSpecName: "kube-api-access-smnrp") pod "19c940c4-3bbf-4ada-964b-809f49b05a82" (UID: "19c940c4-3bbf-4ada-964b-809f49b05a82"). InnerVolumeSpecName "kube-api-access-smnrp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:25:49.377575 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:49.377548    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19c940c4-3bbf-4ada-964b-809f49b05a82-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "19c940c4-3bbf-4ada-964b-809f49b05a82" (UID: "19c940c4-3bbf-4ada-964b-809f49b05a82"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:25:49.379532 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:49.379499    2585 scope.go:117] "RemoveContainer" containerID="6bfe05664a9e70431fccc00e443133b77097e5edcec71ee7bafe91bf3bd221df"
Sep 17 16:25:49.379876 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:25:49.379859    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bfe05664a9e70431fccc00e443133b77097e5edcec71ee7bafe91bf3bd221df\": container with ID starting with 6bfe05664a9e70431fccc00e443133b77097e5edcec71ee7bafe91bf3bd221df not found: ID does not exist" containerID="6bfe05664a9e70431fccc00e443133b77097e5edcec71ee7bafe91bf3bd221df"
Sep 17 16:25:49.379954 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:49.379888    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bfe05664a9e70431fccc00e443133b77097e5edcec71ee7bafe91bf3bd221df"} err="failed to get container status \"6bfe05664a9e70431fccc00e443133b77097e5edcec71ee7bafe91bf3bd221df\": rpc error: code = NotFound desc = could not find container \"6bfe05664a9e70431fccc00e443133b77097e5edcec71ee7bafe91bf3bd221df\": container with ID starting with 6bfe05664a9e70431fccc00e443133b77097e5edcec71ee7bafe91bf3bd221df not found: ID does not exist"
Sep 17 16:25:49.379954 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:49.379912    2585 scope.go:117] "RemoveContainer" containerID="b696aac3fd3018d362fe85e97842d4489ccf313b47ec40e194f2b9644f12599d"
Sep 17 16:25:49.380192 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:25:49.380173    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b696aac3fd3018d362fe85e97842d4489ccf313b47ec40e194f2b9644f12599d\": container with ID starting with b696aac3fd3018d362fe85e97842d4489ccf313b47ec40e194f2b9644f12599d not found: ID does not exist" containerID="b696aac3fd3018d362fe85e97842d4489ccf313b47ec40e194f2b9644f12599d"
Sep 17 16:25:49.380257 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:49.380198    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b696aac3fd3018d362fe85e97842d4489ccf313b47ec40e194f2b9644f12599d"} err="failed to get container status \"b696aac3fd3018d362fe85e97842d4489ccf313b47ec40e194f2b9644f12599d\": rpc error: code = NotFound desc = could not find container \"b696aac3fd3018d362fe85e97842d4489ccf313b47ec40e194f2b9644f12599d\": container with ID starting with b696aac3fd3018d362fe85e97842d4489ccf313b47ec40e194f2b9644f12599d not found: ID does not exist"
Sep 17 16:25:49.380257 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:49.380217    2585 scope.go:117] "RemoveContainer" containerID="2c7443624989cc18225b0389bfc3c3feb076ab208bafa7539b5d848d848b3383"
Sep 17 16:25:49.380517 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:25:49.380500    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c7443624989cc18225b0389bfc3c3feb076ab208bafa7539b5d848d848b3383\": container with ID starting with 2c7443624989cc18225b0389bfc3c3feb076ab208bafa7539b5d848d848b3383 not found: ID does not exist" containerID="2c7443624989cc18225b0389bfc3c3feb076ab208bafa7539b5d848d848b3383"
Sep 17 16:25:49.380579 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:49.380522    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c7443624989cc18225b0389bfc3c3feb076ab208bafa7539b5d848d848b3383"} err="failed to get container status \"2c7443624989cc18225b0389bfc3c3feb076ab208bafa7539b5d848d848b3383\": rpc error: code = NotFound desc = could not find container \"2c7443624989cc18225b0389bfc3c3feb076ab208bafa7539b5d848d848b3383\": container with ID starting with 2c7443624989cc18225b0389bfc3c3feb076ab208bafa7539b5d848d848b3383 not found: ID does not exist"
Sep 17 16:25:49.476769 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:49.476733    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19c940c4-3bbf-4ada-964b-809f49b05a82-cabundle-cert\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:25:49.476769 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:49.476764    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19c940c4-3bbf-4ada-964b-809f49b05a82-kserve-provision-location\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:25:49.476769 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:49.476777    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smnrp\" (UniqueName: \"kubernetes.io/projected/19c940c4-3bbf-4ada-964b-809f49b05a82-kube-api-access-smnrp\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:25:49.476999 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:49.476787    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19c940c4-3bbf-4ada-964b-809f49b05a82-proxy-tls\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:25:49.476999 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:49.476797    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19c940c4-3bbf-4ada-964b-809f49b05a82-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:25:49.678470 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:49.678430    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-9p2f4"]
Sep 17 16:25:49.681442 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:49.681401    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-9p2f4"]
Sep 17 16:25:50.512240 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:50.512206    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19c940c4-3bbf-4ada-964b-809f49b05a82" path="/var/lib/kubelet/pods/19c940c4-3bbf-4ada-964b-809f49b05a82/volumes"
Sep 17 16:25:53.450622 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:53.450584    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-8566v"]
Sep 17 16:25:53.451271 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:53.451111    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19c940c4-3bbf-4ada-964b-809f49b05a82" containerName="kube-rbac-proxy"
Sep 17 16:25:53.451271 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:53.451133    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="19c940c4-3bbf-4ada-964b-809f49b05a82" containerName="kube-rbac-proxy"
Sep 17 16:25:53.451271 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:53.451147    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19c940c4-3bbf-4ada-964b-809f49b05a82" containerName="storage-initializer"
Sep 17 16:25:53.451271 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:53.451157    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="19c940c4-3bbf-4ada-964b-809f49b05a82" containerName="storage-initializer"
Sep 17 16:25:53.451271 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:53.451186    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19c940c4-3bbf-4ada-964b-809f49b05a82" containerName="kserve-container"
Sep 17 16:25:53.451271 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:53.451195    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="19c940c4-3bbf-4ada-964b-809f49b05a82" containerName="kserve-container"
Sep 17 16:25:53.451624 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:53.451275    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="19c940c4-3bbf-4ada-964b-809f49b05a82" containerName="kube-rbac-proxy"
Sep 17 16:25:53.451624 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:53.451290    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="19c940c4-3bbf-4ada-964b-809f49b05a82" containerName="kserve-container"
Sep 17 16:25:53.456009 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:53.455988    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-8566v"
Sep 17 16:25:53.466939 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:53.466918    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xkc4g\""
Sep 17 16:25:53.467068 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:53.466919    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 16:25:53.468058 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:53.468043    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 17 16:25:53.468201 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:53.468190    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 17 16:25:53.468250 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:53.468231    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:25:53.468284 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:53.468245    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 16:25:53.474303 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:53.474286    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 16:25:53.495725 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:53.495702    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-8566v"]
Sep 17 16:25:53.509441 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:53.509407    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50729667-9fba-41a6-8621-6ae3f89c076c-kserve-provision-location\") pod \"isvc-canary-create-predictor-66d85d8bfb-8566v\" (UID: \"50729667-9fba-41a6-8621-6ae3f89c076c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-8566v"
Sep 17 16:25:53.509547 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:53.509457    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhpp8\" (UniqueName: \"kubernetes.io/projected/50729667-9fba-41a6-8621-6ae3f89c076c-kube-api-access-hhpp8\") pod \"isvc-canary-create-predictor-66d85d8bfb-8566v\" (UID: \"50729667-9fba-41a6-8621-6ae3f89c076c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-8566v"
Sep 17 16:25:53.509547 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:53.509478    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50729667-9fba-41a6-8621-6ae3f89c076c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-66d85d8bfb-8566v\" (UID: \"50729667-9fba-41a6-8621-6ae3f89c076c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-8566v"
Sep 17 16:25:53.509547 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:53.509500    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50729667-9fba-41a6-8621-6ae3f89c076c-cabundle-cert\") pod \"isvc-canary-create-predictor-66d85d8bfb-8566v\" (UID: \"50729667-9fba-41a6-8621-6ae3f89c076c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-8566v"
Sep 17 16:25:53.509672 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:53.509559    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50729667-9fba-41a6-8621-6ae3f89c076c-proxy-tls\") pod \"isvc-canary-create-predictor-66d85d8bfb-8566v\" (UID: \"50729667-9fba-41a6-8621-6ae3f89c076c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-8566v"
Sep 17 16:25:53.610904 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:53.610855    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50729667-9fba-41a6-8621-6ae3f89c076c-proxy-tls\") pod \"isvc-canary-create-predictor-66d85d8bfb-8566v\" (UID: \"50729667-9fba-41a6-8621-6ae3f89c076c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-8566v"
Sep 17 16:25:53.611115 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:53.610937    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50729667-9fba-41a6-8621-6ae3f89c076c-kserve-provision-location\") pod \"isvc-canary-create-predictor-66d85d8bfb-8566v\" (UID: \"50729667-9fba-41a6-8621-6ae3f89c076c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-8566v"
Sep 17 16:25:53.611115 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:53.610978    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhpp8\" (UniqueName: \"kubernetes.io/projected/50729667-9fba-41a6-8621-6ae3f89c076c-kube-api-access-hhpp8\") pod \"isvc-canary-create-predictor-66d85d8bfb-8566v\" (UID: \"50729667-9fba-41a6-8621-6ae3f89c076c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-8566v"
Sep 17 16:25:53.611115 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:53.611001    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50729667-9fba-41a6-8621-6ae3f89c076c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-66d85d8bfb-8566v\" (UID: \"50729667-9fba-41a6-8621-6ae3f89c076c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-8566v"
Sep 17 16:25:53.611115 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:53.611039    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50729667-9fba-41a6-8621-6ae3f89c076c-cabundle-cert\") pod \"isvc-canary-create-predictor-66d85d8bfb-8566v\" (UID: \"50729667-9fba-41a6-8621-6ae3f89c076c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-8566v"
Sep 17 16:25:53.611500 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:53.611472    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50729667-9fba-41a6-8621-6ae3f89c076c-kserve-provision-location\") pod \"isvc-canary-create-predictor-66d85d8bfb-8566v\" (UID: \"50729667-9fba-41a6-8621-6ae3f89c076c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-8566v"
Sep 17 16:25:53.611781 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:53.611761    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50729667-9fba-41a6-8621-6ae3f89c076c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-66d85d8bfb-8566v\" (UID: \"50729667-9fba-41a6-8621-6ae3f89c076c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-8566v"
Sep 17 16:25:53.611878 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:53.611846    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50729667-9fba-41a6-8621-6ae3f89c076c-cabundle-cert\") pod \"isvc-canary-create-predictor-66d85d8bfb-8566v\" (UID: \"50729667-9fba-41a6-8621-6ae3f89c076c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-8566v"
Sep 17 16:25:53.613304 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:53.613287    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50729667-9fba-41a6-8621-6ae3f89c076c-proxy-tls\") pod \"isvc-canary-create-predictor-66d85d8bfb-8566v\" (UID: \"50729667-9fba-41a6-8621-6ae3f89c076c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-8566v"
Sep 17 16:25:53.623968 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:53.623930    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhpp8\" (UniqueName: \"kubernetes.io/projected/50729667-9fba-41a6-8621-6ae3f89c076c-kube-api-access-hhpp8\") pod \"isvc-canary-create-predictor-66d85d8bfb-8566v\" (UID: \"50729667-9fba-41a6-8621-6ae3f89c076c\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-8566v"
Sep 17 16:25:53.765381 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:53.765287    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-8566v"
Sep 17 16:25:54.046787 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:54.046754    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-8566v"]
Sep 17 16:25:54.049405 ip-10-0-135-43 kubenswrapper[2585]: W0917 16:25:54.049379    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50729667_9fba_41a6_8621_6ae3f89c076c.slice/crio-8060432bea2770895ca66e6b11f0773028e081f44097a398120463453253ac18 WatchSource:0}: Error finding container 8060432bea2770895ca66e6b11f0773028e081f44097a398120463453253ac18: Status 404 returned error can't find the container with id 8060432bea2770895ca66e6b11f0773028e081f44097a398120463453253ac18
Sep 17 16:25:54.386299 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:54.386204    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-8566v" event={"ID":"50729667-9fba-41a6-8621-6ae3f89c076c","Type":"ContainerStarted","Data":"e05f73f1d90f0059f94cdb64bc7db187951f0e3d470c2933217badba6c511d40"}
Sep 17 16:25:54.386299 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:54.386266    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-8566v" event={"ID":"50729667-9fba-41a6-8621-6ae3f89c076c","Type":"ContainerStarted","Data":"8060432bea2770895ca66e6b11f0773028e081f44097a398120463453253ac18"}
Sep 17 16:25:58.404791 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:58.404706    2585 generic.go:356] "Generic (PLEG): container finished" podID="50729667-9fba-41a6-8621-6ae3f89c076c" containerID="e05f73f1d90f0059f94cdb64bc7db187951f0e3d470c2933217badba6c511d40" exitCode=0
Sep 17 16:25:58.404791 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:58.404777    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-8566v" event={"ID":"50729667-9fba-41a6-8621-6ae3f89c076c","Type":"ContainerDied","Data":"e05f73f1d90f0059f94cdb64bc7db187951f0e3d470c2933217badba6c511d40"}
Sep 17 16:25:58.405972 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:25:58.405954    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 16:26:12.493881 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:26:12.493844    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-8566v" event={"ID":"50729667-9fba-41a6-8621-6ae3f89c076c","Type":"ContainerStarted","Data":"21cb93064b8d803f2585527061f713a48e1a9fe64192ccbb19987a2fce474777"}
Sep 17 16:26:12.494299 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:26:12.493931    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-8566v" event={"ID":"50729667-9fba-41a6-8621-6ae3f89c076c","Type":"ContainerStarted","Data":"babba6b2288d56a9771c5e1e70245a5504fe1a9c05dd9f4f4aecb26483b4bdec"}
Sep 17 16:26:12.494299 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:26:12.494258    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-8566v"
Sep 17 16:26:12.494476 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:26:12.494455    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-8566v"
Sep 17 16:26:12.495824 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:26:12.495798    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-8566v" podUID="50729667-9fba-41a6-8621-6ae3f89c076c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 16:26:12.523249 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:26:12.523203    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-8566v" podStartSLOduration=1.780464077 podStartE2EDuration="19.523190408s" totalImagesPullingTime="13.717377177s" totalInitContainerRuntime="4.025349154s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:25:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:26:12.520466343 +0000 UTC m=+1928.290798139" watchObservedRunningTime="2026-09-17 16:26:12.523190408 +0000 UTC m=+1928.293522219"
Sep 17 16:26:13.499293 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:26:13.499245    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-8566v" podUID="50729667-9fba-41a6-8621-6ae3f89c076c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 16:26:18.503756 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:26:18.503725    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-8566v"
Sep 17 16:26:18.504236 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:26:18.504210    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-8566v" podUID="50729667-9fba-41a6-8621-6ae3f89c076c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 16:26:28.505131 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:26:28.505091    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-8566v" podUID="50729667-9fba-41a6-8621-6ae3f89c076c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 16:26:38.505081 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:26:38.505038    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-8566v" podUID="50729667-9fba-41a6-8621-6ae3f89c076c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 16:26:48.505223 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:26:48.505139    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-8566v" podUID="50729667-9fba-41a6-8621-6ae3f89c076c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 16:26:58.505111 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:26:58.505074    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-8566v" podUID="50729667-9fba-41a6-8621-6ae3f89c076c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 16:27:08.504411 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:08.504371    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-8566v" podUID="50729667-9fba-41a6-8621-6ae3f89c076c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 16:27:18.504399 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:18.504360    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-8566v" podUID="50729667-9fba-41a6-8621-6ae3f89c076c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 17 16:27:28.505389 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:28.505359    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-8566v"
Sep 17 16:27:33.112624 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:33.112585    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-8566v"]
Sep 17 16:27:33.113091 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:33.112922    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-8566v" podUID="50729667-9fba-41a6-8621-6ae3f89c076c" containerName="kserve-container" containerID="cri-o://babba6b2288d56a9771c5e1e70245a5504fe1a9c05dd9f4f4aecb26483b4bdec" gracePeriod=30
Sep 17 16:27:33.113091 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:33.112972    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-8566v" podUID="50729667-9fba-41a6-8621-6ae3f89c076c" containerName="kube-rbac-proxy" containerID="cri-o://21cb93064b8d803f2585527061f713a48e1a9fe64192ccbb19987a2fce474777" gracePeriod=30
Sep 17 16:27:33.499638 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:33.499594    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-8566v" podUID="50729667-9fba-41a6-8621-6ae3f89c076c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
Sep 17 16:27:33.868808 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:33.868717    2585 generic.go:356] "Generic (PLEG): container finished" podID="50729667-9fba-41a6-8621-6ae3f89c076c" containerID="21cb93064b8d803f2585527061f713a48e1a9fe64192ccbb19987a2fce474777" exitCode=2
Sep 17 16:27:33.869044 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:33.868800    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-8566v" event={"ID":"50729667-9fba-41a6-8621-6ae3f89c076c","Type":"ContainerDied","Data":"21cb93064b8d803f2585527061f713a48e1a9fe64192ccbb19987a2fce474777"}
Sep 17 16:27:38.063999 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.063973    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-8566v"
Sep 17 16:27:38.112188 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.112146    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50729667-9fba-41a6-8621-6ae3f89c076c-proxy-tls\") pod \"50729667-9fba-41a6-8621-6ae3f89c076c\" (UID: \"50729667-9fba-41a6-8621-6ae3f89c076c\") "
Sep 17 16:27:38.112188 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.112185    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50729667-9fba-41a6-8621-6ae3f89c076c-cabundle-cert\") pod \"50729667-9fba-41a6-8621-6ae3f89c076c\" (UID: \"50729667-9fba-41a6-8621-6ae3f89c076c\") "
Sep 17 16:27:38.112458 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.112221    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50729667-9fba-41a6-8621-6ae3f89c076c-kserve-provision-location\") pod \"50729667-9fba-41a6-8621-6ae3f89c076c\" (UID: \"50729667-9fba-41a6-8621-6ae3f89c076c\") "
Sep 17 16:27:38.112458 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.112277    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50729667-9fba-41a6-8621-6ae3f89c076c-kube-rbac-proxy-sar-config\") pod \"50729667-9fba-41a6-8621-6ae3f89c076c\" (UID: \"50729667-9fba-41a6-8621-6ae3f89c076c\") "
Sep 17 16:27:38.112458 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.112311    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hhpp8\" (UniqueName: \"kubernetes.io/projected/50729667-9fba-41a6-8621-6ae3f89c076c-kube-api-access-hhpp8\") pod \"50729667-9fba-41a6-8621-6ae3f89c076c\" (UID: \"50729667-9fba-41a6-8621-6ae3f89c076c\") "
Sep 17 16:27:38.112621 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.112602    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50729667-9fba-41a6-8621-6ae3f89c076c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "50729667-9fba-41a6-8621-6ae3f89c076c" (UID: "50729667-9fba-41a6-8621-6ae3f89c076c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:27:38.112704 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.112678    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50729667-9fba-41a6-8621-6ae3f89c076c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "50729667-9fba-41a6-8621-6ae3f89c076c" (UID: "50729667-9fba-41a6-8621-6ae3f89c076c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:27:38.112760 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.112712    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50729667-9fba-41a6-8621-6ae3f89c076c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "50729667-9fba-41a6-8621-6ae3f89c076c" (UID: "50729667-9fba-41a6-8621-6ae3f89c076c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:27:38.114534 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.114501    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50729667-9fba-41a6-8621-6ae3f89c076c-kube-api-access-hhpp8" (OuterVolumeSpecName: "kube-api-access-hhpp8") pod "50729667-9fba-41a6-8621-6ae3f89c076c" (UID: "50729667-9fba-41a6-8621-6ae3f89c076c"). InnerVolumeSpecName "kube-api-access-hhpp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:27:38.114657 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.114636    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50729667-9fba-41a6-8621-6ae3f89c076c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "50729667-9fba-41a6-8621-6ae3f89c076c" (UID: "50729667-9fba-41a6-8621-6ae3f89c076c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:27:38.213985 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.213953    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50729667-9fba-41a6-8621-6ae3f89c076c-proxy-tls\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:27:38.213985 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.213982    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50729667-9fba-41a6-8621-6ae3f89c076c-cabundle-cert\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:27:38.213985 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.213993    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50729667-9fba-41a6-8621-6ae3f89c076c-kserve-provision-location\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:27:38.214222 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.214003    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50729667-9fba-41a6-8621-6ae3f89c076c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:27:38.214222 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.214015    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhpp8\" (UniqueName: \"kubernetes.io/projected/50729667-9fba-41a6-8621-6ae3f89c076c-kube-api-access-hhpp8\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:27:38.279460 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.279401    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9"]
Sep 17 16:27:38.279823 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.279807    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50729667-9fba-41a6-8621-6ae3f89c076c" containerName="kserve-container"
Sep 17 16:27:38.279874 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.279825    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="50729667-9fba-41a6-8621-6ae3f89c076c" containerName="kserve-container"
Sep 17 16:27:38.279874 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.279835    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50729667-9fba-41a6-8621-6ae3f89c076c" containerName="kube-rbac-proxy"
Sep 17 16:27:38.279874 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.279841    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="50729667-9fba-41a6-8621-6ae3f89c076c" containerName="kube-rbac-proxy"
Sep 17 16:27:38.279874 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.279857    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50729667-9fba-41a6-8621-6ae3f89c076c" containerName="storage-initializer"
Sep 17 16:27:38.279874 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.279863    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="50729667-9fba-41a6-8621-6ae3f89c076c" containerName="storage-initializer"
Sep 17 16:27:38.280034 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.279915    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="50729667-9fba-41a6-8621-6ae3f89c076c" containerName="kube-rbac-proxy"
Sep 17 16:27:38.280034 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.279923    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="50729667-9fba-41a6-8621-6ae3f89c076c" containerName="kserve-container"
Sep 17 16:27:38.281938 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.281922    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9"
Sep 17 16:27:38.285019 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.284994    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 17 16:27:38.285303 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.285272    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 17 16:27:38.296971 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.296941    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9"]
Sep 17 16:27:38.315277 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.315240    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8d67e9f-0329-48a5-a509-ca4192beb3e0-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9\" (UID: \"b8d67e9f-0329-48a5-a509-ca4192beb3e0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9"
Sep 17 16:27:38.315455 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.315296    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8d67e9f-0329-48a5-a509-ca4192beb3e0-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9\" (UID: \"b8d67e9f-0329-48a5-a509-ca4192beb3e0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9"
Sep 17 16:27:38.315455 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.315329    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8d67e9f-0329-48a5-a509-ca4192beb3e0-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9\" (UID: \"b8d67e9f-0329-48a5-a509-ca4192beb3e0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9"
Sep 17 16:27:38.315455 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.315382    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xf9s\" (UniqueName: \"kubernetes.io/projected/b8d67e9f-0329-48a5-a509-ca4192beb3e0-kube-api-access-5xf9s\") pod \"isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9\" (UID: \"b8d67e9f-0329-48a5-a509-ca4192beb3e0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9"
Sep 17 16:27:38.315455 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.315407    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8d67e9f-0329-48a5-a509-ca4192beb3e0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9\" (UID: \"b8d67e9f-0329-48a5-a509-ca4192beb3e0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9"
Sep 17 16:27:38.320377 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.320349    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-hk744"]
Sep 17 16:27:38.326487 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.326457    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-hk744"
Sep 17 16:27:38.329760 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.329719    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 17 16:27:38.336684 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.336656    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-hk744"]
Sep 17 16:27:38.416448 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.416395    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8d67e9f-0329-48a5-a509-ca4192beb3e0-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9\" (UID: \"b8d67e9f-0329-48a5-a509-ca4192beb3e0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9"
Sep 17 16:27:38.416635 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.416572    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ded5f04f-0718-41d8-b632-daa6024f4b27-proxy-tls\") pod \"isvc-canary-promote-predictor-5fd5fb8776-hk744\" (UID: \"ded5f04f-0718-41d8-b632-daa6024f4b27\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-hk744"
Sep 17 16:27:38.416635 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.416626    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8d67e9f-0329-48a5-a509-ca4192beb3e0-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9\" (UID: \"b8d67e9f-0329-48a5-a509-ca4192beb3e0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9"
Sep 17 16:27:38.416765 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.416681    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xf9s\" (UniqueName: \"kubernetes.io/projected/b8d67e9f-0329-48a5-a509-ca4192beb3e0-kube-api-access-5xf9s\") pod \"isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9\" (UID: \"b8d67e9f-0329-48a5-a509-ca4192beb3e0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9"
Sep 17 16:27:38.416765 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.416714    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8d67e9f-0329-48a5-a509-ca4192beb3e0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9\" (UID: \"b8d67e9f-0329-48a5-a509-ca4192beb3e0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9"
Sep 17 16:27:38.416765 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.416744    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ded5f04f-0718-41d8-b632-daa6024f4b27-cabundle-cert\") pod \"isvc-canary-promote-predictor-5fd5fb8776-hk744\" (UID: \"ded5f04f-0718-41d8-b632-daa6024f4b27\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-hk744"
Sep 17 16:27:38.416765 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:27:38.416746    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-serving-cert: secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Sep 17 16:27:38.416948 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.416786    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ded5f04f-0718-41d8-b632-daa6024f4b27-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5fd5fb8776-hk744\" (UID: \"ded5f04f-0718-41d8-b632-daa6024f4b27\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-hk744"
Sep 17 16:27:38.416948 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.416822    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ded5f04f-0718-41d8-b632-daa6024f4b27-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5fd5fb8776-hk744\" (UID: \"ded5f04f-0718-41d8-b632-daa6024f4b27\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-hk744"
Sep 17 16:27:38.416948 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:27:38.416868    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8d67e9f-0329-48a5-a509-ca4192beb3e0-proxy-tls podName:b8d67e9f-0329-48a5-a509-ca4192beb3e0 nodeName:}" failed. No retries permitted until 2026-09-17 16:27:38.916839061 +0000 UTC m=+2014.687170849 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b8d67e9f-0329-48a5-a509-ca4192beb3e0-proxy-tls") pod "isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9" (UID: "b8d67e9f-0329-48a5-a509-ca4192beb3e0") : secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Sep 17 16:27:38.416948 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.416865    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8d67e9f-0329-48a5-a509-ca4192beb3e0-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9\" (UID: \"b8d67e9f-0329-48a5-a509-ca4192beb3e0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9"
Sep 17 16:27:38.416948 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.416922    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8d67e9f-0329-48a5-a509-ca4192beb3e0-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9\" (UID: \"b8d67e9f-0329-48a5-a509-ca4192beb3e0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9"
Sep 17 16:27:38.417148 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.416975    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqls9\" (UniqueName: \"kubernetes.io/projected/ded5f04f-0718-41d8-b632-daa6024f4b27-kube-api-access-rqls9\") pod \"isvc-canary-promote-predictor-5fd5fb8776-hk744\" (UID: \"ded5f04f-0718-41d8-b632-daa6024f4b27\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-hk744"
Sep 17 16:27:38.417598 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.417576    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8d67e9f-0329-48a5-a509-ca4192beb3e0-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9\" (UID: \"b8d67e9f-0329-48a5-a509-ca4192beb3e0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9"
Sep 17 16:27:38.417672 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.417577    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8d67e9f-0329-48a5-a509-ca4192beb3e0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9\" (UID: \"b8d67e9f-0329-48a5-a509-ca4192beb3e0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9"
Sep 17 16:27:38.425715 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.425691    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xf9s\" (UniqueName: \"kubernetes.io/projected/b8d67e9f-0329-48a5-a509-ca4192beb3e0-kube-api-access-5xf9s\") pod \"isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9\" (UID: \"b8d67e9f-0329-48a5-a509-ca4192beb3e0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9"
Sep 17 16:27:38.517623 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.517531    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ded5f04f-0718-41d8-b632-daa6024f4b27-cabundle-cert\") pod \"isvc-canary-promote-predictor-5fd5fb8776-hk744\" (UID: \"ded5f04f-0718-41d8-b632-daa6024f4b27\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-hk744"
Sep 17 16:27:38.517623 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.517601    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ded5f04f-0718-41d8-b632-daa6024f4b27-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5fd5fb8776-hk744\" (UID: \"ded5f04f-0718-41d8-b632-daa6024f4b27\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-hk744"
Sep 17 16:27:38.517860 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.517632    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ded5f04f-0718-41d8-b632-daa6024f4b27-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5fd5fb8776-hk744\" (UID: \"ded5f04f-0718-41d8-b632-daa6024f4b27\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-hk744"
Sep 17 16:27:38.517860 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.517681    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqls9\" (UniqueName: \"kubernetes.io/projected/ded5f04f-0718-41d8-b632-daa6024f4b27-kube-api-access-rqls9\") pod \"isvc-canary-promote-predictor-5fd5fb8776-hk744\" (UID: \"ded5f04f-0718-41d8-b632-daa6024f4b27\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-hk744"
Sep 17 16:27:38.517860 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.517738    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ded5f04f-0718-41d8-b632-daa6024f4b27-proxy-tls\") pod \"isvc-canary-promote-predictor-5fd5fb8776-hk744\" (UID: \"ded5f04f-0718-41d8-b632-daa6024f4b27\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-hk744"
Sep 17 16:27:38.518029 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:27:38.517898    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 17 16:27:38.518029 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:27:38.517972    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ded5f04f-0718-41d8-b632-daa6024f4b27-proxy-tls podName:ded5f04f-0718-41d8-b632-daa6024f4b27 nodeName:}" failed. No retries permitted until 2026-09-17 16:27:39.017952708 +0000 UTC m=+2014.788284503 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ded5f04f-0718-41d8-b632-daa6024f4b27-proxy-tls") pod "isvc-canary-promote-predictor-5fd5fb8776-hk744" (UID: "ded5f04f-0718-41d8-b632-daa6024f4b27") : secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 17 16:27:38.518029 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.518004    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ded5f04f-0718-41d8-b632-daa6024f4b27-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5fd5fb8776-hk744\" (UID: \"ded5f04f-0718-41d8-b632-daa6024f4b27\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-hk744"
Sep 17 16:27:38.518273 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.518254    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ded5f04f-0718-41d8-b632-daa6024f4b27-cabundle-cert\") pod \"isvc-canary-promote-predictor-5fd5fb8776-hk744\" (UID: \"ded5f04f-0718-41d8-b632-daa6024f4b27\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-hk744"
Sep 17 16:27:38.518389 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.518366    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ded5f04f-0718-41d8-b632-daa6024f4b27-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5fd5fb8776-hk744\" (UID: \"ded5f04f-0718-41d8-b632-daa6024f4b27\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-hk744"
Sep 17 16:27:38.527116 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.527097    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqls9\" (UniqueName: \"kubernetes.io/projected/ded5f04f-0718-41d8-b632-daa6024f4b27-kube-api-access-rqls9\") pod \"isvc-canary-promote-predictor-5fd5fb8776-hk744\" (UID: \"ded5f04f-0718-41d8-b632-daa6024f4b27\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-hk744"
Sep 17 16:27:38.893781 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.893681    2585 generic.go:356] "Generic (PLEG): container finished" podID="50729667-9fba-41a6-8621-6ae3f89c076c" containerID="babba6b2288d56a9771c5e1e70245a5504fe1a9c05dd9f4f4aecb26483b4bdec" exitCode=0
Sep 17 16:27:38.893781 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.893762    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-8566v" event={"ID":"50729667-9fba-41a6-8621-6ae3f89c076c","Type":"ContainerDied","Data":"babba6b2288d56a9771c5e1e70245a5504fe1a9c05dd9f4f4aecb26483b4bdec"}
Sep 17 16:27:38.893971 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.893822    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-8566v" event={"ID":"50729667-9fba-41a6-8621-6ae3f89c076c","Type":"ContainerDied","Data":"8060432bea2770895ca66e6b11f0773028e081f44097a398120463453253ac18"}
Sep 17 16:27:38.893971 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.893770    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-8566v"
Sep 17 16:27:38.893971 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.893849    2585 scope.go:117] "RemoveContainer" containerID="21cb93064b8d803f2585527061f713a48e1a9fe64192ccbb19987a2fce474777"
Sep 17 16:27:38.902719 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.902701    2585 scope.go:117] "RemoveContainer" containerID="babba6b2288d56a9771c5e1e70245a5504fe1a9c05dd9f4f4aecb26483b4bdec"
Sep 17 16:27:38.910851 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.910826    2585 scope.go:117] "RemoveContainer" containerID="e05f73f1d90f0059f94cdb64bc7db187951f0e3d470c2933217badba6c511d40"
Sep 17 16:27:38.912738 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.912713    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-8566v"]
Sep 17 16:27:38.918625 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.918598    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-8566v"]
Sep 17 16:27:38.921700 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.921673    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8d67e9f-0329-48a5-a509-ca4192beb3e0-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9\" (UID: \"b8d67e9f-0329-48a5-a509-ca4192beb3e0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9"
Sep 17 16:27:38.924258 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.924229    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8d67e9f-0329-48a5-a509-ca4192beb3e0-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9\" (UID: \"b8d67e9f-0329-48a5-a509-ca4192beb3e0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9"
Sep 17 16:27:38.927124 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.927097    2585 scope.go:117] "RemoveContainer" containerID="21cb93064b8d803f2585527061f713a48e1a9fe64192ccbb19987a2fce474777"
Sep 17 16:27:38.927523 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:27:38.927502    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21cb93064b8d803f2585527061f713a48e1a9fe64192ccbb19987a2fce474777\": container with ID starting with 21cb93064b8d803f2585527061f713a48e1a9fe64192ccbb19987a2fce474777 not found: ID does not exist" containerID="21cb93064b8d803f2585527061f713a48e1a9fe64192ccbb19987a2fce474777"
Sep 17 16:27:38.927589 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.927534    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21cb93064b8d803f2585527061f713a48e1a9fe64192ccbb19987a2fce474777"} err="failed to get container status \"21cb93064b8d803f2585527061f713a48e1a9fe64192ccbb19987a2fce474777\": rpc error: code = NotFound desc = could not find container \"21cb93064b8d803f2585527061f713a48e1a9fe64192ccbb19987a2fce474777\": container with ID starting with 21cb93064b8d803f2585527061f713a48e1a9fe64192ccbb19987a2fce474777 not found: ID does not exist"
Sep 17 16:27:38.927589 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.927555    2585 scope.go:117] "RemoveContainer" containerID="babba6b2288d56a9771c5e1e70245a5504fe1a9c05dd9f4f4aecb26483b4bdec"
Sep 17 16:27:38.927868 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:27:38.927850    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"babba6b2288d56a9771c5e1e70245a5504fe1a9c05dd9f4f4aecb26483b4bdec\": container with ID starting with babba6b2288d56a9771c5e1e70245a5504fe1a9c05dd9f4f4aecb26483b4bdec not found: ID does not exist" containerID="babba6b2288d56a9771c5e1e70245a5504fe1a9c05dd9f4f4aecb26483b4bdec"
Sep 17 16:27:38.927943 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.927877    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"babba6b2288d56a9771c5e1e70245a5504fe1a9c05dd9f4f4aecb26483b4bdec"} err="failed to get container status \"babba6b2288d56a9771c5e1e70245a5504fe1a9c05dd9f4f4aecb26483b4bdec\": rpc error: code = NotFound desc = could not find container \"babba6b2288d56a9771c5e1e70245a5504fe1a9c05dd9f4f4aecb26483b4bdec\": container with ID starting with babba6b2288d56a9771c5e1e70245a5504fe1a9c05dd9f4f4aecb26483b4bdec not found: ID does not exist"
Sep 17 16:27:38.927943 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.927903    2585 scope.go:117] "RemoveContainer" containerID="e05f73f1d90f0059f94cdb64bc7db187951f0e3d470c2933217badba6c511d40"
Sep 17 16:27:38.928194 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:27:38.928175    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e05f73f1d90f0059f94cdb64bc7db187951f0e3d470c2933217badba6c511d40\": container with ID starting with e05f73f1d90f0059f94cdb64bc7db187951f0e3d470c2933217badba6c511d40 not found: ID does not exist" containerID="e05f73f1d90f0059f94cdb64bc7db187951f0e3d470c2933217badba6c511d40"
Sep 17 16:27:38.928283 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:38.928195    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e05f73f1d90f0059f94cdb64bc7db187951f0e3d470c2933217badba6c511d40"} err="failed to get container status \"e05f73f1d90f0059f94cdb64bc7db187951f0e3d470c2933217badba6c511d40\": rpc error: code = NotFound desc = could not find container \"e05f73f1d90f0059f94cdb64bc7db187951f0e3d470c2933217badba6c511d40\": container with ID starting with e05f73f1d90f0059f94cdb64bc7db187951f0e3d470c2933217badba6c511d40 not found: ID does not exist"
Sep 17 16:27:39.022461 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:39.022395    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ded5f04f-0718-41d8-b632-daa6024f4b27-proxy-tls\") pod \"isvc-canary-promote-predictor-5fd5fb8776-hk744\" (UID: \"ded5f04f-0718-41d8-b632-daa6024f4b27\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-hk744"
Sep 17 16:27:39.024935 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:39.024901    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ded5f04f-0718-41d8-b632-daa6024f4b27-proxy-tls\") pod \"isvc-canary-promote-predictor-5fd5fb8776-hk744\" (UID: \"ded5f04f-0718-41d8-b632-daa6024f4b27\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-hk744"
Sep 17 16:27:39.193070 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:39.192970    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9"
Sep 17 16:27:39.239192 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:39.239161    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-hk744"
Sep 17 16:27:39.467274 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:39.467243    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9"]
Sep 17 16:27:39.469478 ip-10-0-135-43 kubenswrapper[2585]: W0917 16:27:39.469442    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8d67e9f_0329_48a5_a509_ca4192beb3e0.slice/crio-e4fe3ff6b38d9eb0929fd5b377b5d51215baf0be1ec8cb11ef7a20f355631234 WatchSource:0}: Error finding container e4fe3ff6b38d9eb0929fd5b377b5d51215baf0be1ec8cb11ef7a20f355631234: Status 404 returned error can't find the container with id e4fe3ff6b38d9eb0929fd5b377b5d51215baf0be1ec8cb11ef7a20f355631234
Sep 17 16:27:39.521064 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:39.521037    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-hk744"]
Sep 17 16:27:39.524645 ip-10-0-135-43 kubenswrapper[2585]: W0917 16:27:39.524620    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podded5f04f_0718_41d8_b632_daa6024f4b27.slice/crio-0dc17dc49502ff5a4ede114b8b2ee7470b756e1acda42d738ea1d3702015df8e WatchSource:0}: Error finding container 0dc17dc49502ff5a4ede114b8b2ee7470b756e1acda42d738ea1d3702015df8e: Status 404 returned error can't find the container with id 0dc17dc49502ff5a4ede114b8b2ee7470b756e1acda42d738ea1d3702015df8e
Sep 17 16:27:39.902405 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:39.902309    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9" event={"ID":"b8d67e9f-0329-48a5-a509-ca4192beb3e0","Type":"ContainerStarted","Data":"b8c9f85b207bcc3fae882b25610226cc092680631fdc665c64da6c351b5ddd4d"}
Sep 17 16:27:39.902405 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:39.902383    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9" event={"ID":"b8d67e9f-0329-48a5-a509-ca4192beb3e0","Type":"ContainerStarted","Data":"e4fe3ff6b38d9eb0929fd5b377b5d51215baf0be1ec8cb11ef7a20f355631234"}
Sep 17 16:27:39.904167 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:39.904135    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-hk744" event={"ID":"ded5f04f-0718-41d8-b632-daa6024f4b27","Type":"ContainerStarted","Data":"6d9492343b912c862bc8b29b1d48778e4fb113eae562bbb1b59353f9e0ef3954"}
Sep 17 16:27:39.904299 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:39.904189    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-hk744" event={"ID":"ded5f04f-0718-41d8-b632-daa6024f4b27","Type":"ContainerStarted","Data":"0dc17dc49502ff5a4ede114b8b2ee7470b756e1acda42d738ea1d3702015df8e"}
Sep 17 16:27:40.512549 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:40.512517    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50729667-9fba-41a6-8621-6ae3f89c076c" path="/var/lib/kubelet/pods/50729667-9fba-41a6-8621-6ae3f89c076c/volumes"
Sep 17 16:27:43.925504 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:43.925400    2585 generic.go:356] "Generic (PLEG): container finished" podID="b8d67e9f-0329-48a5-a509-ca4192beb3e0" containerID="b8c9f85b207bcc3fae882b25610226cc092680631fdc665c64da6c351b5ddd4d" exitCode=0
Sep 17 16:27:43.925504 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:43.925475    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9" event={"ID":"b8d67e9f-0329-48a5-a509-ca4192beb3e0","Type":"ContainerDied","Data":"b8c9f85b207bcc3fae882b25610226cc092680631fdc665c64da6c351b5ddd4d"}
Sep 17 16:27:43.927173 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:43.927153    2585 generic.go:356] "Generic (PLEG): container finished" podID="ded5f04f-0718-41d8-b632-daa6024f4b27" containerID="6d9492343b912c862bc8b29b1d48778e4fb113eae562bbb1b59353f9e0ef3954" exitCode=0
Sep 17 16:27:43.927267 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:43.927230    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-hk744" event={"ID":"ded5f04f-0718-41d8-b632-daa6024f4b27","Type":"ContainerDied","Data":"6d9492343b912c862bc8b29b1d48778e4fb113eae562bbb1b59353f9e0ef3954"}
Sep 17 16:27:44.933731 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:44.933693    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9" event={"ID":"b8d67e9f-0329-48a5-a509-ca4192beb3e0","Type":"ContainerStarted","Data":"01b58af769f2d21d1ace8d911592161618a944d69519cc835c5133148fb0f9d1"}
Sep 17 16:27:44.934207 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:44.933768    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9" event={"ID":"b8d67e9f-0329-48a5-a509-ca4192beb3e0","Type":"ContainerStarted","Data":"1d2834b341e78aaae6b4b613d8d49c8f0db24405e54a5703e6bd10d94acf19cf"}
Sep 17 16:27:44.934207 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:44.934080    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9"
Sep 17 16:27:44.934319 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:44.934301    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9"
Sep 17 16:27:44.935838 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:44.935810    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9" podUID="b8d67e9f-0329-48a5-a509-ca4192beb3e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 16:27:44.936074 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:44.936049    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-hk744" event={"ID":"ded5f04f-0718-41d8-b632-daa6024f4b27","Type":"ContainerStarted","Data":"10b00697bd802e3fac182b392adce402c2c8f6684033860508429a8d4be7fa44"}
Sep 17 16:27:44.936149 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:44.936115    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-hk744" event={"ID":"ded5f04f-0718-41d8-b632-daa6024f4b27","Type":"ContainerStarted","Data":"f03a1a7b998010284507c110260be67ebcaf56a8f0ce6498b292fbb2bc28d9fb"}
Sep 17 16:27:44.936334 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:44.936315    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-hk744"
Sep 17 16:27:44.936415 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:44.936367    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-hk744"
Sep 17 16:27:44.937273 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:44.937250    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-hk744" podUID="ded5f04f-0718-41d8-b632-daa6024f4b27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 17 16:27:44.955311 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:44.955261    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9" podStartSLOduration=3.215014949 podStartE2EDuration="6.955233859s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.74021891s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:27:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:27:44.954553384 +0000 UTC m=+2020.724885183" watchObservedRunningTime="2026-09-17 16:27:44.955233859 +0000 UTC m=+2020.725565657"
Sep 17 16:27:44.974940 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:44.974876    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-hk744" podStartSLOduration=2.968752465 podStartE2EDuration="6.974855395s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.00610293s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:27:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:27:44.974292915 +0000 UTC m=+2020.744624714" watchObservedRunningTime="2026-09-17 16:27:44.974855395 +0000 UTC m=+2020.745187194"
Sep 17 16:27:45.940848 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:45.940805    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9" podUID="b8d67e9f-0329-48a5-a509-ca4192beb3e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 16:27:45.941253 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:45.940931    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-hk744" podUID="ded5f04f-0718-41d8-b632-daa6024f4b27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 17 16:27:50.946729 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:50.946696    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9"
Sep 17 16:27:50.947214 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:50.946783    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-hk744"
Sep 17 16:27:50.947348 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:50.947322    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9" podUID="b8d67e9f-0329-48a5-a509-ca4192beb3e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 16:27:50.947414 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:27:50.947369    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-hk744" podUID="ded5f04f-0718-41d8-b632-daa6024f4b27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 17 16:28:00.947318 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:28:00.947273    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-hk744" podUID="ded5f04f-0718-41d8-b632-daa6024f4b27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 17 16:28:00.947749 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:28:00.947273    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9" podUID="b8d67e9f-0329-48a5-a509-ca4192beb3e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 16:28:10.947299 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:28:10.947256    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9" podUID="b8d67e9f-0329-48a5-a509-ca4192beb3e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 16:28:10.947888 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:28:10.947256    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-hk744" podUID="ded5f04f-0718-41d8-b632-daa6024f4b27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 17 16:28:15.108908 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:28:15.108870    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-7n6v6"]
Sep 17 16:28:15.119298 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:28:15.119268    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-7n6v6"
Sep 17 16:28:15.124383 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:28:15.124342    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-predictor-serving-cert\""
Sep 17 16:28:15.125363 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:28:15.125340    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\""
Sep 17 16:28:15.128087 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:28:15.128060    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-7n6v6"]
Sep 17 16:28:15.246005 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:28:15.245963    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87d5d833-1cfc-45d0-b84d-93e50972cd6d-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-7n6v6\" (UID: \"87d5d833-1cfc-45d0-b84d-93e50972cd6d\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-7n6v6"
Sep 17 16:28:15.246005 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:28:15.246007    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87d5d833-1cfc-45d0-b84d-93e50972cd6d-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-7n6v6\" (UID: \"87d5d833-1cfc-45d0-b84d-93e50972cd6d\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-7n6v6"
Sep 17 16:28:15.246221 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:28:15.246028    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87d5d833-1cfc-45d0-b84d-93e50972cd6d-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-7n6v6\" (UID: \"87d5d833-1cfc-45d0-b84d-93e50972cd6d\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-7n6v6"
Sep 17 16:28:15.246221 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:28:15.246113    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87d5d833-1cfc-45d0-b84d-93e50972cd6d-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-7n6v6\" (UID: \"87d5d833-1cfc-45d0-b84d-93e50972cd6d\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-7n6v6"
Sep 17 16:28:15.246221 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:28:15.246144    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tddns\" (UniqueName: \"kubernetes.io/projected/87d5d833-1cfc-45d0-b84d-93e50972cd6d-kube-api-access-tddns\") pod \"isvc-tensorflow-predictor-547dfc5669-7n6v6\" (UID: \"87d5d833-1cfc-45d0-b84d-93e50972cd6d\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-7n6v6"
Sep 17 16:28:15.346914 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:28:15.346874    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87d5d833-1cfc-45d0-b84d-93e50972cd6d-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-7n6v6\" (UID: \"87d5d833-1cfc-45d0-b84d-93e50972cd6d\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-7n6v6"
Sep 17 16:28:15.346914 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:28:15.346924    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tddns\" (UniqueName: \"kubernetes.io/projected/87d5d833-1cfc-45d0-b84d-93e50972cd6d-kube-api-access-tddns\") pod \"isvc-tensorflow-predictor-547dfc5669-7n6v6\" (UID: \"87d5d833-1cfc-45d0-b84d-93e50972cd6d\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-7n6v6"
Sep 17 16:28:15.347202 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:28:15.347014    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87d5d833-1cfc-45d0-b84d-93e50972cd6d-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-7n6v6\" (UID: \"87d5d833-1cfc-45d0-b84d-93e50972cd6d\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-7n6v6"
Sep 17 16:28:15.347202 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:28:15.347039    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87d5d833-1cfc-45d0-b84d-93e50972cd6d-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-7n6v6\" (UID: \"87d5d833-1cfc-45d0-b84d-93e50972cd6d\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-7n6v6"
Sep 17 16:28:15.347202 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:28:15.347050    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-tensorflow-predictor-serving-cert: secret "isvc-tensorflow-predictor-serving-cert" not found
Sep 17 16:28:15.347202 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:28:15.347062    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87d5d833-1cfc-45d0-b84d-93e50972cd6d-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-7n6v6\" (UID: \"87d5d833-1cfc-45d0-b84d-93e50972cd6d\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-7n6v6"
Sep 17 16:28:15.347202 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:28:15.347135    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87d5d833-1cfc-45d0-b84d-93e50972cd6d-proxy-tls podName:87d5d833-1cfc-45d0-b84d-93e50972cd6d nodeName:}" failed. No retries permitted until 2026-09-17 16:28:15.847114837 +0000 UTC m=+2051.617446627 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/87d5d833-1cfc-45d0-b84d-93e50972cd6d-proxy-tls") pod "isvc-tensorflow-predictor-547dfc5669-7n6v6" (UID: "87d5d833-1cfc-45d0-b84d-93e50972cd6d") : secret "isvc-tensorflow-predictor-serving-cert" not found
Sep 17 16:28:15.347527 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:28:15.347508    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87d5d833-1cfc-45d0-b84d-93e50972cd6d-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-7n6v6\" (UID: \"87d5d833-1cfc-45d0-b84d-93e50972cd6d\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-7n6v6"
Sep 17 16:28:15.347830 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:28:15.347813    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87d5d833-1cfc-45d0-b84d-93e50972cd6d-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-7n6v6\" (UID: \"87d5d833-1cfc-45d0-b84d-93e50972cd6d\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-7n6v6"
Sep 17 16:28:15.347878 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:28:15.347814    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87d5d833-1cfc-45d0-b84d-93e50972cd6d-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-7n6v6\" (UID: \"87d5d833-1cfc-45d0-b84d-93e50972cd6d\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-7n6v6"
Sep 17 16:28:15.357197 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:28:15.357173    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tddns\" (UniqueName: \"kubernetes.io/projected/87d5d833-1cfc-45d0-b84d-93e50972cd6d-kube-api-access-tddns\") pod \"isvc-tensorflow-predictor-547dfc5669-7n6v6\" (UID: \"87d5d833-1cfc-45d0-b84d-93e50972cd6d\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-7n6v6"
Sep 17 16:28:15.852321 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:28:15.852282    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87d5d833-1cfc-45d0-b84d-93e50972cd6d-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-7n6v6\" (UID: \"87d5d833-1cfc-45d0-b84d-93e50972cd6d\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-7n6v6"
Sep 17 16:28:15.854744 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:28:15.854720    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87d5d833-1cfc-45d0-b84d-93e50972cd6d-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-7n6v6\" (UID: \"87d5d833-1cfc-45d0-b84d-93e50972cd6d\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-7n6v6"
Sep 17 16:28:16.033531 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:28:16.033477    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-7n6v6"
Sep 17 16:28:16.309551 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:28:16.309523    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-7n6v6"]
Sep 17 16:28:16.312135 ip-10-0-135-43 kubenswrapper[2585]: W0917 16:28:16.312103    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87d5d833_1cfc_45d0_b84d_93e50972cd6d.slice/crio-565c22ee85fe16753edd77f823a7f339636c9bce30fc562ab9bc89db4f8b7912 WatchSource:0}: Error finding container 565c22ee85fe16753edd77f823a7f339636c9bce30fc562ab9bc89db4f8b7912: Status 404 returned error can't find the container with id 565c22ee85fe16753edd77f823a7f339636c9bce30fc562ab9bc89db4f8b7912
Sep 17 16:28:17.095254 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:28:17.095219    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-7n6v6" event={"ID":"87d5d833-1cfc-45d0-b84d-93e50972cd6d","Type":"ContainerStarted","Data":"581a3cbe476cef0a5beba49c62a789d9eca68b5d9902b5844059bb089473e12b"}
Sep 17 16:28:17.095430 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:28:17.095275    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-7n6v6" event={"ID":"87d5d833-1cfc-45d0-b84d-93e50972cd6d","Type":"ContainerStarted","Data":"565c22ee85fe16753edd77f823a7f339636c9bce30fc562ab9bc89db4f8b7912"}
Sep 17 16:28:20.947975 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:28:20.947934    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-hk744" podUID="ded5f04f-0718-41d8-b632-daa6024f4b27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 17 16:28:20.947975 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:28:20.947955    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9" podUID="b8d67e9f-0329-48a5-a509-ca4192beb3e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 16:28:22.122951 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:28:22.122918    2585 generic.go:356] "Generic (PLEG): container finished" podID="87d5d833-1cfc-45d0-b84d-93e50972cd6d" containerID="581a3cbe476cef0a5beba49c62a789d9eca68b5d9902b5844059bb089473e12b" exitCode=0
Sep 17 16:28:22.123560 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:28:22.122996    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-7n6v6" event={"ID":"87d5d833-1cfc-45d0-b84d-93e50972cd6d","Type":"ContainerDied","Data":"581a3cbe476cef0a5beba49c62a789d9eca68b5d9902b5844059bb089473e12b"}
Sep 17 16:28:26.147438 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:28:26.147388    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-7n6v6" event={"ID":"87d5d833-1cfc-45d0-b84d-93e50972cd6d","Type":"ContainerStarted","Data":"65b92a68348b6dd96d382a9e8d6fcc83fa144c515367819c15a30a5aaf184ae1"}
Sep 17 16:28:26.147850 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:28:26.147469    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-7n6v6" event={"ID":"87d5d833-1cfc-45d0-b84d-93e50972cd6d","Type":"ContainerStarted","Data":"f965f2290a858592f04d49cdec852a23ff998408c8fba68865b50e9d6b6f018a"}
Sep 17 16:28:26.147850 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:28:26.147649    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-7n6v6"
Sep 17 16:28:26.173832 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:28:26.173677    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-7n6v6" podStartSLOduration=2.487318679 podStartE2EDuration="11.17365326s" totalImagesPullingTime="3.7948789s" totalInitContainerRuntime="4.891455681s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:28:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:28:26.170602024 +0000 UTC m=+2061.940933822" watchObservedRunningTime="2026-09-17 16:28:26.17365326 +0000 UTC m=+2061.943985059"
Sep 17 16:28:27.152501 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:28:27.152460    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-7n6v6"
Sep 17 16:28:27.153820 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:28:27.153791    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-7n6v6" podUID="87d5d833-1cfc-45d0-b84d-93e50972cd6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 17 16:28:28.157728 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:28:28.157692    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-7n6v6" podUID="87d5d833-1cfc-45d0-b84d-93e50972cd6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 17 16:28:30.947709 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:28:30.947671    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9" podUID="b8d67e9f-0329-48a5-a509-ca4192beb3e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 16:28:30.948146 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:28:30.947671    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-hk744" podUID="ded5f04f-0718-41d8-b632-daa6024f4b27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 17 16:28:33.163029 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:28:33.162998    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-7n6v6"
Sep 17 16:28:33.163534 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:28:33.163500    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-7n6v6" podUID="87d5d833-1cfc-45d0-b84d-93e50972cd6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 17 16:28:40.947438 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:28:40.947381    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9" podUID="b8d67e9f-0329-48a5-a509-ca4192beb3e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 16:28:40.947912 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:28:40.947380    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-hk744" podUID="ded5f04f-0718-41d8-b632-daa6024f4b27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 17 16:28:43.164142 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:28:43.164114    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-7n6v6"
Sep 17 16:28:50.948219 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:28:50.948180    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9" podUID="b8d67e9f-0329-48a5-a509-ca4192beb3e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 16:28:50.948695 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:28:50.948185    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-hk744" podUID="ded5f04f-0718-41d8-b632-daa6024f4b27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 17 16:28:54.507510 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:28:54.507472    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9" podUID="b8d67e9f-0329-48a5-a509-ca4192beb3e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 16:28:56.972361 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:28:56.972329    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-7n6v6"]
Sep 17 16:28:56.972907 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:28:56.972639    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-7n6v6" podUID="87d5d833-1cfc-45d0-b84d-93e50972cd6d" containerName="kserve-container" containerID="cri-o://f965f2290a858592f04d49cdec852a23ff998408c8fba68865b50e9d6b6f018a" gracePeriod=30
Sep 17 16:28:56.972907 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:28:56.972671    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-7n6v6" podUID="87d5d833-1cfc-45d0-b84d-93e50972cd6d" containerName="kube-rbac-proxy" containerID="cri-o://65b92a68348b6dd96d382a9e8d6fcc83fa144c515367819c15a30a5aaf184ae1" gracePeriod=30
Sep 17 16:28:57.294737 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:28:57.294646    2585 generic.go:356] "Generic (PLEG): container finished" podID="87d5d833-1cfc-45d0-b84d-93e50972cd6d" containerID="65b92a68348b6dd96d382a9e8d6fcc83fa144c515367819c15a30a5aaf184ae1" exitCode=2
Sep 17 16:28:57.294737 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:28:57.294717    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-7n6v6" event={"ID":"87d5d833-1cfc-45d0-b84d-93e50972cd6d","Type":"ContainerDied","Data":"65b92a68348b6dd96d382a9e8d6fcc83fa144c515367819c15a30a5aaf184ae1"}
Sep 17 16:28:58.158754 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:28:58.158707    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-7n6v6" podUID="87d5d833-1cfc-45d0-b84d-93e50972cd6d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused"
Sep 17 16:29:00.948028 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:00.947997    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-hk744"
Sep 17 16:29:03.158534 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:03.158488    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-7n6v6" podUID="87d5d833-1cfc-45d0-b84d-93e50972cd6d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused"
Sep 17 16:29:04.512603 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:04.512573    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9"
Sep 17 16:29:08.158938 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:08.158894    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-7n6v6" podUID="87d5d833-1cfc-45d0-b84d-93e50972cd6d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused"
Sep 17 16:29:08.159326 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:08.159045    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-7n6v6"
Sep 17 16:29:08.362982 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:08.362941    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-hk744"]
Sep 17 16:29:08.363318 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:08.363283    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-hk744" podUID="ded5f04f-0718-41d8-b632-daa6024f4b27" containerName="kserve-container" containerID="cri-o://f03a1a7b998010284507c110260be67ebcaf56a8f0ce6498b292fbb2bc28d9fb" gracePeriod=30
Sep 17 16:29:08.363477 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:08.363332    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-hk744" podUID="ded5f04f-0718-41d8-b632-daa6024f4b27" containerName="kube-rbac-proxy" containerID="cri-o://10b00697bd802e3fac182b392adce402c2c8f6684033860508429a8d4be7fa44" gracePeriod=30
Sep 17 16:29:09.352941 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:09.352908    2585 generic.go:356] "Generic (PLEG): container finished" podID="ded5f04f-0718-41d8-b632-daa6024f4b27" containerID="10b00697bd802e3fac182b392adce402c2c8f6684033860508429a8d4be7fa44" exitCode=2
Sep 17 16:29:09.353355 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:09.352977    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-hk744" event={"ID":"ded5f04f-0718-41d8-b632-daa6024f4b27","Type":"ContainerDied","Data":"10b00697bd802e3fac182b392adce402c2c8f6684033860508429a8d4be7fa44"}
Sep 17 16:29:10.941811 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:10.941764    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-hk744" podUID="ded5f04f-0718-41d8-b632-daa6024f4b27" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Sep 17 16:29:10.947358 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:10.947324    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-hk744" podUID="ded5f04f-0718-41d8-b632-daa6024f4b27" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 17 16:29:13.103985 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:13.103962    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-hk744"
Sep 17 16:29:13.158222 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:13.158166    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-7n6v6" podUID="87d5d833-1cfc-45d0-b84d-93e50972cd6d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused"
Sep 17 16:29:13.251114 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:13.251069    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ded5f04f-0718-41d8-b632-daa6024f4b27-kube-rbac-proxy-sar-config\") pod \"ded5f04f-0718-41d8-b632-daa6024f4b27\" (UID: \"ded5f04f-0718-41d8-b632-daa6024f4b27\") "
Sep 17 16:29:13.251327 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:13.251127    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqls9\" (UniqueName: \"kubernetes.io/projected/ded5f04f-0718-41d8-b632-daa6024f4b27-kube-api-access-rqls9\") pod \"ded5f04f-0718-41d8-b632-daa6024f4b27\" (UID: \"ded5f04f-0718-41d8-b632-daa6024f4b27\") "
Sep 17 16:29:13.251327 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:13.251168    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ded5f04f-0718-41d8-b632-daa6024f4b27-cabundle-cert\") pod \"ded5f04f-0718-41d8-b632-daa6024f4b27\" (UID: \"ded5f04f-0718-41d8-b632-daa6024f4b27\") "
Sep 17 16:29:13.251327 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:13.251215    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ded5f04f-0718-41d8-b632-daa6024f4b27-proxy-tls\") pod \"ded5f04f-0718-41d8-b632-daa6024f4b27\" (UID: \"ded5f04f-0718-41d8-b632-daa6024f4b27\") "
Sep 17 16:29:13.251327 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:13.251249    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ded5f04f-0718-41d8-b632-daa6024f4b27-kserve-provision-location\") pod \"ded5f04f-0718-41d8-b632-daa6024f4b27\" (UID: \"ded5f04f-0718-41d8-b632-daa6024f4b27\") "
Sep 17 16:29:13.251644 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:13.251610    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ded5f04f-0718-41d8-b632-daa6024f4b27-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ded5f04f-0718-41d8-b632-daa6024f4b27" (UID: "ded5f04f-0718-41d8-b632-daa6024f4b27"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:29:13.251644 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:13.251624    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ded5f04f-0718-41d8-b632-daa6024f4b27-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ded5f04f-0718-41d8-b632-daa6024f4b27" (UID: "ded5f04f-0718-41d8-b632-daa6024f4b27"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:29:13.251813 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:13.251785    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ded5f04f-0718-41d8-b632-daa6024f4b27-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ded5f04f-0718-41d8-b632-daa6024f4b27" (UID: "ded5f04f-0718-41d8-b632-daa6024f4b27"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:29:13.253440 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:13.253400    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ded5f04f-0718-41d8-b632-daa6024f4b27-kube-api-access-rqls9" (OuterVolumeSpecName: "kube-api-access-rqls9") pod "ded5f04f-0718-41d8-b632-daa6024f4b27" (UID: "ded5f04f-0718-41d8-b632-daa6024f4b27"). InnerVolumeSpecName "kube-api-access-rqls9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:29:13.253558 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:13.253531    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ded5f04f-0718-41d8-b632-daa6024f4b27-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ded5f04f-0718-41d8-b632-daa6024f4b27" (UID: "ded5f04f-0718-41d8-b632-daa6024f4b27"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:29:13.352915 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:13.352873    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ded5f04f-0718-41d8-b632-daa6024f4b27-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:29:13.352915 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:13.352908    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqls9\" (UniqueName: \"kubernetes.io/projected/ded5f04f-0718-41d8-b632-daa6024f4b27-kube-api-access-rqls9\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:29:13.352915 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:13.352920    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ded5f04f-0718-41d8-b632-daa6024f4b27-cabundle-cert\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:29:13.353145 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:13.352930    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ded5f04f-0718-41d8-b632-daa6024f4b27-proxy-tls\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:29:13.353145 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:13.352939    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ded5f04f-0718-41d8-b632-daa6024f4b27-kserve-provision-location\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:29:13.373318 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:13.373283    2585 generic.go:356] "Generic (PLEG): container finished" podID="ded5f04f-0718-41d8-b632-daa6024f4b27" containerID="f03a1a7b998010284507c110260be67ebcaf56a8f0ce6498b292fbb2bc28d9fb" exitCode=0
Sep 17 16:29:13.373477 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:13.373359    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-hk744"
Sep 17 16:29:13.373477 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:13.373364    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-hk744" event={"ID":"ded5f04f-0718-41d8-b632-daa6024f4b27","Type":"ContainerDied","Data":"f03a1a7b998010284507c110260be67ebcaf56a8f0ce6498b292fbb2bc28d9fb"}
Sep 17 16:29:13.373477 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:13.373471    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-hk744" event={"ID":"ded5f04f-0718-41d8-b632-daa6024f4b27","Type":"ContainerDied","Data":"0dc17dc49502ff5a4ede114b8b2ee7470b756e1acda42d738ea1d3702015df8e"}
Sep 17 16:29:13.373585 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:13.373499    2585 scope.go:117] "RemoveContainer" containerID="10b00697bd802e3fac182b392adce402c2c8f6684033860508429a8d4be7fa44"
Sep 17 16:29:13.382178 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:13.382152    2585 scope.go:117] "RemoveContainer" containerID="f03a1a7b998010284507c110260be67ebcaf56a8f0ce6498b292fbb2bc28d9fb"
Sep 17 16:29:13.389878 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:13.389859    2585 scope.go:117] "RemoveContainer" containerID="6d9492343b912c862bc8b29b1d48778e4fb113eae562bbb1b59353f9e0ef3954"
Sep 17 16:29:13.397242 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:13.397225    2585 scope.go:117] "RemoveContainer" containerID="10b00697bd802e3fac182b392adce402c2c8f6684033860508429a8d4be7fa44"
Sep 17 16:29:13.397548 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:29:13.397527    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10b00697bd802e3fac182b392adce402c2c8f6684033860508429a8d4be7fa44\": container with ID starting with 10b00697bd802e3fac182b392adce402c2c8f6684033860508429a8d4be7fa44 not found: ID does not exist" containerID="10b00697bd802e3fac182b392adce402c2c8f6684033860508429a8d4be7fa44"
Sep 17 16:29:13.397623 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:13.397567    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10b00697bd802e3fac182b392adce402c2c8f6684033860508429a8d4be7fa44"} err="failed to get container status \"10b00697bd802e3fac182b392adce402c2c8f6684033860508429a8d4be7fa44\": rpc error: code = NotFound desc = could not find container \"10b00697bd802e3fac182b392adce402c2c8f6684033860508429a8d4be7fa44\": container with ID starting with 10b00697bd802e3fac182b392adce402c2c8f6684033860508429a8d4be7fa44 not found: ID does not exist"
Sep 17 16:29:13.397623 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:13.397585    2585 scope.go:117] "RemoveContainer" containerID="f03a1a7b998010284507c110260be67ebcaf56a8f0ce6498b292fbb2bc28d9fb"
Sep 17 16:29:13.397838 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:29:13.397821    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f03a1a7b998010284507c110260be67ebcaf56a8f0ce6498b292fbb2bc28d9fb\": container with ID starting with f03a1a7b998010284507c110260be67ebcaf56a8f0ce6498b292fbb2bc28d9fb not found: ID does not exist" containerID="f03a1a7b998010284507c110260be67ebcaf56a8f0ce6498b292fbb2bc28d9fb"
Sep 17 16:29:13.397881 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:13.397845    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f03a1a7b998010284507c110260be67ebcaf56a8f0ce6498b292fbb2bc28d9fb"} err="failed to get container status \"f03a1a7b998010284507c110260be67ebcaf56a8f0ce6498b292fbb2bc28d9fb\": rpc error: code = NotFound desc = could not find container \"f03a1a7b998010284507c110260be67ebcaf56a8f0ce6498b292fbb2bc28d9fb\": container with ID starting with f03a1a7b998010284507c110260be67ebcaf56a8f0ce6498b292fbb2bc28d9fb not found: ID does not exist"
Sep 17 16:29:13.397881 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:13.397861    2585 scope.go:117] "RemoveContainer" containerID="6d9492343b912c862bc8b29b1d48778e4fb113eae562bbb1b59353f9e0ef3954"
Sep 17 16:29:13.398114 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:29:13.398098    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d9492343b912c862bc8b29b1d48778e4fb113eae562bbb1b59353f9e0ef3954\": container with ID starting with 6d9492343b912c862bc8b29b1d48778e4fb113eae562bbb1b59353f9e0ef3954 not found: ID does not exist" containerID="6d9492343b912c862bc8b29b1d48778e4fb113eae562bbb1b59353f9e0ef3954"
Sep 17 16:29:13.398168 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:13.398115    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d9492343b912c862bc8b29b1d48778e4fb113eae562bbb1b59353f9e0ef3954"} err="failed to get container status \"6d9492343b912c862bc8b29b1d48778e4fb113eae562bbb1b59353f9e0ef3954\": rpc error: code = NotFound desc = could not find container \"6d9492343b912c862bc8b29b1d48778e4fb113eae562bbb1b59353f9e0ef3954\": container with ID starting with 6d9492343b912c862bc8b29b1d48778e4fb113eae562bbb1b59353f9e0ef3954 not found: ID does not exist"
Sep 17 16:29:13.403570 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:13.403542    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-hk744"]
Sep 17 16:29:13.405527 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:13.405505    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-hk744"]
Sep 17 16:29:14.513434 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:14.513382    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ded5f04f-0718-41d8-b632-daa6024f4b27" path="/var/lib/kubelet/pods/ded5f04f-0718-41d8-b632-daa6024f4b27/volumes"
Sep 17 16:29:18.158848 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:18.158809    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-7n6v6" podUID="87d5d833-1cfc-45d0-b84d-93e50972cd6d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused"
Sep 17 16:29:23.158654 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:23.158602    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-7n6v6" podUID="87d5d833-1cfc-45d0-b84d-93e50972cd6d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused"
Sep 17 16:29:27.441990 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:27.441906    2585 generic.go:356] "Generic (PLEG): container finished" podID="87d5d833-1cfc-45d0-b84d-93e50972cd6d" containerID="f965f2290a858592f04d49cdec852a23ff998408c8fba68865b50e9d6b6f018a" exitCode=137
Sep 17 16:29:27.441990 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:27.441978    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-7n6v6" event={"ID":"87d5d833-1cfc-45d0-b84d-93e50972cd6d","Type":"ContainerDied","Data":"f965f2290a858592f04d49cdec852a23ff998408c8fba68865b50e9d6b6f018a"}
Sep 17 16:29:27.623071 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:27.623046    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-7n6v6"
Sep 17 16:29:27.665092 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:27.665065    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87d5d833-1cfc-45d0-b84d-93e50972cd6d-kserve-provision-location\") pod \"87d5d833-1cfc-45d0-b84d-93e50972cd6d\" (UID: \"87d5d833-1cfc-45d0-b84d-93e50972cd6d\") "
Sep 17 16:29:27.665295 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:27.665161    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87d5d833-1cfc-45d0-b84d-93e50972cd6d-cabundle-cert\") pod \"87d5d833-1cfc-45d0-b84d-93e50972cd6d\" (UID: \"87d5d833-1cfc-45d0-b84d-93e50972cd6d\") "
Sep 17 16:29:27.665295 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:27.665186    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87d5d833-1cfc-45d0-b84d-93e50972cd6d-kube-rbac-proxy-sar-config\") pod \"87d5d833-1cfc-45d0-b84d-93e50972cd6d\" (UID: \"87d5d833-1cfc-45d0-b84d-93e50972cd6d\") "
Sep 17 16:29:27.665295 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:27.665203    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tddns\" (UniqueName: \"kubernetes.io/projected/87d5d833-1cfc-45d0-b84d-93e50972cd6d-kube-api-access-tddns\") pod \"87d5d833-1cfc-45d0-b84d-93e50972cd6d\" (UID: \"87d5d833-1cfc-45d0-b84d-93e50972cd6d\") "
Sep 17 16:29:27.665295 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:27.665231    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87d5d833-1cfc-45d0-b84d-93e50972cd6d-proxy-tls\") pod \"87d5d833-1cfc-45d0-b84d-93e50972cd6d\" (UID: \"87d5d833-1cfc-45d0-b84d-93e50972cd6d\") "
Sep 17 16:29:27.665605 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:27.665567    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87d5d833-1cfc-45d0-b84d-93e50972cd6d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "87d5d833-1cfc-45d0-b84d-93e50972cd6d" (UID: "87d5d833-1cfc-45d0-b84d-93e50972cd6d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:29:27.665733 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:27.665603    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87d5d833-1cfc-45d0-b84d-93e50972cd6d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "87d5d833-1cfc-45d0-b84d-93e50972cd6d" (UID: "87d5d833-1cfc-45d0-b84d-93e50972cd6d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:29:27.667506 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:27.667468    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87d5d833-1cfc-45d0-b84d-93e50972cd6d-kube-api-access-tddns" (OuterVolumeSpecName: "kube-api-access-tddns") pod "87d5d833-1cfc-45d0-b84d-93e50972cd6d" (UID: "87d5d833-1cfc-45d0-b84d-93e50972cd6d"). InnerVolumeSpecName "kube-api-access-tddns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:29:27.667785 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:27.667759    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87d5d833-1cfc-45d0-b84d-93e50972cd6d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "87d5d833-1cfc-45d0-b84d-93e50972cd6d" (UID: "87d5d833-1cfc-45d0-b84d-93e50972cd6d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:29:27.681690 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:27.681651    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87d5d833-1cfc-45d0-b84d-93e50972cd6d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "87d5d833-1cfc-45d0-b84d-93e50972cd6d" (UID: "87d5d833-1cfc-45d0-b84d-93e50972cd6d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:29:27.765743 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:27.765702    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87d5d833-1cfc-45d0-b84d-93e50972cd6d-cabundle-cert\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:29:27.765743 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:27.765740    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87d5d833-1cfc-45d0-b84d-93e50972cd6d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:29:27.765942 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:27.765755    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tddns\" (UniqueName: \"kubernetes.io/projected/87d5d833-1cfc-45d0-b84d-93e50972cd6d-kube-api-access-tddns\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:29:27.765942 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:27.765769    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87d5d833-1cfc-45d0-b84d-93e50972cd6d-proxy-tls\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:29:27.765942 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:27.765783    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87d5d833-1cfc-45d0-b84d-93e50972cd6d-kserve-provision-location\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:29:28.449118 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:28.449090    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-7n6v6"
Sep 17 16:29:28.449751 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:28.449092    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-7n6v6" event={"ID":"87d5d833-1cfc-45d0-b84d-93e50972cd6d","Type":"ContainerDied","Data":"565c22ee85fe16753edd77f823a7f339636c9bce30fc562ab9bc89db4f8b7912"}
Sep 17 16:29:28.449751 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:28.449231    2585 scope.go:117] "RemoveContainer" containerID="65b92a68348b6dd96d382a9e8d6fcc83fa144c515367819c15a30a5aaf184ae1"
Sep 17 16:29:28.458736 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:28.458715    2585 scope.go:117] "RemoveContainer" containerID="f965f2290a858592f04d49cdec852a23ff998408c8fba68865b50e9d6b6f018a"
Sep 17 16:29:28.466285 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:28.466264    2585 scope.go:117] "RemoveContainer" containerID="581a3cbe476cef0a5beba49c62a789d9eca68b5d9902b5844059bb089473e12b"
Sep 17 16:29:28.472844 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:28.472822    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-7n6v6"]
Sep 17 16:29:28.476687 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:28.476664    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-7n6v6"]
Sep 17 16:29:28.513376 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:28.513347    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87d5d833-1cfc-45d0-b84d-93e50972cd6d" path="/var/lib/kubelet/pods/87d5d833-1cfc-45d0-b84d-93e50972cd6d/volumes"
Sep 17 16:29:47.739951 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:47.739826    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2b47p_a1850a4d-4d01-4a71-8636-b5918f1426c2/ovn-acl-logging/0.log"
Sep 17 16:29:47.747673 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:47.743020    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2b47p_a1850a4d-4d01-4a71-8636-b5918f1426c2/ovn-acl-logging/0.log"
Sep 17 16:29:49.265722 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:49.265689    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8f94m"]
Sep 17 16:29:49.266102 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:49.266034    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ded5f04f-0718-41d8-b632-daa6024f4b27" containerName="storage-initializer"
Sep 17 16:29:49.266102 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:49.266046    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ded5f04f-0718-41d8-b632-daa6024f4b27" containerName="storage-initializer"
Sep 17 16:29:49.266102 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:49.266054    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ded5f04f-0718-41d8-b632-daa6024f4b27" containerName="kserve-container"
Sep 17 16:29:49.266102 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:49.266060    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ded5f04f-0718-41d8-b632-daa6024f4b27" containerName="kserve-container"
Sep 17 16:29:49.266102 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:49.266070    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87d5d833-1cfc-45d0-b84d-93e50972cd6d" containerName="kube-rbac-proxy"
Sep 17 16:29:49.266102 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:49.266076    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="87d5d833-1cfc-45d0-b84d-93e50972cd6d" containerName="kube-rbac-proxy"
Sep 17 16:29:49.266102 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:49.266087    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87d5d833-1cfc-45d0-b84d-93e50972cd6d" containerName="storage-initializer"
Sep 17 16:29:49.266102 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:49.266092    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="87d5d833-1cfc-45d0-b84d-93e50972cd6d" containerName="storage-initializer"
Sep 17 16:29:49.266353 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:49.266109    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ded5f04f-0718-41d8-b632-daa6024f4b27" containerName="kube-rbac-proxy"
Sep 17 16:29:49.266353 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:49.266114    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ded5f04f-0718-41d8-b632-daa6024f4b27" containerName="kube-rbac-proxy"
Sep 17 16:29:49.266353 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:49.266120    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87d5d833-1cfc-45d0-b84d-93e50972cd6d" containerName="kserve-container"
Sep 17 16:29:49.266353 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:49.266125    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="87d5d833-1cfc-45d0-b84d-93e50972cd6d" containerName="kserve-container"
Sep 17 16:29:49.266353 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:49.266173    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="ded5f04f-0718-41d8-b632-daa6024f4b27" containerName="kserve-container"
Sep 17 16:29:49.266353 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:49.266181    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="ded5f04f-0718-41d8-b632-daa6024f4b27" containerName="kube-rbac-proxy"
Sep 17 16:29:49.266353 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:49.266187    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="87d5d833-1cfc-45d0-b84d-93e50972cd6d" containerName="kube-rbac-proxy"
Sep 17 16:29:49.266353 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:49.266192    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="87d5d833-1cfc-45d0-b84d-93e50972cd6d" containerName="kserve-container"
Sep 17 16:29:49.269284 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:49.269263    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8f94m"
Sep 17 16:29:49.271851 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:49.271826    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-predictor-serving-cert\""
Sep 17 16:29:49.271960 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:49.271903    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-kube-rbac-proxy-sar-config\""
Sep 17 16:29:49.280056 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:49.280030    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8f94m"]
Sep 17 16:29:49.349799 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:49.349762    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzggl\" (UniqueName: \"kubernetes.io/projected/e7745c71-f370-4edb-88bb-09092c06c0a3-kube-api-access-mzggl\") pod \"isvc-triton-predictor-5d7db58676-8f94m\" (UID: \"e7745c71-f370-4edb-88bb-09092c06c0a3\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8f94m"
Sep 17 16:29:49.349978 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:49.349852    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7745c71-f370-4edb-88bb-09092c06c0a3-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-8f94m\" (UID: \"e7745c71-f370-4edb-88bb-09092c06c0a3\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8f94m"
Sep 17 16:29:49.349978 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:49.349889    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7745c71-f370-4edb-88bb-09092c06c0a3-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-8f94m\" (UID: \"e7745c71-f370-4edb-88bb-09092c06c0a3\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8f94m"
Sep 17 16:29:49.349978 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:49.349910    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7745c71-f370-4edb-88bb-09092c06c0a3-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-8f94m\" (UID: \"e7745c71-f370-4edb-88bb-09092c06c0a3\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8f94m"
Sep 17 16:29:49.350093 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:49.349975    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7745c71-f370-4edb-88bb-09092c06c0a3-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-8f94m\" (UID: \"e7745c71-f370-4edb-88bb-09092c06c0a3\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8f94m"
Sep 17 16:29:49.450617 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:49.450577    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzggl\" (UniqueName: \"kubernetes.io/projected/e7745c71-f370-4edb-88bb-09092c06c0a3-kube-api-access-mzggl\") pod \"isvc-triton-predictor-5d7db58676-8f94m\" (UID: \"e7745c71-f370-4edb-88bb-09092c06c0a3\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8f94m"
Sep 17 16:29:49.450764 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:49.450675    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7745c71-f370-4edb-88bb-09092c06c0a3-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-8f94m\" (UID: \"e7745c71-f370-4edb-88bb-09092c06c0a3\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8f94m"
Sep 17 16:29:49.450764 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:49.450717    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7745c71-f370-4edb-88bb-09092c06c0a3-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-8f94m\" (UID: \"e7745c71-f370-4edb-88bb-09092c06c0a3\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8f94m"
Sep 17 16:29:49.450764 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:49.450740    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7745c71-f370-4edb-88bb-09092c06c0a3-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-8f94m\" (UID: \"e7745c71-f370-4edb-88bb-09092c06c0a3\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8f94m"
Sep 17 16:29:49.450912 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:49.450774    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7745c71-f370-4edb-88bb-09092c06c0a3-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-8f94m\" (UID: \"e7745c71-f370-4edb-88bb-09092c06c0a3\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8f94m"
Sep 17 16:29:49.450912 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:29:49.450893    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-triton-predictor-serving-cert: secret "isvc-triton-predictor-serving-cert" not found
Sep 17 16:29:49.451010 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:29:49.450974    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7745c71-f370-4edb-88bb-09092c06c0a3-proxy-tls podName:e7745c71-f370-4edb-88bb-09092c06c0a3 nodeName:}" failed. No retries permitted until 2026-09-17 16:29:49.950950023 +0000 UTC m=+2145.721281813 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e7745c71-f370-4edb-88bb-09092c06c0a3-proxy-tls") pod "isvc-triton-predictor-5d7db58676-8f94m" (UID: "e7745c71-f370-4edb-88bb-09092c06c0a3") : secret "isvc-triton-predictor-serving-cert" not found
Sep 17 16:29:49.451120 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:49.451097    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7745c71-f370-4edb-88bb-09092c06c0a3-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-8f94m\" (UID: \"e7745c71-f370-4edb-88bb-09092c06c0a3\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8f94m"
Sep 17 16:29:49.451384 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:49.451357    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7745c71-f370-4edb-88bb-09092c06c0a3-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-8f94m\" (UID: \"e7745c71-f370-4edb-88bb-09092c06c0a3\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8f94m"
Sep 17 16:29:49.451499 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:49.451476    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7745c71-f370-4edb-88bb-09092c06c0a3-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-8f94m\" (UID: \"e7745c71-f370-4edb-88bb-09092c06c0a3\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8f94m"
Sep 17 16:29:49.474814 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:49.474786    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzggl\" (UniqueName: \"kubernetes.io/projected/e7745c71-f370-4edb-88bb-09092c06c0a3-kube-api-access-mzggl\") pod \"isvc-triton-predictor-5d7db58676-8f94m\" (UID: \"e7745c71-f370-4edb-88bb-09092c06c0a3\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8f94m"
Sep 17 16:29:49.954498 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:49.954463    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7745c71-f370-4edb-88bb-09092c06c0a3-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-8f94m\" (UID: \"e7745c71-f370-4edb-88bb-09092c06c0a3\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8f94m"
Sep 17 16:29:49.954676 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:29:49.954581    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-triton-predictor-serving-cert: secret "isvc-triton-predictor-serving-cert" not found
Sep 17 16:29:49.954676 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:29:49.954632    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7745c71-f370-4edb-88bb-09092c06c0a3-proxy-tls podName:e7745c71-f370-4edb-88bb-09092c06c0a3 nodeName:}" failed. No retries permitted until 2026-09-17 16:29:50.954619515 +0000 UTC m=+2146.724951291 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e7745c71-f370-4edb-88bb-09092c06c0a3-proxy-tls") pod "isvc-triton-predictor-5d7db58676-8f94m" (UID: "e7745c71-f370-4edb-88bb-09092c06c0a3") : secret "isvc-triton-predictor-serving-cert" not found
Sep 17 16:29:50.963029 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:50.962973    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7745c71-f370-4edb-88bb-09092c06c0a3-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-8f94m\" (UID: \"e7745c71-f370-4edb-88bb-09092c06c0a3\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8f94m"
Sep 17 16:29:50.965525 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:50.965498    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7745c71-f370-4edb-88bb-09092c06c0a3-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-8f94m\" (UID: \"e7745c71-f370-4edb-88bb-09092c06c0a3\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8f94m"
Sep 17 16:29:51.080597 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:51.080548    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8f94m"
Sep 17 16:29:51.359513 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:51.359484    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8f94m"]
Sep 17 16:29:51.362504 ip-10-0-135-43 kubenswrapper[2585]: W0917 16:29:51.362474    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7745c71_f370_4edb_88bb_09092c06c0a3.slice/crio-979cf96f9950b7734c161b1772027597c9ff6648dc40b476f39b81236ca1ae0f WatchSource:0}: Error finding container 979cf96f9950b7734c161b1772027597c9ff6648dc40b476f39b81236ca1ae0f: Status 404 returned error can't find the container with id 979cf96f9950b7734c161b1772027597c9ff6648dc40b476f39b81236ca1ae0f
Sep 17 16:29:51.559704 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:51.559669    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8f94m" event={"ID":"e7745c71-f370-4edb-88bb-09092c06c0a3","Type":"ContainerStarted","Data":"69a5013640ad9986a12f8ae5f77f1e18eaa94c1244a78d3d93a0759b6ed2a4f7"}
Sep 17 16:29:51.559865 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:51.559736    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8f94m" event={"ID":"e7745c71-f370-4edb-88bb-09092c06c0a3","Type":"ContainerStarted","Data":"979cf96f9950b7734c161b1772027597c9ff6648dc40b476f39b81236ca1ae0f"}
Sep 17 16:29:56.584716 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:56.584684    2585 generic.go:356] "Generic (PLEG): container finished" podID="e7745c71-f370-4edb-88bb-09092c06c0a3" containerID="69a5013640ad9986a12f8ae5f77f1e18eaa94c1244a78d3d93a0759b6ed2a4f7" exitCode=0
Sep 17 16:29:56.585114 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:29:56.584752    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8f94m" event={"ID":"e7745c71-f370-4edb-88bb-09092c06c0a3","Type":"ContainerDied","Data":"69a5013640ad9986a12f8ae5f77f1e18eaa94c1244a78d3d93a0759b6ed2a4f7"}
Sep 17 16:30:38.519082 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:30:38.518207    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9"]
Sep 17 16:30:38.519082 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:30:38.518495    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9" podUID="b8d67e9f-0329-48a5-a509-ca4192beb3e0" containerName="kserve-container" containerID="cri-o://1d2834b341e78aaae6b4b613d8d49c8f0db24405e54a5703e6bd10d94acf19cf" gracePeriod=30
Sep 17 16:30:38.519082 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:30:38.518810    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9" podUID="b8d67e9f-0329-48a5-a509-ca4192beb3e0" containerName="kube-rbac-proxy" containerID="cri-o://01b58af769f2d21d1ace8d911592161618a944d69519cc835c5133148fb0f9d1" gracePeriod=30
Sep 17 16:30:38.908809 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:30:38.908734    2585 generic.go:356] "Generic (PLEG): container finished" podID="b8d67e9f-0329-48a5-a509-ca4192beb3e0" containerID="01b58af769f2d21d1ace8d911592161618a944d69519cc835c5133148fb0f9d1" exitCode=2
Sep 17 16:30:38.908965 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:30:38.908801    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9" event={"ID":"b8d67e9f-0329-48a5-a509-ca4192beb3e0","Type":"ContainerDied","Data":"01b58af769f2d21d1ace8d911592161618a944d69519cc835c5133148fb0f9d1"}
Sep 17 16:30:40.941603 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:30:40.941557    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9" podUID="b8d67e9f-0329-48a5-a509-ca4192beb3e0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Sep 17 16:30:43.793513 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:30:43.793479    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt"]
Sep 17 16:30:43.810329 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:30:43.810296    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt"
Sep 17 16:30:43.816749 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:30:43.815998    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Sep 17 16:30:43.817213 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:30:43.817188    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 17 16:30:43.818805 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:30:43.818776    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt"]
Sep 17 16:30:43.881343 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:30:43.881086    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd02dd34-37df-478d-8435-fadb17eb6c39-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt\" (UID: \"cd02dd34-37df-478d-8435-fadb17eb6c39\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt"
Sep 17 16:30:43.881343 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:30:43.881179    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd02dd34-37df-478d-8435-fadb17eb6c39-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt\" (UID: \"cd02dd34-37df-478d-8435-fadb17eb6c39\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt"
Sep 17 16:30:43.881343 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:30:43.881222    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cd02dd34-37df-478d-8435-fadb17eb6c39-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt\" (UID: \"cd02dd34-37df-478d-8435-fadb17eb6c39\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt"
Sep 17 16:30:43.881343 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:30:43.881245    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ltmd\" (UniqueName: \"kubernetes.io/projected/cd02dd34-37df-478d-8435-fadb17eb6c39-kube-api-access-2ltmd\") pod \"isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt\" (UID: \"cd02dd34-37df-478d-8435-fadb17eb6c39\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt"
Sep 17 16:30:43.881892 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:30:43.881410    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd02dd34-37df-478d-8435-fadb17eb6c39-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt\" (UID: \"cd02dd34-37df-478d-8435-fadb17eb6c39\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt"
Sep 17 16:30:43.945216 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:30:43.944624    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-g2rvn"]
Sep 17 16:30:43.958931 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:30:43.958895    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-g2rvn"
Sep 17 16:30:43.963449 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:30:43.963318    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 16:30:43.965013 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:30:43.964498    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-g2rvn"]
Sep 17 16:30:43.982224 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:30:43.982169    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbbq8\" (UniqueName: \"kubernetes.io/projected/b8a20890-c2fa-4ba5-94ae-76aec84184d7-kube-api-access-qbbq8\") pod \"isvc-canary-rollback-predictor-6b9dfd44-g2rvn\" (UID: \"b8a20890-c2fa-4ba5-94ae-76aec84184d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-g2rvn"
Sep 17 16:30:43.982409 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:30:43.982231    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd02dd34-37df-478d-8435-fadb17eb6c39-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt\" (UID: \"cd02dd34-37df-478d-8435-fadb17eb6c39\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt"
Sep 17 16:30:43.982409 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:30:43.982272    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cd02dd34-37df-478d-8435-fadb17eb6c39-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt\" (UID: \"cd02dd34-37df-478d-8435-fadb17eb6c39\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt"
Sep 17 16:30:43.982409 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:30:43.982297    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ltmd\" (UniqueName: \"kubernetes.io/projected/cd02dd34-37df-478d-8435-fadb17eb6c39-kube-api-access-2ltmd\") pod \"isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt\" (UID: \"cd02dd34-37df-478d-8435-fadb17eb6c39\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt"
Sep 17 16:30:43.982409 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:30:43.982388    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8a20890-c2fa-4ba5-94ae-76aec84184d7-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b9dfd44-g2rvn\" (UID: \"b8a20890-c2fa-4ba5-94ae-76aec84184d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-g2rvn"
Sep 17 16:30:43.982675 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:30:43.982441    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd02dd34-37df-478d-8435-fadb17eb6c39-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt\" (UID: \"cd02dd34-37df-478d-8435-fadb17eb6c39\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt"
Sep 17 16:30:43.982675 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:30:43.982474    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8a20890-c2fa-4ba5-94ae-76aec84184d7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b9dfd44-g2rvn\" (UID: \"b8a20890-c2fa-4ba5-94ae-76aec84184d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-g2rvn"
Sep 17 16:30:43.982675 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:30:43.982593    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-serving-cert: secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Sep 17 16:30:43.982675 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:30:43.982677    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd02dd34-37df-478d-8435-fadb17eb6c39-proxy-tls podName:cd02dd34-37df-478d-8435-fadb17eb6c39 nodeName:}" failed. No retries permitted until 2026-09-17 16:30:44.482655796 +0000 UTC m=+2200.252987577 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/cd02dd34-37df-478d-8435-fadb17eb6c39-proxy-tls") pod "isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt" (UID: "cd02dd34-37df-478d-8435-fadb17eb6c39") : secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Sep 17 16:30:43.982884 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:30:43.982793    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd02dd34-37df-478d-8435-fadb17eb6c39-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt\" (UID: \"cd02dd34-37df-478d-8435-fadb17eb6c39\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt"
Sep 17 16:30:43.982884 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:30:43.982850    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8a20890-c2fa-4ba5-94ae-76aec84184d7-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b9dfd44-g2rvn\" (UID: \"b8a20890-c2fa-4ba5-94ae-76aec84184d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-g2rvn"
Sep 17 16:30:43.983000 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:30:43.982892    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8a20890-c2fa-4ba5-94ae-76aec84184d7-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b9dfd44-g2rvn\" (UID: \"b8a20890-c2fa-4ba5-94ae-76aec84184d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-g2rvn"
Sep 17 16:30:43.983264 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:30:43.983237    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd02dd34-37df-478d-8435-fadb17eb6c39-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt\" (UID: \"cd02dd34-37df-478d-8435-fadb17eb6c39\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt"
Sep 17 16:30:43.983346 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:30:43.983285    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd02dd34-37df-478d-8435-fadb17eb6c39-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt\" (UID: \"cd02dd34-37df-478d-8435-fadb17eb6c39\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt"
Sep 17 16:30:43.983708 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:30:43.983680    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cd02dd34-37df-478d-8435-fadb17eb6c39-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt\" (UID: \"cd02dd34-37df-478d-8435-fadb17eb6c39\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt"
Sep 17 16:30:43.996728 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:30:43.996699    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ltmd\" (UniqueName: \"kubernetes.io/projected/cd02dd34-37df-478d-8435-fadb17eb6c39-kube-api-access-2ltmd\") pod \"isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt\" (UID: \"cd02dd34-37df-478d-8435-fadb17eb6c39\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt"
Sep 17 16:30:44.084089 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:30:44.083982    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8a20890-c2fa-4ba5-94ae-76aec84184d7-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b9dfd44-g2rvn\" (UID: \"b8a20890-c2fa-4ba5-94ae-76aec84184d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-g2rvn"
Sep 17 16:30:44.084374 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:30:44.084149    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8a20890-c2fa-4ba5-94ae-76aec84184d7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b9dfd44-g2rvn\" (UID: \"b8a20890-c2fa-4ba5-94ae-76aec84184d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-g2rvn"
Sep 17 16:30:44.084374 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:30:44.084190    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8a20890-c2fa-4ba5-94ae-76aec84184d7-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b9dfd44-g2rvn\" (UID: \"b8a20890-c2fa-4ba5-94ae-76aec84184d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-g2rvn"
Sep 17 16:30:44.084374 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:30:44.084240    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8a20890-c2fa-4ba5-94ae-76aec84184d7-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b9dfd44-g2rvn\" (UID: \"b8a20890-c2fa-4ba5-94ae-76aec84184d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-g2rvn"
Sep 17 16:30:44.084374 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:30:44.084292    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbbq8\" (UniqueName: \"kubernetes.io/projected/b8a20890-c2fa-4ba5-94ae-76aec84184d7-kube-api-access-qbbq8\") pod \"isvc-canary-rollback-predictor-6b9dfd44-g2rvn\" (UID: \"b8a20890-c2fa-4ba5-94ae-76aec84184d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-g2rvn"
Sep 17 16:30:44.084905 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:30:44.084862    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8a20890-c2fa-4ba5-94ae-76aec84184d7-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b9dfd44-g2rvn\" (UID: \"b8a20890-c2fa-4ba5-94ae-76aec84184d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-g2rvn"
Sep 17 16:30:44.085481 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:30:44.085440    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8a20890-c2fa-4ba5-94ae-76aec84184d7-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b9dfd44-g2rvn\" (UID: \"b8a20890-c2fa-4ba5-94ae-76aec84184d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-g2rvn"
Sep 17 16:30:44.085651 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:30:44.085597    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8a20890-c2fa-4ba5-94ae-76aec84184d7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b9dfd44-g2rvn\" (UID: \"b8a20890-c2fa-4ba5-94ae-76aec84184d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-g2rvn"
Sep 17 16:30:44.087274 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:30:44.087247    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8a20890-c2fa-4ba5-94ae-76aec84184d7-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b9dfd44-g2rvn\" (UID: \"b8a20890-c2fa-4ba5-94ae-76aec84184d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-g2rvn"
Sep 17 16:30:44.102073 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:30:44.102040    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbbq8\" (UniqueName: \"kubernetes.io/projected/b8a20890-c2fa-4ba5-94ae-76aec84184d7-kube-api-access-qbbq8\") pod \"isvc-canary-rollback-predictor-6b9dfd44-g2rvn\" (UID: \"b8a20890-c2fa-4ba5-94ae-76aec84184d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-g2rvn"
Sep 17 16:30:44.275439 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:30:44.275367    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-g2rvn"
Sep 17 16:30:44.488719 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:30:44.488677    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd02dd34-37df-478d-8435-fadb17eb6c39-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt\" (UID: \"cd02dd34-37df-478d-8435-fadb17eb6c39\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt"
Sep 17 16:30:44.491315 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:30:44.491292    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd02dd34-37df-478d-8435-fadb17eb6c39-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt\" (UID: \"cd02dd34-37df-478d-8435-fadb17eb6c39\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt"
Sep 17 16:30:44.508382 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:30:44.508347    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9" podUID="b8d67e9f-0329-48a5-a509-ca4192beb3e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 16:30:44.729310 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:30:44.729269    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt"
Sep 17 16:30:45.941067 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:30:45.941026    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9" podUID="b8d67e9f-0329-48a5-a509-ca4192beb3e0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Sep 17 16:30:50.942070 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:30:50.942011    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9" podUID="b8d67e9f-0329-48a5-a509-ca4192beb3e0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Sep 17 16:30:50.942626 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:30:50.942160    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9"
Sep 17 16:30:54.507662 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:30:54.507623    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9" podUID="b8d67e9f-0329-48a5-a509-ca4192beb3e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 16:30:55.941108 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:30:55.941064    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9" podUID="b8d67e9f-0329-48a5-a509-ca4192beb3e0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Sep 17 16:31:00.941972 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:00.941920    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9" podUID="b8d67e9f-0329-48a5-a509-ca4192beb3e0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Sep 17 16:31:04.508309 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:04.508267    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9" podUID="b8d67e9f-0329-48a5-a509-ca4192beb3e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 17 16:31:04.513187 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:04.513163    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9"
Sep 17 16:31:05.941101 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:05.941048    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9" podUID="b8d67e9f-0329-48a5-a509-ca4192beb3e0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Sep 17 16:31:09.081926 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:09.081895    2585 generic.go:356] "Generic (PLEG): container finished" podID="b8d67e9f-0329-48a5-a509-ca4192beb3e0" containerID="1d2834b341e78aaae6b4b613d8d49c8f0db24405e54a5703e6bd10d94acf19cf" exitCode=0
Sep 17 16:31:09.082269 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:09.081960    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9" event={"ID":"b8d67e9f-0329-48a5-a509-ca4192beb3e0","Type":"ContainerDied","Data":"1d2834b341e78aaae6b4b613d8d49c8f0db24405e54a5703e6bd10d94acf19cf"}
Sep 17 16:31:09.082269 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:09.082013    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9" event={"ID":"b8d67e9f-0329-48a5-a509-ca4192beb3e0","Type":"ContainerDied","Data":"e4fe3ff6b38d9eb0929fd5b377b5d51215baf0be1ec8cb11ef7a20f355631234"}
Sep 17 16:31:09.082269 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:09.082034    2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4fe3ff6b38d9eb0929fd5b377b5d51215baf0be1ec8cb11ef7a20f355631234"
Sep 17 16:31:09.100481 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:09.100363    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9"
Sep 17 16:31:09.227053 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:09.226967    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xf9s\" (UniqueName: \"kubernetes.io/projected/b8d67e9f-0329-48a5-a509-ca4192beb3e0-kube-api-access-5xf9s\") pod \"b8d67e9f-0329-48a5-a509-ca4192beb3e0\" (UID: \"b8d67e9f-0329-48a5-a509-ca4192beb3e0\") "
Sep 17 16:31:09.227206 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:09.227066    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8d67e9f-0329-48a5-a509-ca4192beb3e0-cabundle-cert\") pod \"b8d67e9f-0329-48a5-a509-ca4192beb3e0\" (UID: \"b8d67e9f-0329-48a5-a509-ca4192beb3e0\") "
Sep 17 16:31:09.227206 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:09.227095    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8d67e9f-0329-48a5-a509-ca4192beb3e0-kube-rbac-proxy-sar-config\") pod \"b8d67e9f-0329-48a5-a509-ca4192beb3e0\" (UID: \"b8d67e9f-0329-48a5-a509-ca4192beb3e0\") "
Sep 17 16:31:09.227206 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:09.227127    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8d67e9f-0329-48a5-a509-ca4192beb3e0-proxy-tls\") pod \"b8d67e9f-0329-48a5-a509-ca4192beb3e0\" (UID: \"b8d67e9f-0329-48a5-a509-ca4192beb3e0\") "
Sep 17 16:31:09.227206 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:09.227158    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8d67e9f-0329-48a5-a509-ca4192beb3e0-kserve-provision-location\") pod \"b8d67e9f-0329-48a5-a509-ca4192beb3e0\" (UID: \"b8d67e9f-0329-48a5-a509-ca4192beb3e0\") "
Sep 17 16:31:09.227953 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:09.227747    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8d67e9f-0329-48a5-a509-ca4192beb3e0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b8d67e9f-0329-48a5-a509-ca4192beb3e0" (UID: "b8d67e9f-0329-48a5-a509-ca4192beb3e0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:31:09.227953 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:09.227804    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8d67e9f-0329-48a5-a509-ca4192beb3e0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b8d67e9f-0329-48a5-a509-ca4192beb3e0" (UID: "b8d67e9f-0329-48a5-a509-ca4192beb3e0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:31:09.228133 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:09.228016    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8d67e9f-0329-48a5-a509-ca4192beb3e0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b8d67e9f-0329-48a5-a509-ca4192beb3e0" (UID: "b8d67e9f-0329-48a5-a509-ca4192beb3e0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:31:09.231789 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:09.231652    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8d67e9f-0329-48a5-a509-ca4192beb3e0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b8d67e9f-0329-48a5-a509-ca4192beb3e0" (UID: "b8d67e9f-0329-48a5-a509-ca4192beb3e0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:31:09.231789 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:09.231755    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8d67e9f-0329-48a5-a509-ca4192beb3e0-kube-api-access-5xf9s" (OuterVolumeSpecName: "kube-api-access-5xf9s") pod "b8d67e9f-0329-48a5-a509-ca4192beb3e0" (UID: "b8d67e9f-0329-48a5-a509-ca4192beb3e0"). InnerVolumeSpecName "kube-api-access-5xf9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:31:09.328405 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:09.328363    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xf9s\" (UniqueName: \"kubernetes.io/projected/b8d67e9f-0329-48a5-a509-ca4192beb3e0-kube-api-access-5xf9s\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:31:09.328405 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:09.328397    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8d67e9f-0329-48a5-a509-ca4192beb3e0-cabundle-cert\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:31:09.328405 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:09.328406    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8d67e9f-0329-48a5-a509-ca4192beb3e0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:31:09.328661 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:09.328415    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8d67e9f-0329-48a5-a509-ca4192beb3e0-proxy-tls\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:31:09.328661 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:09.328447    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8d67e9f-0329-48a5-a509-ca4192beb3e0-kserve-provision-location\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:31:09.450680 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:09.450649    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-g2rvn"]
Sep 17 16:31:09.565919 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:09.565754    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt"]
Sep 17 16:31:10.082787 ip-10-0-135-43 kubenswrapper[2585]: W0917 16:31:10.082729    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd02dd34_37df_478d_8435_fadb17eb6c39.slice/crio-7432cafe65dbfd4a8fd7702c76015750c1d3ce797ebd250070ab0317f3e0fc54 WatchSource:0}: Error finding container 7432cafe65dbfd4a8fd7702c76015750c1d3ce797ebd250070ab0317f3e0fc54: Status 404 returned error can't find the container with id 7432cafe65dbfd4a8fd7702c76015750c1d3ce797ebd250070ab0317f3e0fc54
Sep 17 16:31:10.083632 ip-10-0-135-43 kubenswrapper[2585]: W0917 16:31:10.083385    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8a20890_c2fa_4ba5_94ae_76aec84184d7.slice/crio-e07d59bf0154f3402806179c3d5a02d92faf2c06ba1c130b576fd24e3e2327ef WatchSource:0}: Error finding container e07d59bf0154f3402806179c3d5a02d92faf2c06ba1c130b576fd24e3e2327ef: Status 404 returned error can't find the container with id e07d59bf0154f3402806179c3d5a02d92faf2c06ba1c130b576fd24e3e2327ef
Sep 17 16:31:10.086712 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:10.086691    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9"
Sep 17 16:31:10.111234 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:10.111204    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9"]
Sep 17 16:31:10.115600 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:10.115577    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-b5sd9"]
Sep 17 16:31:10.516115 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:10.516080    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8d67e9f-0329-48a5-a509-ca4192beb3e0" path="/var/lib/kubelet/pods/b8d67e9f-0329-48a5-a509-ca4192beb3e0/volumes"
Sep 17 16:31:11.096708 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:11.096658    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-g2rvn" event={"ID":"b8a20890-c2fa-4ba5-94ae-76aec84184d7","Type":"ContainerStarted","Data":"e07d59bf0154f3402806179c3d5a02d92faf2c06ba1c130b576fd24e3e2327ef"}
Sep 17 16:31:11.098475 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:11.098445    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt" event={"ID":"cd02dd34-37df-478d-8435-fadb17eb6c39","Type":"ContainerStarted","Data":"7432cafe65dbfd4a8fd7702c76015750c1d3ce797ebd250070ab0317f3e0fc54"}
Sep 17 16:31:17.134206 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:17.134160    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt" event={"ID":"cd02dd34-37df-478d-8435-fadb17eb6c39","Type":"ContainerStarted","Data":"7b5af2e72395774ab2f19b871022cfe84d896077d90cf469c58e87bff757260c"}
Sep 17 16:31:21.156340 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:21.156297    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-g2rvn" event={"ID":"b8a20890-c2fa-4ba5-94ae-76aec84184d7","Type":"ContainerStarted","Data":"b8e184ff363d5040c1f8af26cd72c536fb1f7cc32bcb3219640ba5e20ecb2608"}
Sep 17 16:31:22.165051 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:22.164968    2585 generic.go:356] "Generic (PLEG): container finished" podID="cd02dd34-37df-478d-8435-fadb17eb6c39" containerID="7b5af2e72395774ab2f19b871022cfe84d896077d90cf469c58e87bff757260c" exitCode=0
Sep 17 16:31:22.165523 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:22.165041    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt" event={"ID":"cd02dd34-37df-478d-8435-fadb17eb6c39","Type":"ContainerDied","Data":"7b5af2e72395774ab2f19b871022cfe84d896077d90cf469c58e87bff757260c"}
Sep 17 16:31:50.322200 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:50.322113    2585 generic.go:356] "Generic (PLEG): container finished" podID="b8a20890-c2fa-4ba5-94ae-76aec84184d7" containerID="b8e184ff363d5040c1f8af26cd72c536fb1f7cc32bcb3219640ba5e20ecb2608" exitCode=0
Sep 17 16:31:50.322200 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:50.322184    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-g2rvn" event={"ID":"b8a20890-c2fa-4ba5-94ae-76aec84184d7","Type":"ContainerDied","Data":"b8e184ff363d5040c1f8af26cd72c536fb1f7cc32bcb3219640ba5e20ecb2608"}
Sep 17 16:31:51.329988 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:51.329950    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-g2rvn" event={"ID":"b8a20890-c2fa-4ba5-94ae-76aec84184d7","Type":"ContainerStarted","Data":"b16f52d14e5f597dc33ed31ee4387a260f56d1bef6d07744a20fb3ad815ca7ef"}
Sep 17 16:31:51.330480 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:51.330021    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-g2rvn" event={"ID":"b8a20890-c2fa-4ba5-94ae-76aec84184d7","Type":"ContainerStarted","Data":"f8376bc5319d7de6799d6e317c7fc0b00dd05f7326275208927526e3f609087a"}
Sep 17 16:31:51.330480 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:51.330362    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-g2rvn"
Sep 17 16:31:51.330480 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:51.330456    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-g2rvn"
Sep 17 16:31:51.332323 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:51.332282    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-g2rvn" podUID="b8a20890-c2fa-4ba5-94ae-76aec84184d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 16:31:51.332610 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:51.332587    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt" event={"ID":"cd02dd34-37df-478d-8435-fadb17eb6c39","Type":"ContainerStarted","Data":"d7502fe71002b3258d7f5a74960e8e60a898c852a4e0cc53835daea12a875a73"}
Sep 17 16:31:51.332698 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:51.332657    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt" event={"ID":"cd02dd34-37df-478d-8435-fadb17eb6c39","Type":"ContainerStarted","Data":"68d7be8c4773b7a7d60bf2456789e427f7856bc35c0827a178e2e9cbe77d0f0f"}
Sep 17 16:31:51.332887 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:51.332868    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt"
Sep 17 16:31:51.352231 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:51.352149    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-g2rvn" podStartSLOduration=64.38926594 podStartE2EDuration="1m8.352131029s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.962865089s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:30:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:31:51.351788523 +0000 UTC m=+2267.122120323" watchObservedRunningTime="2026-09-17 16:31:51.352131029 +0000 UTC m=+2267.122462826"
Sep 17 16:31:52.339146 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:52.339110    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8f94m" event={"ID":"e7745c71-f370-4edb-88bb-09092c06c0a3","Type":"ContainerStarted","Data":"19e7ffcb8c95982d6880afe7c645c86633e28a7495725ca6983992262a81a2e7"}
Sep 17 16:31:52.339543 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:52.339172    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8f94m" event={"ID":"e7745c71-f370-4edb-88bb-09092c06c0a3","Type":"ContainerStarted","Data":"355ea334ad439a3a4f5ab1f5f5b1a7fba71a0ffee0353e709e4f9ad85610cbf5"}
Sep 17 16:31:52.339543 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:52.339199    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8f94m"
Sep 17 16:31:52.339674 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:52.339581    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8f94m"
Sep 17 16:31:52.339674 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:52.339638    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt"
Sep 17 16:31:52.339942 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:52.339905    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-g2rvn" podUID="b8a20890-c2fa-4ba5-94ae-76aec84184d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 16:31:52.340703 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:52.340679    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8f94m" podUID="e7745c71-f370-4edb-88bb-09092c06c0a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 16:31:52.340814 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:52.340759    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt" podUID="cd02dd34-37df-478d-8435-fadb17eb6c39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 16:31:52.366104 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:52.366048    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8f94m" podStartSLOduration=4.454028767 podStartE2EDuration="2m3.366029321s" totalImagesPullingTime="1m54.674774496s" totalInitContainerRuntime="4.237226058s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:29:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:31:52.363576531 +0000 UTC m=+2268.133908343" watchObservedRunningTime="2026-09-17 16:31:52.366029321 +0000 UTC m=+2268.136361119"
Sep 17 16:31:52.367091 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:52.367059    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt" podStartSLOduration=64.930959112 podStartE2EDuration="1m9.367050181s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.436091069s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:30:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:31:51.375472813 +0000 UTC m=+2267.145804611" watchObservedRunningTime="2026-09-17 16:31:52.367050181 +0000 UTC m=+2268.137381978"
Sep 17 16:31:53.345177 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:53.345134    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt" podUID="cd02dd34-37df-478d-8435-fadb17eb6c39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 16:31:53.345614 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:53.345188    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8f94m" podUID="e7745c71-f370-4edb-88bb-09092c06c0a3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 17 16:31:57.348757 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:57.348718    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-g2rvn"
Sep 17 16:31:57.349563 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:57.349530    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-g2rvn" podUID="b8a20890-c2fa-4ba5-94ae-76aec84184d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 16:31:58.349908 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:58.349880    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt"
Sep 17 16:31:58.350384 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:58.350283    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8f94m"
Sep 17 16:31:58.350384 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:58.350332    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt" podUID="cd02dd34-37df-478d-8435-fadb17eb6c39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 16:31:58.350871 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:31:58.350853    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8f94m"
Sep 17 16:32:01.837485 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:32:01.837444    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8f94m"]
Sep 17 16:32:01.837967 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:32:01.837729    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8f94m" podUID="e7745c71-f370-4edb-88bb-09092c06c0a3" containerName="kserve-container" containerID="cri-o://355ea334ad439a3a4f5ab1f5f5b1a7fba71a0ffee0353e709e4f9ad85610cbf5" gracePeriod=30
Sep 17 16:32:01.837967 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:32:01.837762    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8f94m" podUID="e7745c71-f370-4edb-88bb-09092c06c0a3" containerName="kube-rbac-proxy" containerID="cri-o://19e7ffcb8c95982d6880afe7c645c86633e28a7495725ca6983992262a81a2e7" gracePeriod=30
Sep 17 16:32:02.389882 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:32:02.389846    2585 generic.go:356] "Generic (PLEG): container finished" podID="e7745c71-f370-4edb-88bb-09092c06c0a3" containerID="19e7ffcb8c95982d6880afe7c645c86633e28a7495725ca6983992262a81a2e7" exitCode=2
Sep 17 16:32:02.390059 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:32:02.389902    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8f94m" event={"ID":"e7745c71-f370-4edb-88bb-09092c06c0a3","Type":"ContainerDied","Data":"19e7ffcb8c95982d6880afe7c645c86633e28a7495725ca6983992262a81a2e7"}
Sep 17 16:32:03.345610 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:32:03.345563    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8f94m" podUID="e7745c71-f370-4edb-88bb-09092c06c0a3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.46:8643/healthz\": dial tcp 10.134.0.46:8643: connect: connection refused"
Sep 17 16:32:04.402607 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:32:04.402572    2585 generic.go:356] "Generic (PLEG): container finished" podID="e7745c71-f370-4edb-88bb-09092c06c0a3" containerID="355ea334ad439a3a4f5ab1f5f5b1a7fba71a0ffee0353e709e4f9ad85610cbf5" exitCode=0
Sep 17 16:32:04.402963 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:32:04.402641    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8f94m" event={"ID":"e7745c71-f370-4edb-88bb-09092c06c0a3","Type":"ContainerDied","Data":"355ea334ad439a3a4f5ab1f5f5b1a7fba71a0ffee0353e709e4f9ad85610cbf5"}
Sep 17 16:32:04.454024 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:32:04.454000    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8f94m"
Sep 17 16:32:04.535950 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:32:04.535860    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7745c71-f370-4edb-88bb-09092c06c0a3-kserve-provision-location\") pod \"e7745c71-f370-4edb-88bb-09092c06c0a3\" (UID: \"e7745c71-f370-4edb-88bb-09092c06c0a3\") "
Sep 17 16:32:04.535950 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:32:04.535913    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzggl\" (UniqueName: \"kubernetes.io/projected/e7745c71-f370-4edb-88bb-09092c06c0a3-kube-api-access-mzggl\") pod \"e7745c71-f370-4edb-88bb-09092c06c0a3\" (UID: \"e7745c71-f370-4edb-88bb-09092c06c0a3\") "
Sep 17 16:32:04.535950 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:32:04.535937    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7745c71-f370-4edb-88bb-09092c06c0a3-proxy-tls\") pod \"e7745c71-f370-4edb-88bb-09092c06c0a3\" (UID: \"e7745c71-f370-4edb-88bb-09092c06c0a3\") "
Sep 17 16:32:04.536213 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:32:04.535975    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7745c71-f370-4edb-88bb-09092c06c0a3-cabundle-cert\") pod \"e7745c71-f370-4edb-88bb-09092c06c0a3\" (UID: \"e7745c71-f370-4edb-88bb-09092c06c0a3\") "
Sep 17 16:32:04.536213 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:32:04.536085    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7745c71-f370-4edb-88bb-09092c06c0a3-kube-rbac-proxy-sar-config\") pod \"e7745c71-f370-4edb-88bb-09092c06c0a3\" (UID: \"e7745c71-f370-4edb-88bb-09092c06c0a3\") "
Sep 17 16:32:04.536479 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:32:04.536332    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7745c71-f370-4edb-88bb-09092c06c0a3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e7745c71-f370-4edb-88bb-09092c06c0a3" (UID: "e7745c71-f370-4edb-88bb-09092c06c0a3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:32:04.536599 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:32:04.536481    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7745c71-f370-4edb-88bb-09092c06c0a3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e7745c71-f370-4edb-88bb-09092c06c0a3" (UID: "e7745c71-f370-4edb-88bb-09092c06c0a3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:32:04.536599 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:32:04.536536    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7745c71-f370-4edb-88bb-09092c06c0a3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e7745c71-f370-4edb-88bb-09092c06c0a3" (UID: "e7745c71-f370-4edb-88bb-09092c06c0a3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:32:04.538127 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:32:04.538101    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7745c71-f370-4edb-88bb-09092c06c0a3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e7745c71-f370-4edb-88bb-09092c06c0a3" (UID: "e7745c71-f370-4edb-88bb-09092c06c0a3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:32:04.538243 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:32:04.538176    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7745c71-f370-4edb-88bb-09092c06c0a3-kube-api-access-mzggl" (OuterVolumeSpecName: "kube-api-access-mzggl") pod "e7745c71-f370-4edb-88bb-09092c06c0a3" (UID: "e7745c71-f370-4edb-88bb-09092c06c0a3"). InnerVolumeSpecName "kube-api-access-mzggl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:32:04.637470 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:32:04.637401    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7745c71-f370-4edb-88bb-09092c06c0a3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:32:04.637470 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:32:04.637462    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7745c71-f370-4edb-88bb-09092c06c0a3-kserve-provision-location\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:32:04.637470 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:32:04.637474    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzggl\" (UniqueName: \"kubernetes.io/projected/e7745c71-f370-4edb-88bb-09092c06c0a3-kube-api-access-mzggl\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:32:04.637710 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:32:04.637484    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7745c71-f370-4edb-88bb-09092c06c0a3-proxy-tls\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:32:04.637710 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:32:04.637494    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7745c71-f370-4edb-88bb-09092c06c0a3-cabundle-cert\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:32:05.410264 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:32:05.410227    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8f94m"
Sep 17 16:32:05.410264 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:32:05.410258    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8f94m" event={"ID":"e7745c71-f370-4edb-88bb-09092c06c0a3","Type":"ContainerDied","Data":"979cf96f9950b7734c161b1772027597c9ff6648dc40b476f39b81236ca1ae0f"}
Sep 17 16:32:05.410796 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:32:05.410325    2585 scope.go:117] "RemoveContainer" containerID="19e7ffcb8c95982d6880afe7c645c86633e28a7495725ca6983992262a81a2e7"
Sep 17 16:32:05.421360 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:32:05.421340    2585 scope.go:117] "RemoveContainer" containerID="355ea334ad439a3a4f5ab1f5f5b1a7fba71a0ffee0353e709e4f9ad85610cbf5"
Sep 17 16:32:05.431714 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:32:05.431695    2585 scope.go:117] "RemoveContainer" containerID="69a5013640ad9986a12f8ae5f77f1e18eaa94c1244a78d3d93a0759b6ed2a4f7"
Sep 17 16:32:05.437882 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:32:05.437839    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8f94m"]
Sep 17 16:32:05.439534 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:32:05.439513    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-8f94m"]
Sep 17 16:32:06.513349 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:32:06.513312    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7745c71-f370-4edb-88bb-09092c06c0a3" path="/var/lib/kubelet/pods/e7745c71-f370-4edb-88bb-09092c06c0a3/volumes"
Sep 17 16:32:07.349961 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:32:07.349922    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-g2rvn" podUID="b8a20890-c2fa-4ba5-94ae-76aec84184d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 16:32:08.350585 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:32:08.350543    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt" podUID="cd02dd34-37df-478d-8435-fadb17eb6c39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 16:32:17.350478 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:32:17.350405    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-g2rvn" podUID="b8a20890-c2fa-4ba5-94ae-76aec84184d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 16:32:18.350710 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:32:18.350661    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt" podUID="cd02dd34-37df-478d-8435-fadb17eb6c39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 16:32:27.350085 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:32:27.350040    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-g2rvn" podUID="b8a20890-c2fa-4ba5-94ae-76aec84184d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 16:32:28.350576 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:32:28.350539    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt" podUID="cd02dd34-37df-478d-8435-fadb17eb6c39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 16:32:37.350219 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:32:37.350174    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-g2rvn" podUID="b8a20890-c2fa-4ba5-94ae-76aec84184d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 16:32:38.351151 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:32:38.351109    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt" podUID="cd02dd34-37df-478d-8435-fadb17eb6c39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 16:32:47.349691 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:32:47.349649    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-g2rvn" podUID="b8a20890-c2fa-4ba5-94ae-76aec84184d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 16:32:48.351022 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:32:48.350977    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt" podUID="cd02dd34-37df-478d-8435-fadb17eb6c39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 16:32:57.350365 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:32:57.350326    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-g2rvn" podUID="b8a20890-c2fa-4ba5-94ae-76aec84184d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 16:32:58.350519 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:32:58.350478    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt" podUID="cd02dd34-37df-478d-8435-fadb17eb6c39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 16:33:04.507375 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:04.507335    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt" podUID="cd02dd34-37df-478d-8435-fadb17eb6c39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 17 16:33:07.350593 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:07.350559    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-g2rvn"
Sep 17 16:33:14.512303 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:14.512268    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt"
Sep 17 16:33:14.664565 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:14.664528    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-g2rvn"]
Sep 17 16:33:14.664815 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:14.664789    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-g2rvn" podUID="b8a20890-c2fa-4ba5-94ae-76aec84184d7" containerName="kserve-container" containerID="cri-o://f8376bc5319d7de6799d6e317c7fc0b00dd05f7326275208927526e3f609087a" gracePeriod=30
Sep 17 16:33:14.664922 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:14.664838    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-g2rvn" podUID="b8a20890-c2fa-4ba5-94ae-76aec84184d7" containerName="kube-rbac-proxy" containerID="cri-o://b16f52d14e5f597dc33ed31ee4387a260f56d1bef6d07744a20fb3ad815ca7ef" gracePeriod=30
Sep 17 16:33:15.757242 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:15.757206    2585 generic.go:356] "Generic (PLEG): container finished" podID="b8a20890-c2fa-4ba5-94ae-76aec84184d7" containerID="b16f52d14e5f597dc33ed31ee4387a260f56d1bef6d07744a20fb3ad815ca7ef" exitCode=2
Sep 17 16:33:15.757681 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:15.757283    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-g2rvn" event={"ID":"b8a20890-c2fa-4ba5-94ae-76aec84184d7","Type":"ContainerDied","Data":"b16f52d14e5f597dc33ed31ee4387a260f56d1bef6d07744a20fb3ad815ca7ef"}
Sep 17 16:33:17.340610 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:17.340561    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-g2rvn" podUID="b8a20890-c2fa-4ba5-94ae-76aec84184d7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.48:8643/healthz\": dial tcp 10.134.0.48:8643: connect: connection refused"
Sep 17 16:33:17.350287 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:17.350253    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-g2rvn" podUID="b8a20890-c2fa-4ba5-94ae-76aec84184d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 17 16:33:18.817795 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:18.817749    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt"]
Sep 17 16:33:18.818261 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:18.818081    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt" podUID="cd02dd34-37df-478d-8435-fadb17eb6c39" containerName="kserve-container" containerID="cri-o://68d7be8c4773b7a7d60bf2456789e427f7856bc35c0827a178e2e9cbe77d0f0f" gracePeriod=30
Sep 17 16:33:18.818261 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:18.818177    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt" podUID="cd02dd34-37df-478d-8435-fadb17eb6c39" containerName="kube-rbac-proxy" containerID="cri-o://d7502fe71002b3258d7f5a74960e8e60a898c852a4e0cc53835daea12a875a73" gracePeriod=30
Sep 17 16:33:18.884186 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:18.884150    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-5m945"]
Sep 17 16:33:18.884550 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:18.884537    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8d67e9f-0329-48a5-a509-ca4192beb3e0" containerName="storage-initializer"
Sep 17 16:33:18.884612 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:18.884552    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8d67e9f-0329-48a5-a509-ca4192beb3e0" containerName="storage-initializer"
Sep 17 16:33:18.884612 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:18.884563    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8d67e9f-0329-48a5-a509-ca4192beb3e0" containerName="kube-rbac-proxy"
Sep 17 16:33:18.884612 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:18.884568    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8d67e9f-0329-48a5-a509-ca4192beb3e0" containerName="kube-rbac-proxy"
Sep 17 16:33:18.884612 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:18.884576    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7745c71-f370-4edb-88bb-09092c06c0a3" containerName="kserve-container"
Sep 17 16:33:18.884612 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:18.884582    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7745c71-f370-4edb-88bb-09092c06c0a3" containerName="kserve-container"
Sep 17 16:33:18.884612 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:18.884590    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7745c71-f370-4edb-88bb-09092c06c0a3" containerName="kube-rbac-proxy"
Sep 17 16:33:18.884612 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:18.884595    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7745c71-f370-4edb-88bb-09092c06c0a3" containerName="kube-rbac-proxy"
Sep 17 16:33:18.884612 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:18.884603    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7745c71-f370-4edb-88bb-09092c06c0a3" containerName="storage-initializer"
Sep 17 16:33:18.884612 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:18.884608    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7745c71-f370-4edb-88bb-09092c06c0a3" containerName="storage-initializer"
Sep 17 16:33:18.884895 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:18.884618    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8d67e9f-0329-48a5-a509-ca4192beb3e0" containerName="kserve-container"
Sep 17 16:33:18.884895 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:18.884622    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8d67e9f-0329-48a5-a509-ca4192beb3e0" containerName="kserve-container"
Sep 17 16:33:18.884895 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:18.884675    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7745c71-f370-4edb-88bb-09092c06c0a3" containerName="kserve-container"
Sep 17 16:33:18.884895 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:18.884683    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8d67e9f-0329-48a5-a509-ca4192beb3e0" containerName="kserve-container"
Sep 17 16:33:18.884895 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:18.884691    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8d67e9f-0329-48a5-a509-ca4192beb3e0" containerName="kube-rbac-proxy"
Sep 17 16:33:18.884895 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:18.884699    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7745c71-f370-4edb-88bb-09092c06c0a3" containerName="kube-rbac-proxy"
Sep 17 16:33:18.886855 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:18.886837    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-5m945"
Sep 17 16:33:18.899009 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:18.898979    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-5m945"]
Sep 17 16:33:18.965983 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:18.965938    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e65398b0-e29f-48c6-86cb-3a33bcf8c3aa-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b9dfd44-5m945\" (UID: \"e65398b0-e29f-48c6-86cb-3a33bcf8c3aa\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-5m945"
Sep 17 16:33:18.965983 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:18.965978    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e65398b0-e29f-48c6-86cb-3a33bcf8c3aa-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b9dfd44-5m945\" (UID: \"e65398b0-e29f-48c6-86cb-3a33bcf8c3aa\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-5m945"
Sep 17 16:33:18.966255 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:18.966006    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e65398b0-e29f-48c6-86cb-3a33bcf8c3aa-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b9dfd44-5m945\" (UID: \"e65398b0-e29f-48c6-86cb-3a33bcf8c3aa\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-5m945"
Sep 17 16:33:18.966255 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:18.966138    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e65398b0-e29f-48c6-86cb-3a33bcf8c3aa-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b9dfd44-5m945\" (UID: \"e65398b0-e29f-48c6-86cb-3a33bcf8c3aa\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-5m945"
Sep 17 16:33:18.966255 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:18.966235    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w77km\" (UniqueName: \"kubernetes.io/projected/e65398b0-e29f-48c6-86cb-3a33bcf8c3aa-kube-api-access-w77km\") pod \"isvc-canary-rollback-predictor-6b9dfd44-5m945\" (UID: \"e65398b0-e29f-48c6-86cb-3a33bcf8c3aa\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-5m945"
Sep 17 16:33:19.067055 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:19.067012    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w77km\" (UniqueName: \"kubernetes.io/projected/e65398b0-e29f-48c6-86cb-3a33bcf8c3aa-kube-api-access-w77km\") pod \"isvc-canary-rollback-predictor-6b9dfd44-5m945\" (UID: \"e65398b0-e29f-48c6-86cb-3a33bcf8c3aa\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-5m945"
Sep 17 16:33:19.067055 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:19.067059    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e65398b0-e29f-48c6-86cb-3a33bcf8c3aa-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b9dfd44-5m945\" (UID: \"e65398b0-e29f-48c6-86cb-3a33bcf8c3aa\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-5m945"
Sep 17 16:33:19.067349 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:19.067079    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e65398b0-e29f-48c6-86cb-3a33bcf8c3aa-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b9dfd44-5m945\" (UID: \"e65398b0-e29f-48c6-86cb-3a33bcf8c3aa\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-5m945"
Sep 17 16:33:19.067349 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:19.067102    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e65398b0-e29f-48c6-86cb-3a33bcf8c3aa-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b9dfd44-5m945\" (UID: \"e65398b0-e29f-48c6-86cb-3a33bcf8c3aa\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-5m945"
Sep 17 16:33:19.067349 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:19.067175    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e65398b0-e29f-48c6-86cb-3a33bcf8c3aa-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b9dfd44-5m945\" (UID: \"e65398b0-e29f-48c6-86cb-3a33bcf8c3aa\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-5m945"
Sep 17 16:33:19.067609 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:19.067589    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e65398b0-e29f-48c6-86cb-3a33bcf8c3aa-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b9dfd44-5m945\" (UID: \"e65398b0-e29f-48c6-86cb-3a33bcf8c3aa\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-5m945"
Sep 17 16:33:19.067949 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:19.067904    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e65398b0-e29f-48c6-86cb-3a33bcf8c3aa-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b9dfd44-5m945\" (UID: \"e65398b0-e29f-48c6-86cb-3a33bcf8c3aa\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-5m945"
Sep 17 16:33:19.068000 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:19.067972    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e65398b0-e29f-48c6-86cb-3a33bcf8c3aa-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b9dfd44-5m945\" (UID: \"e65398b0-e29f-48c6-86cb-3a33bcf8c3aa\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-5m945"
Sep 17 16:33:19.069679 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:19.069661    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e65398b0-e29f-48c6-86cb-3a33bcf8c3aa-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b9dfd44-5m945\" (UID: \"e65398b0-e29f-48c6-86cb-3a33bcf8c3aa\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-5m945"
Sep 17 16:33:19.076714 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:19.076691    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w77km\" (UniqueName: \"kubernetes.io/projected/e65398b0-e29f-48c6-86cb-3a33bcf8c3aa-kube-api-access-w77km\") pod \"isvc-canary-rollback-predictor-6b9dfd44-5m945\" (UID: \"e65398b0-e29f-48c6-86cb-3a33bcf8c3aa\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-5m945"
Sep 17 16:33:19.197927 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:19.197877    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-5m945"
Sep 17 16:33:19.476544 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:19.476517    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-5m945"]
Sep 17 16:33:19.478830 ip-10-0-135-43 kubenswrapper[2585]: W0917 16:33:19.478801    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode65398b0_e29f_48c6_86cb_3a33bcf8c3aa.slice/crio-adc3ae82d0e54d0b2bf3284ee4ff7ab95f6816b16bc97f68661e60334fa59544 WatchSource:0}: Error finding container adc3ae82d0e54d0b2bf3284ee4ff7ab95f6816b16bc97f68661e60334fa59544: Status 404 returned error can't find the container with id adc3ae82d0e54d0b2bf3284ee4ff7ab95f6816b16bc97f68661e60334fa59544
Sep 17 16:33:19.785960 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:19.785926    2585 generic.go:356] "Generic (PLEG): container finished" podID="b8a20890-c2fa-4ba5-94ae-76aec84184d7" containerID="f8376bc5319d7de6799d6e317c7fc0b00dd05f7326275208927526e3f609087a" exitCode=0
Sep 17 16:33:19.786157 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:19.786012    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-g2rvn" event={"ID":"b8a20890-c2fa-4ba5-94ae-76aec84184d7","Type":"ContainerDied","Data":"f8376bc5319d7de6799d6e317c7fc0b00dd05f7326275208927526e3f609087a"}
Sep 17 16:33:19.787967 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:19.787928    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-5m945" event={"ID":"e65398b0-e29f-48c6-86cb-3a33bcf8c3aa","Type":"ContainerStarted","Data":"68e8522e8bff7cb2f23671ace1b2c5a7bd7463d7da1f28621f2e3240693bd4f5"}
Sep 17 16:33:19.788113 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:19.787999    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-5m945" event={"ID":"e65398b0-e29f-48c6-86cb-3a33bcf8c3aa","Type":"ContainerStarted","Data":"adc3ae82d0e54d0b2bf3284ee4ff7ab95f6816b16bc97f68661e60334fa59544"}
Sep 17 16:33:19.790538 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:19.790513    2585 generic.go:356] "Generic (PLEG): container finished" podID="cd02dd34-37df-478d-8435-fadb17eb6c39" containerID="d7502fe71002b3258d7f5a74960e8e60a898c852a4e0cc53835daea12a875a73" exitCode=2
Sep 17 16:33:19.790626 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:19.790552    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt" event={"ID":"cd02dd34-37df-478d-8435-fadb17eb6c39","Type":"ContainerDied","Data":"d7502fe71002b3258d7f5a74960e8e60a898c852a4e0cc53835daea12a875a73"}
Sep 17 16:33:19.797198 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:19.797176    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-g2rvn"
Sep 17 16:33:19.874720 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:19.874682    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8a20890-c2fa-4ba5-94ae-76aec84184d7-kserve-provision-location\") pod \"b8a20890-c2fa-4ba5-94ae-76aec84184d7\" (UID: \"b8a20890-c2fa-4ba5-94ae-76aec84184d7\") "
Sep 17 16:33:19.875214 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:19.874728    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8a20890-c2fa-4ba5-94ae-76aec84184d7-cabundle-cert\") pod \"b8a20890-c2fa-4ba5-94ae-76aec84184d7\" (UID: \"b8a20890-c2fa-4ba5-94ae-76aec84184d7\") "
Sep 17 16:33:19.875214 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:19.874750    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8a20890-c2fa-4ba5-94ae-76aec84184d7-proxy-tls\") pod \"b8a20890-c2fa-4ba5-94ae-76aec84184d7\" (UID: \"b8a20890-c2fa-4ba5-94ae-76aec84184d7\") "
Sep 17 16:33:19.875214 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:19.874794    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8a20890-c2fa-4ba5-94ae-76aec84184d7-kube-rbac-proxy-sar-config\") pod \"b8a20890-c2fa-4ba5-94ae-76aec84184d7\" (UID: \"b8a20890-c2fa-4ba5-94ae-76aec84184d7\") "
Sep 17 16:33:19.875214 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:19.874848    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qbbq8\" (UniqueName: \"kubernetes.io/projected/b8a20890-c2fa-4ba5-94ae-76aec84184d7-kube-api-access-qbbq8\") pod \"b8a20890-c2fa-4ba5-94ae-76aec84184d7\" (UID: \"b8a20890-c2fa-4ba5-94ae-76aec84184d7\") "
Sep 17 16:33:19.875214 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:19.875030    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a20890-c2fa-4ba5-94ae-76aec84184d7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b8a20890-c2fa-4ba5-94ae-76aec84184d7" (UID: "b8a20890-c2fa-4ba5-94ae-76aec84184d7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:33:19.875214 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:19.875118    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8a20890-c2fa-4ba5-94ae-76aec84184d7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b8a20890-c2fa-4ba5-94ae-76aec84184d7" (UID: "b8a20890-c2fa-4ba5-94ae-76aec84184d7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:33:19.875214 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:19.875144    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8a20890-c2fa-4ba5-94ae-76aec84184d7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b8a20890-c2fa-4ba5-94ae-76aec84184d7" (UID: "b8a20890-c2fa-4ba5-94ae-76aec84184d7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:33:19.875590 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:19.875221    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8a20890-c2fa-4ba5-94ae-76aec84184d7-kserve-provision-location\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:33:19.875590 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:19.875246    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8a20890-c2fa-4ba5-94ae-76aec84184d7-cabundle-cert\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:33:19.875590 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:19.875261    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8a20890-c2fa-4ba5-94ae-76aec84184d7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:33:19.877360 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:19.877326    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8a20890-c2fa-4ba5-94ae-76aec84184d7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b8a20890-c2fa-4ba5-94ae-76aec84184d7" (UID: "b8a20890-c2fa-4ba5-94ae-76aec84184d7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:33:19.877542 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:19.877482    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8a20890-c2fa-4ba5-94ae-76aec84184d7-kube-api-access-qbbq8" (OuterVolumeSpecName: "kube-api-access-qbbq8") pod "b8a20890-c2fa-4ba5-94ae-76aec84184d7" (UID: "b8a20890-c2fa-4ba5-94ae-76aec84184d7"). InnerVolumeSpecName "kube-api-access-qbbq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:33:19.976181 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:19.976132    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbbq8\" (UniqueName: \"kubernetes.io/projected/b8a20890-c2fa-4ba5-94ae-76aec84184d7-kube-api-access-qbbq8\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:33:19.976181 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:19.976181    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8a20890-c2fa-4ba5-94ae-76aec84184d7-proxy-tls\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:33:20.800057 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:20.799971    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-g2rvn"
Sep 17 16:33:20.800057 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:20.799978    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-g2rvn" event={"ID":"b8a20890-c2fa-4ba5-94ae-76aec84184d7","Type":"ContainerDied","Data":"e07d59bf0154f3402806179c3d5a02d92faf2c06ba1c130b576fd24e3e2327ef"}
Sep 17 16:33:20.800263 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:20.800065    2585 scope.go:117] "RemoveContainer" containerID="b16f52d14e5f597dc33ed31ee4387a260f56d1bef6d07744a20fb3ad815ca7ef"
Sep 17 16:33:20.808582 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:20.808498    2585 scope.go:117] "RemoveContainer" containerID="f8376bc5319d7de6799d6e317c7fc0b00dd05f7326275208927526e3f609087a"
Sep 17 16:33:20.816891 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:20.816871    2585 scope.go:117] "RemoveContainer" containerID="b8e184ff363d5040c1f8af26cd72c536fb1f7cc32bcb3219640ba5e20ecb2608"
Sep 17 16:33:20.822073 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:20.822044    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-g2rvn"]
Sep 17 16:33:20.823932 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:20.823909    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-g2rvn"]
Sep 17 16:33:22.512977 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:22.512944    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8a20890-c2fa-4ba5-94ae-76aec84184d7" path="/var/lib/kubelet/pods/b8a20890-c2fa-4ba5-94ae-76aec84184d7/volumes"
Sep 17 16:33:23.346302 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:23.346254    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt" podUID="cd02dd34-37df-478d-8435-fadb17eb6c39" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.47:8643/healthz\": dial tcp 10.134.0.47:8643: connect: connection refused"
Sep 17 16:33:23.802074 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:23.802046    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt"
Sep 17 16:33:23.817763 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:23.817720    2585 generic.go:356] "Generic (PLEG): container finished" podID="e65398b0-e29f-48c6-86cb-3a33bcf8c3aa" containerID="68e8522e8bff7cb2f23671ace1b2c5a7bd7463d7da1f28621f2e3240693bd4f5" exitCode=0
Sep 17 16:33:23.817957 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:23.817789    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-5m945" event={"ID":"e65398b0-e29f-48c6-86cb-3a33bcf8c3aa","Type":"ContainerDied","Data":"68e8522e8bff7cb2f23671ace1b2c5a7bd7463d7da1f28621f2e3240693bd4f5"}
Sep 17 16:33:23.820130 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:23.820100    2585 generic.go:356] "Generic (PLEG): container finished" podID="cd02dd34-37df-478d-8435-fadb17eb6c39" containerID="68d7be8c4773b7a7d60bf2456789e427f7856bc35c0827a178e2e9cbe77d0f0f" exitCode=0
Sep 17 16:33:23.820210 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:23.820178    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt"
Sep 17 16:33:23.820280 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:23.820180    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt" event={"ID":"cd02dd34-37df-478d-8435-fadb17eb6c39","Type":"ContainerDied","Data":"68d7be8c4773b7a7d60bf2456789e427f7856bc35c0827a178e2e9cbe77d0f0f"}
Sep 17 16:33:23.820343 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:23.820331    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt" event={"ID":"cd02dd34-37df-478d-8435-fadb17eb6c39","Type":"ContainerDied","Data":"7432cafe65dbfd4a8fd7702c76015750c1d3ce797ebd250070ab0317f3e0fc54"}
Sep 17 16:33:23.820393 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:23.820352    2585 scope.go:117] "RemoveContainer" containerID="d7502fe71002b3258d7f5a74960e8e60a898c852a4e0cc53835daea12a875a73"
Sep 17 16:33:23.831394 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:23.831366    2585 scope.go:117] "RemoveContainer" containerID="68d7be8c4773b7a7d60bf2456789e427f7856bc35c0827a178e2e9cbe77d0f0f"
Sep 17 16:33:23.847147 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:23.847126    2585 scope.go:117] "RemoveContainer" containerID="7b5af2e72395774ab2f19b871022cfe84d896077d90cf469c58e87bff757260c"
Sep 17 16:33:23.860121 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:23.860100    2585 scope.go:117] "RemoveContainer" containerID="d7502fe71002b3258d7f5a74960e8e60a898c852a4e0cc53835daea12a875a73"
Sep 17 16:33:23.860577 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:33:23.860555    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7502fe71002b3258d7f5a74960e8e60a898c852a4e0cc53835daea12a875a73\": container with ID starting with d7502fe71002b3258d7f5a74960e8e60a898c852a4e0cc53835daea12a875a73 not found: ID does not exist" containerID="d7502fe71002b3258d7f5a74960e8e60a898c852a4e0cc53835daea12a875a73"
Sep 17 16:33:23.860687 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:23.860589    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7502fe71002b3258d7f5a74960e8e60a898c852a4e0cc53835daea12a875a73"} err="failed to get container status \"d7502fe71002b3258d7f5a74960e8e60a898c852a4e0cc53835daea12a875a73\": rpc error: code = NotFound desc = could not find container \"d7502fe71002b3258d7f5a74960e8e60a898c852a4e0cc53835daea12a875a73\": container with ID starting with d7502fe71002b3258d7f5a74960e8e60a898c852a4e0cc53835daea12a875a73 not found: ID does not exist"
Sep 17 16:33:23.860687 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:23.860615    2585 scope.go:117] "RemoveContainer" containerID="68d7be8c4773b7a7d60bf2456789e427f7856bc35c0827a178e2e9cbe77d0f0f"
Sep 17 16:33:23.860942 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:33:23.860926    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68d7be8c4773b7a7d60bf2456789e427f7856bc35c0827a178e2e9cbe77d0f0f\": container with ID starting with 68d7be8c4773b7a7d60bf2456789e427f7856bc35c0827a178e2e9cbe77d0f0f not found: ID does not exist" containerID="68d7be8c4773b7a7d60bf2456789e427f7856bc35c0827a178e2e9cbe77d0f0f"
Sep 17 16:33:23.861031 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:23.860949    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68d7be8c4773b7a7d60bf2456789e427f7856bc35c0827a178e2e9cbe77d0f0f"} err="failed to get container status \"68d7be8c4773b7a7d60bf2456789e427f7856bc35c0827a178e2e9cbe77d0f0f\": rpc error: code = NotFound desc = could not find container \"68d7be8c4773b7a7d60bf2456789e427f7856bc35c0827a178e2e9cbe77d0f0f\": container with ID starting with 68d7be8c4773b7a7d60bf2456789e427f7856bc35c0827a178e2e9cbe77d0f0f not found: ID does not exist"
Sep 17 16:33:23.861031 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:23.860969    2585 scope.go:117] "RemoveContainer" containerID="7b5af2e72395774ab2f19b871022cfe84d896077d90cf469c58e87bff757260c"
Sep 17 16:33:23.861246 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:33:23.861225    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b5af2e72395774ab2f19b871022cfe84d896077d90cf469c58e87bff757260c\": container with ID starting with 7b5af2e72395774ab2f19b871022cfe84d896077d90cf469c58e87bff757260c not found: ID does not exist" containerID="7b5af2e72395774ab2f19b871022cfe84d896077d90cf469c58e87bff757260c"
Sep 17 16:33:23.861329 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:23.861250    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b5af2e72395774ab2f19b871022cfe84d896077d90cf469c58e87bff757260c"} err="failed to get container status \"7b5af2e72395774ab2f19b871022cfe84d896077d90cf469c58e87bff757260c\": rpc error: code = NotFound desc = could not find container \"7b5af2e72395774ab2f19b871022cfe84d896077d90cf469c58e87bff757260c\": container with ID starting with 7b5af2e72395774ab2f19b871022cfe84d896077d90cf469c58e87bff757260c not found: ID does not exist"
Sep 17 16:33:23.908095 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:23.908072    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd02dd34-37df-478d-8435-fadb17eb6c39-cabundle-cert\") pod \"cd02dd34-37df-478d-8435-fadb17eb6c39\" (UID: \"cd02dd34-37df-478d-8435-fadb17eb6c39\") "
Sep 17 16:33:23.908203 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:23.908126    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ltmd\" (UniqueName: \"kubernetes.io/projected/cd02dd34-37df-478d-8435-fadb17eb6c39-kube-api-access-2ltmd\") pod \"cd02dd34-37df-478d-8435-fadb17eb6c39\" (UID: \"cd02dd34-37df-478d-8435-fadb17eb6c39\") "
Sep 17 16:33:23.908260 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:23.908241    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd02dd34-37df-478d-8435-fadb17eb6c39-kserve-provision-location\") pod \"cd02dd34-37df-478d-8435-fadb17eb6c39\" (UID: \"cd02dd34-37df-478d-8435-fadb17eb6c39\") "
Sep 17 16:33:23.908299 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:23.908287    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cd02dd34-37df-478d-8435-fadb17eb6c39-kube-rbac-proxy-sar-config\") pod \"cd02dd34-37df-478d-8435-fadb17eb6c39\" (UID: \"cd02dd34-37df-478d-8435-fadb17eb6c39\") "
Sep 17 16:33:23.908344 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:23.908305    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd02dd34-37df-478d-8435-fadb17eb6c39-proxy-tls\") pod \"cd02dd34-37df-478d-8435-fadb17eb6c39\" (UID: \"cd02dd34-37df-478d-8435-fadb17eb6c39\") "
Sep 17 16:33:23.908562 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:23.908529    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd02dd34-37df-478d-8435-fadb17eb6c39-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cd02dd34-37df-478d-8435-fadb17eb6c39" (UID: "cd02dd34-37df-478d-8435-fadb17eb6c39"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:33:23.908562 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:23.908546    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd02dd34-37df-478d-8435-fadb17eb6c39-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cd02dd34-37df-478d-8435-fadb17eb6c39" (UID: "cd02dd34-37df-478d-8435-fadb17eb6c39"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:33:23.908740 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:23.908622    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd02dd34-37df-478d-8435-fadb17eb6c39-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cd02dd34-37df-478d-8435-fadb17eb6c39" (UID: "cd02dd34-37df-478d-8435-fadb17eb6c39"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:33:23.908740 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:23.908707    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd02dd34-37df-478d-8435-fadb17eb6c39-kserve-provision-location\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:33:23.908740 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:23.908728    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cd02dd34-37df-478d-8435-fadb17eb6c39-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:33:23.908878 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:23.908743    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd02dd34-37df-478d-8435-fadb17eb6c39-cabundle-cert\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:33:23.910461 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:23.910435    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd02dd34-37df-478d-8435-fadb17eb6c39-kube-api-access-2ltmd" (OuterVolumeSpecName: "kube-api-access-2ltmd") pod "cd02dd34-37df-478d-8435-fadb17eb6c39" (UID: "cd02dd34-37df-478d-8435-fadb17eb6c39"). InnerVolumeSpecName "kube-api-access-2ltmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:33:23.910565 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:23.910499    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd02dd34-37df-478d-8435-fadb17eb6c39-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cd02dd34-37df-478d-8435-fadb17eb6c39" (UID: "cd02dd34-37df-478d-8435-fadb17eb6c39"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:33:24.009489 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:24.009449    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ltmd\" (UniqueName: \"kubernetes.io/projected/cd02dd34-37df-478d-8435-fadb17eb6c39-kube-api-access-2ltmd\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:33:24.009489 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:24.009489    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd02dd34-37df-478d-8435-fadb17eb6c39-proxy-tls\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:33:24.143783 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:24.143745    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt"]
Sep 17 16:33:24.147359 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:24.147330    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f8f8b45-wbzmt"]
Sep 17 16:33:24.513647 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:24.513607    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd02dd34-37df-478d-8435-fadb17eb6c39" path="/var/lib/kubelet/pods/cd02dd34-37df-478d-8435-fadb17eb6c39/volumes"
Sep 17 16:33:24.827182 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:24.827089    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-5m945" event={"ID":"e65398b0-e29f-48c6-86cb-3a33bcf8c3aa","Type":"ContainerStarted","Data":"81e7e9e3f9058a60cb55bef4e0a1a1e23340aa1da64fd4415786087f5b96d246"}
Sep 17 16:33:24.827182 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:24.827154    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-5m945" event={"ID":"e65398b0-e29f-48c6-86cb-3a33bcf8c3aa","Type":"ContainerStarted","Data":"2e2b304511339f48eb9751598c7bfc20f69faa73615df7c5a34562a2b1ec76c3"}
Sep 17 16:33:24.827676 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:24.827358    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-5m945"
Sep 17 16:33:24.852567 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:24.852518    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-5m945" podStartSLOduration=2.774342067 podStartE2EDuration="6.852502444s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.078160377s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:33:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:33:24.851385463 +0000 UTC m=+2360.621717295" watchObservedRunningTime="2026-09-17 16:33:24.852502444 +0000 UTC m=+2360.622834241"
Sep 17 16:33:25.833110 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:25.833063    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-5m945"
Sep 17 16:33:25.834572 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:25.834540    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-5m945" podUID="e65398b0-e29f-48c6-86cb-3a33bcf8c3aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 16:33:26.837649 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:26.837604    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-5m945" podUID="e65398b0-e29f-48c6-86cb-3a33bcf8c3aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 16:33:31.842046 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:31.842016    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-5m945"
Sep 17 16:33:31.842658 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:31.842629    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-5m945" podUID="e65398b0-e29f-48c6-86cb-3a33bcf8c3aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 16:33:41.843352 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:41.843311    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-5m945" podUID="e65398b0-e29f-48c6-86cb-3a33bcf8c3aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 16:33:51.842713 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:51.842673    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-5m945" podUID="e65398b0-e29f-48c6-86cb-3a33bcf8c3aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 16:33:52.927413 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:52.927374    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hhvtx"]
Sep 17 16:33:52.927806 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:52.927757    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd02dd34-37df-478d-8435-fadb17eb6c39" containerName="kserve-container"
Sep 17 16:33:52.927806 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:52.927769    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd02dd34-37df-478d-8435-fadb17eb6c39" containerName="kserve-container"
Sep 17 16:33:52.927806 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:52.927781    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd02dd34-37df-478d-8435-fadb17eb6c39" containerName="storage-initializer"
Sep 17 16:33:52.927806 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:52.927787    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd02dd34-37df-478d-8435-fadb17eb6c39" containerName="storage-initializer"
Sep 17 16:33:52.927806 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:52.927803    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8a20890-c2fa-4ba5-94ae-76aec84184d7" containerName="storage-initializer"
Sep 17 16:33:52.927975 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:52.927812    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8a20890-c2fa-4ba5-94ae-76aec84184d7" containerName="storage-initializer"
Sep 17 16:33:52.927975 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:52.927820    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8a20890-c2fa-4ba5-94ae-76aec84184d7" containerName="kserve-container"
Sep 17 16:33:52.927975 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:52.927825    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8a20890-c2fa-4ba5-94ae-76aec84184d7" containerName="kserve-container"
Sep 17 16:33:52.927975 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:52.927833    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8a20890-c2fa-4ba5-94ae-76aec84184d7" containerName="kube-rbac-proxy"
Sep 17 16:33:52.927975 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:52.927838    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8a20890-c2fa-4ba5-94ae-76aec84184d7" containerName="kube-rbac-proxy"
Sep 17 16:33:52.927975 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:52.927849    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd02dd34-37df-478d-8435-fadb17eb6c39" containerName="kube-rbac-proxy"
Sep 17 16:33:52.927975 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:52.927854    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd02dd34-37df-478d-8435-fadb17eb6c39" containerName="kube-rbac-proxy"
Sep 17 16:33:52.927975 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:52.927908    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd02dd34-37df-478d-8435-fadb17eb6c39" containerName="kube-rbac-proxy"
Sep 17 16:33:52.927975 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:52.927918    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8a20890-c2fa-4ba5-94ae-76aec84184d7" containerName="kube-rbac-proxy"
Sep 17 16:33:52.927975 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:52.927923    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8a20890-c2fa-4ba5-94ae-76aec84184d7" containerName="kserve-container"
Sep 17 16:33:52.927975 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:52.927929    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd02dd34-37df-478d-8435-fadb17eb6c39" containerName="kserve-container"
Sep 17 16:33:52.930156 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:52.930139    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hhvtx"
Sep 17 16:33:52.932835 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:52.932806    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
Sep 17 16:33:52.932979 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:52.932872    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 17 16:33:52.953439 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:52.953392    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hhvtx"]
Sep 17 16:33:53.065877 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:53.065829    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76e07fe9-066f-4da0-aca2-bc2a7f64cc82-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hhvtx\" (UID: \"76e07fe9-066f-4da0-aca2-bc2a7f64cc82\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hhvtx"
Sep 17 16:33:53.066076 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:53.065885    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/76e07fe9-066f-4da0-aca2-bc2a7f64cc82-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hhvtx\" (UID: \"76e07fe9-066f-4da0-aca2-bc2a7f64cc82\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hhvtx"
Sep 17 16:33:53.066076 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:53.065972    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xq2r\" (UniqueName: \"kubernetes.io/projected/76e07fe9-066f-4da0-aca2-bc2a7f64cc82-kube-api-access-4xq2r\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hhvtx\" (UID: \"76e07fe9-066f-4da0-aca2-bc2a7f64cc82\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hhvtx"
Sep 17 16:33:53.066076 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:53.066009    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/76e07fe9-066f-4da0-aca2-bc2a7f64cc82-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hhvtx\" (UID: \"76e07fe9-066f-4da0-aca2-bc2a7f64cc82\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hhvtx"
Sep 17 16:33:53.066076 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:53.066069    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76e07fe9-066f-4da0-aca2-bc2a7f64cc82-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hhvtx\" (UID: \"76e07fe9-066f-4da0-aca2-bc2a7f64cc82\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hhvtx"
Sep 17 16:33:53.167204 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:53.167159    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xq2r\" (UniqueName: \"kubernetes.io/projected/76e07fe9-066f-4da0-aca2-bc2a7f64cc82-kube-api-access-4xq2r\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hhvtx\" (UID: \"76e07fe9-066f-4da0-aca2-bc2a7f64cc82\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hhvtx"
Sep 17 16:33:53.167204 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:53.167206    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/76e07fe9-066f-4da0-aca2-bc2a7f64cc82-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hhvtx\" (UID: \"76e07fe9-066f-4da0-aca2-bc2a7f64cc82\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hhvtx"
Sep 17 16:33:53.167509 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:53.167234    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76e07fe9-066f-4da0-aca2-bc2a7f64cc82-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hhvtx\" (UID: \"76e07fe9-066f-4da0-aca2-bc2a7f64cc82\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hhvtx"
Sep 17 16:33:53.167509 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:53.167276    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76e07fe9-066f-4da0-aca2-bc2a7f64cc82-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hhvtx\" (UID: \"76e07fe9-066f-4da0-aca2-bc2a7f64cc82\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hhvtx"
Sep 17 16:33:53.167509 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:53.167300    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/76e07fe9-066f-4da0-aca2-bc2a7f64cc82-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hhvtx\" (UID: \"76e07fe9-066f-4da0-aca2-bc2a7f64cc82\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hhvtx"
Sep 17 16:33:53.167509 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:33:53.167372    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-serving-cert: secret "isvc-xgboost-v2-mlserver-predictor-serving-cert" not found
Sep 17 16:33:53.167509 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:33:53.167475    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76e07fe9-066f-4da0-aca2-bc2a7f64cc82-proxy-tls podName:76e07fe9-066f-4da0-aca2-bc2a7f64cc82 nodeName:}" failed. No retries permitted until 2026-09-17 16:33:53.667454529 +0000 UTC m=+2389.437786316 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/76e07fe9-066f-4da0-aca2-bc2a7f64cc82-proxy-tls") pod "isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hhvtx" (UID: "76e07fe9-066f-4da0-aca2-bc2a7f64cc82") : secret "isvc-xgboost-v2-mlserver-predictor-serving-cert" not found
Sep 17 16:33:53.167797 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:53.167755    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76e07fe9-066f-4da0-aca2-bc2a7f64cc82-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hhvtx\" (UID: \"76e07fe9-066f-4da0-aca2-bc2a7f64cc82\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hhvtx"
Sep 17 16:33:53.168165 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:53.168141    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76e07fe9-066f-4da0-aca2-bc2a7f64cc82-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hhvtx\" (UID: \"76e07fe9-066f-4da0-aca2-bc2a7f64cc82\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hhvtx"
Sep 17 16:33:53.168203 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:53.168141    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/76e07fe9-066f-4da0-aca2-bc2a7f64cc82-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hhvtx\" (UID: \"76e07fe9-066f-4da0-aca2-bc2a7f64cc82\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hhvtx"
Sep 17 16:33:53.180350 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:53.180281    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xq2r\" (UniqueName: \"kubernetes.io/projected/76e07fe9-066f-4da0-aca2-bc2a7f64cc82-kube-api-access-4xq2r\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hhvtx\" (UID: \"76e07fe9-066f-4da0-aca2-bc2a7f64cc82\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hhvtx"
Sep 17 16:33:53.672658 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:53.672615    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/76e07fe9-066f-4da0-aca2-bc2a7f64cc82-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hhvtx\" (UID: \"76e07fe9-066f-4da0-aca2-bc2a7f64cc82\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hhvtx"
Sep 17 16:33:53.675150 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:53.675128    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/76e07fe9-066f-4da0-aca2-bc2a7f64cc82-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hhvtx\" (UID: \"76e07fe9-066f-4da0-aca2-bc2a7f64cc82\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hhvtx"
Sep 17 16:33:53.840992 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:53.840938    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hhvtx"
Sep 17 16:33:54.123837 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:54.123807    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hhvtx"]
Sep 17 16:33:54.126231 ip-10-0-135-43 kubenswrapper[2585]: W0917 16:33:54.126206    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76e07fe9_066f_4da0_aca2_bc2a7f64cc82.slice/crio-c0647f3cee504079a0df5b23906969967fcb07f412966fb96f0d23f7f2b7d768 WatchSource:0}: Error finding container c0647f3cee504079a0df5b23906969967fcb07f412966fb96f0d23f7f2b7d768: Status 404 returned error can't find the container with id c0647f3cee504079a0df5b23906969967fcb07f412966fb96f0d23f7f2b7d768
Sep 17 16:33:54.970114 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:54.970074    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hhvtx" event={"ID":"76e07fe9-066f-4da0-aca2-bc2a7f64cc82","Type":"ContainerStarted","Data":"7fbc4506405e4077a4f29f6c0475bcc5c0aecba2700f21729889f525f53e869f"}
Sep 17 16:33:54.970300 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:54.970141    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hhvtx" event={"ID":"76e07fe9-066f-4da0-aca2-bc2a7f64cc82","Type":"ContainerStarted","Data":"c0647f3cee504079a0df5b23906969967fcb07f412966fb96f0d23f7f2b7d768"}
Sep 17 16:33:57.985338 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:57.985296    2585 generic.go:356] "Generic (PLEG): container finished" podID="76e07fe9-066f-4da0-aca2-bc2a7f64cc82" containerID="7fbc4506405e4077a4f29f6c0475bcc5c0aecba2700f21729889f525f53e869f" exitCode=0
Sep 17 16:33:57.985869 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:57.985372    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hhvtx" event={"ID":"76e07fe9-066f-4da0-aca2-bc2a7f64cc82","Type":"ContainerDied","Data":"7fbc4506405e4077a4f29f6c0475bcc5c0aecba2700f21729889f525f53e869f"}
Sep 17 16:33:58.992750 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:58.992714    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hhvtx" event={"ID":"76e07fe9-066f-4da0-aca2-bc2a7f64cc82","Type":"ContainerStarted","Data":"c227097908a637446f035dce21aa81c10984d5fbdc85c7eab94404547b5a6619"}
Sep 17 16:33:58.993257 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:58.992789    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hhvtx" event={"ID":"76e07fe9-066f-4da0-aca2-bc2a7f64cc82","Type":"ContainerStarted","Data":"c40454f446cf3934a346bea554e3fe848dbe931e825c4c7d14b80b7a71dda332"}
Sep 17 16:33:58.993257 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:58.993028    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hhvtx"
Sep 17 16:33:59.026928 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:59.026875    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hhvtx" podStartSLOduration=3.342187359 podStartE2EDuration="7.026858212s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.684670853s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:33:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:33:59.025741093 +0000 UTC m=+2394.796072894" watchObservedRunningTime="2026-09-17 16:33:59.026858212 +0000 UTC m=+2394.797190009"
Sep 17 16:33:59.998062 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:33:59.998022    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hhvtx"
Sep 17 16:34:01.842833 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:01.842790    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-5m945" podUID="e65398b0-e29f-48c6-86cb-3a33bcf8c3aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 16:34:06.008101 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:06.008071    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hhvtx"
Sep 17 16:34:11.842744 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:11.842698    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-5m945" podUID="e65398b0-e29f-48c6-86cb-3a33bcf8c3aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 16:34:12.104603 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:12.104521    2585 scope.go:117] "RemoveContainer" containerID="1d2834b341e78aaae6b4b613d8d49c8f0db24405e54a5703e6bd10d94acf19cf"
Sep 17 16:34:12.113221 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:12.113199    2585 scope.go:117] "RemoveContainer" containerID="b8c9f85b207bcc3fae882b25610226cc092680631fdc665c64da6c351b5ddd4d"
Sep 17 16:34:12.122531 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:12.122508    2585 scope.go:117] "RemoveContainer" containerID="01b58af769f2d21d1ace8d911592161618a944d69519cc835c5133148fb0f9d1"
Sep 17 16:34:21.843290 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:21.843205    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-5m945" podUID="e65398b0-e29f-48c6-86cb-3a33bcf8c3aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 16:34:31.842670 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:31.842629    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-5m945" podUID="e65398b0-e29f-48c6-86cb-3a33bcf8c3aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 16:34:36.014294 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:36.014244    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hhvtx"
Sep 17 16:34:41.843140 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:41.843109    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-5m945"
Sep 17 16:34:42.839365 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:42.839327    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hhvtx"]
Sep 17 16:34:42.839655 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:42.839628    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hhvtx" podUID="76e07fe9-066f-4da0-aca2-bc2a7f64cc82" containerName="kserve-container" containerID="cri-o://c40454f446cf3934a346bea554e3fe848dbe931e825c4c7d14b80b7a71dda332" gracePeriod=30
Sep 17 16:34:42.839797 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:42.839704    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hhvtx" podUID="76e07fe9-066f-4da0-aca2-bc2a7f64cc82" containerName="kube-rbac-proxy" containerID="cri-o://c227097908a637446f035dce21aa81c10984d5fbdc85c7eab94404547b5a6619" gracePeriod=30
Sep 17 16:34:43.205553 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:43.205520    2585 generic.go:356] "Generic (PLEG): container finished" podID="76e07fe9-066f-4da0-aca2-bc2a7f64cc82" containerID="c227097908a637446f035dce21aa81c10984d5fbdc85c7eab94404547b5a6619" exitCode=2
Sep 17 16:34:43.205971 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:43.205593    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hhvtx" event={"ID":"76e07fe9-066f-4da0-aca2-bc2a7f64cc82","Type":"ContainerDied","Data":"c227097908a637446f035dce21aa81c10984d5fbdc85c7eab94404547b5a6619"}
Sep 17 16:34:46.003625 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:46.003561    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hhvtx" podUID="76e07fe9-066f-4da0-aca2-bc2a7f64cc82" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.50:8643/healthz\": dial tcp 10.134.0.50:8643: connect: connection refused"
Sep 17 16:34:46.009086 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:46.009057    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hhvtx" podUID="76e07fe9-066f-4da0-aca2-bc2a7f64cc82" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.50:8080/v2/models/isvc-xgboost-v2-mlserver/ready\": dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 17 16:34:47.769257 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:47.769136    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2b47p_a1850a4d-4d01-4a71-8636-b5918f1426c2/ovn-acl-logging/0.log"
Sep 17 16:34:47.773493 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:47.773475    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2b47p_a1850a4d-4d01-4a71-8636-b5918f1426c2/ovn-acl-logging/0.log"
Sep 17 16:34:48.063566 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:48.063488    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d7cvs"]
Sep 17 16:34:48.067127 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:48.067103    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d7cvs"
Sep 17 16:34:48.069671 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:48.069650    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
Sep 17 16:34:48.069794 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:48.069684    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 17 16:34:48.079884 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:48.079858    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d7cvs"]
Sep 17 16:34:48.234740 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:48.234699    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bc509df-381e-4e44-b59e-01e4b4cc1325-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-d7cvs\" (UID: \"7bc509df-381e-4e44-b59e-01e4b4cc1325\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d7cvs"
Sep 17 16:34:48.234921 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:48.234758    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bc509df-381e-4e44-b59e-01e4b4cc1325-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-d7cvs\" (UID: \"7bc509df-381e-4e44-b59e-01e4b4cc1325\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d7cvs"
Sep 17 16:34:48.234921 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:48.234822    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bc509df-381e-4e44-b59e-01e4b4cc1325-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-d7cvs\" (UID: \"7bc509df-381e-4e44-b59e-01e4b4cc1325\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d7cvs"
Sep 17 16:34:48.234921 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:48.234867    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w49l\" (UniqueName: \"kubernetes.io/projected/7bc509df-381e-4e44-b59e-01e4b4cc1325-kube-api-access-4w49l\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-d7cvs\" (UID: \"7bc509df-381e-4e44-b59e-01e4b4cc1325\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d7cvs"
Sep 17 16:34:48.234921 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:48.234888    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7bc509df-381e-4e44-b59e-01e4b4cc1325-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-d7cvs\" (UID: \"7bc509df-381e-4e44-b59e-01e4b4cc1325\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d7cvs"
Sep 17 16:34:48.335539 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:48.335412    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bc509df-381e-4e44-b59e-01e4b4cc1325-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-d7cvs\" (UID: \"7bc509df-381e-4e44-b59e-01e4b4cc1325\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d7cvs"
Sep 17 16:34:48.335539 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:48.335498    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bc509df-381e-4e44-b59e-01e4b4cc1325-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-d7cvs\" (UID: \"7bc509df-381e-4e44-b59e-01e4b4cc1325\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d7cvs"
Sep 17 16:34:48.335786 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:48.335546    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w49l\" (UniqueName: \"kubernetes.io/projected/7bc509df-381e-4e44-b59e-01e4b4cc1325-kube-api-access-4w49l\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-d7cvs\" (UID: \"7bc509df-381e-4e44-b59e-01e4b4cc1325\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d7cvs"
Sep 17 16:34:48.335786 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:48.335568    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7bc509df-381e-4e44-b59e-01e4b4cc1325-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-d7cvs\" (UID: \"7bc509df-381e-4e44-b59e-01e4b4cc1325\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d7cvs"
Sep 17 16:34:48.335786 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:48.335628    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bc509df-381e-4e44-b59e-01e4b4cc1325-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-d7cvs\" (UID: \"7bc509df-381e-4e44-b59e-01e4b4cc1325\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d7cvs"
Sep 17 16:34:48.336058 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:48.336036    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bc509df-381e-4e44-b59e-01e4b4cc1325-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-d7cvs\" (UID: \"7bc509df-381e-4e44-b59e-01e4b4cc1325\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d7cvs"
Sep 17 16:34:48.336360 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:48.336337    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bc509df-381e-4e44-b59e-01e4b4cc1325-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-d7cvs\" (UID: \"7bc509df-381e-4e44-b59e-01e4b4cc1325\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d7cvs"
Sep 17 16:34:48.336404 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:48.336337    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7bc509df-381e-4e44-b59e-01e4b4cc1325-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-d7cvs\" (UID: \"7bc509df-381e-4e44-b59e-01e4b4cc1325\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d7cvs"
Sep 17 16:34:48.337910 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:48.337892    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bc509df-381e-4e44-b59e-01e4b4cc1325-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-d7cvs\" (UID: \"7bc509df-381e-4e44-b59e-01e4b4cc1325\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d7cvs"
Sep 17 16:34:48.344212 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:48.344189    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w49l\" (UniqueName: \"kubernetes.io/projected/7bc509df-381e-4e44-b59e-01e4b4cc1325-kube-api-access-4w49l\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-d7cvs\" (UID: \"7bc509df-381e-4e44-b59e-01e4b4cc1325\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d7cvs"
Sep 17 16:34:48.377188 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:48.377144    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d7cvs"
Sep 17 16:34:48.657806 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:48.657618    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d7cvs"]
Sep 17 16:34:48.660206 ip-10-0-135-43 kubenswrapper[2585]: W0917 16:34:48.660177    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bc509df_381e_4e44_b59e_01e4b4cc1325.slice/crio-4fe9a2d1a1f7214d7705340d3e8ecc479e9fb223b48458a82e646b9dae463fd7 WatchSource:0}: Error finding container 4fe9a2d1a1f7214d7705340d3e8ecc479e9fb223b48458a82e646b9dae463fd7: Status 404 returned error can't find the container with id 4fe9a2d1a1f7214d7705340d3e8ecc479e9fb223b48458a82e646b9dae463fd7
Sep 17 16:34:48.945948 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:48.945855    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-5m945"]
Sep 17 16:34:48.946366 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:48.946165    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-5m945" podUID="e65398b0-e29f-48c6-86cb-3a33bcf8c3aa" containerName="kserve-container" containerID="cri-o://2e2b304511339f48eb9751598c7bfc20f69faa73615df7c5a34562a2b1ec76c3" gracePeriod=30
Sep 17 16:34:48.946366 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:48.946195    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-5m945" podUID="e65398b0-e29f-48c6-86cb-3a33bcf8c3aa" containerName="kube-rbac-proxy" containerID="cri-o://81e7e9e3f9058a60cb55bef4e0a1a1e23340aa1da64fd4415786087f5b96d246" gracePeriod=30
Sep 17 16:34:49.239896 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:49.239859    2585 generic.go:356] "Generic (PLEG): container finished" podID="e65398b0-e29f-48c6-86cb-3a33bcf8c3aa" containerID="81e7e9e3f9058a60cb55bef4e0a1a1e23340aa1da64fd4415786087f5b96d246" exitCode=2
Sep 17 16:34:49.240071 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:49.239931    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-5m945" event={"ID":"e65398b0-e29f-48c6-86cb-3a33bcf8c3aa","Type":"ContainerDied","Data":"81e7e9e3f9058a60cb55bef4e0a1a1e23340aa1da64fd4415786087f5b96d246"}
Sep 17 16:34:49.241502 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:49.241475    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d7cvs" event={"ID":"7bc509df-381e-4e44-b59e-01e4b4cc1325","Type":"ContainerStarted","Data":"15506425acaf09bae724716a1e2ab34fd0b43148a25ea6c68cfedc323357eda1"}
Sep 17 16:34:49.241627 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:49.241538    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d7cvs" event={"ID":"7bc509df-381e-4e44-b59e-01e4b4cc1325","Type":"ContainerStarted","Data":"4fe9a2d1a1f7214d7705340d3e8ecc479e9fb223b48458a82e646b9dae463fd7"}
Sep 17 16:34:49.680932 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:49.680908    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hhvtx"
Sep 17 16:34:49.851008 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:49.850905    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76e07fe9-066f-4da0-aca2-bc2a7f64cc82-cabundle-cert\") pod \"76e07fe9-066f-4da0-aca2-bc2a7f64cc82\" (UID: \"76e07fe9-066f-4da0-aca2-bc2a7f64cc82\") "
Sep 17 16:34:49.851008 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:49.850950    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/76e07fe9-066f-4da0-aca2-bc2a7f64cc82-kube-rbac-proxy-sar-config\") pod \"76e07fe9-066f-4da0-aca2-bc2a7f64cc82\" (UID: \"76e07fe9-066f-4da0-aca2-bc2a7f64cc82\") "
Sep 17 16:34:49.851008 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:49.850975    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4xq2r\" (UniqueName: \"kubernetes.io/projected/76e07fe9-066f-4da0-aca2-bc2a7f64cc82-kube-api-access-4xq2r\") pod \"76e07fe9-066f-4da0-aca2-bc2a7f64cc82\" (UID: \"76e07fe9-066f-4da0-aca2-bc2a7f64cc82\") "
Sep 17 16:34:49.851276 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:49.851029    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/76e07fe9-066f-4da0-aca2-bc2a7f64cc82-proxy-tls\") pod \"76e07fe9-066f-4da0-aca2-bc2a7f64cc82\" (UID: \"76e07fe9-066f-4da0-aca2-bc2a7f64cc82\") "
Sep 17 16:34:49.851276 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:49.851086    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76e07fe9-066f-4da0-aca2-bc2a7f64cc82-kserve-provision-location\") pod \"76e07fe9-066f-4da0-aca2-bc2a7f64cc82\" (UID: \"76e07fe9-066f-4da0-aca2-bc2a7f64cc82\") "
Sep 17 16:34:49.851276 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:49.851252    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76e07fe9-066f-4da0-aca2-bc2a7f64cc82-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "76e07fe9-066f-4da0-aca2-bc2a7f64cc82" (UID: "76e07fe9-066f-4da0-aca2-bc2a7f64cc82"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:34:49.851414 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:49.851334    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76e07fe9-066f-4da0-aca2-bc2a7f64cc82-cabundle-cert\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:34:49.851414 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:49.851355    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76e07fe9-066f-4da0-aca2-bc2a7f64cc82-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "76e07fe9-066f-4da0-aca2-bc2a7f64cc82" (UID: "76e07fe9-066f-4da0-aca2-bc2a7f64cc82"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:34:49.851516 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:49.851446    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76e07fe9-066f-4da0-aca2-bc2a7f64cc82-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "76e07fe9-066f-4da0-aca2-bc2a7f64cc82" (UID: "76e07fe9-066f-4da0-aca2-bc2a7f64cc82"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:34:49.853144 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:49.853124    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76e07fe9-066f-4da0-aca2-bc2a7f64cc82-kube-api-access-4xq2r" (OuterVolumeSpecName: "kube-api-access-4xq2r") pod "76e07fe9-066f-4da0-aca2-bc2a7f64cc82" (UID: "76e07fe9-066f-4da0-aca2-bc2a7f64cc82"). InnerVolumeSpecName "kube-api-access-4xq2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:34:49.853197 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:49.853161    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76e07fe9-066f-4da0-aca2-bc2a7f64cc82-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "76e07fe9-066f-4da0-aca2-bc2a7f64cc82" (UID: "76e07fe9-066f-4da0-aca2-bc2a7f64cc82"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:34:49.952303 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:49.952264    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/76e07fe9-066f-4da0-aca2-bc2a7f64cc82-proxy-tls\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:34:49.952303 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:49.952295    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76e07fe9-066f-4da0-aca2-bc2a7f64cc82-kserve-provision-location\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:34:49.952303 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:49.952304    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/76e07fe9-066f-4da0-aca2-bc2a7f64cc82-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:34:49.952303 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:49.952313    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xq2r\" (UniqueName: \"kubernetes.io/projected/76e07fe9-066f-4da0-aca2-bc2a7f64cc82-kube-api-access-4xq2r\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:34:50.247488 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:50.247448    2585 generic.go:356] "Generic (PLEG): container finished" podID="76e07fe9-066f-4da0-aca2-bc2a7f64cc82" containerID="c40454f446cf3934a346bea554e3fe848dbe931e825c4c7d14b80b7a71dda332" exitCode=0
Sep 17 16:34:50.247664 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:50.247510    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hhvtx"
Sep 17 16:34:50.247664 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:50.247508    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hhvtx" event={"ID":"76e07fe9-066f-4da0-aca2-bc2a7f64cc82","Type":"ContainerDied","Data":"c40454f446cf3934a346bea554e3fe848dbe931e825c4c7d14b80b7a71dda332"}
Sep 17 16:34:50.247664 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:50.247570    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hhvtx" event={"ID":"76e07fe9-066f-4da0-aca2-bc2a7f64cc82","Type":"ContainerDied","Data":"c0647f3cee504079a0df5b23906969967fcb07f412966fb96f0d23f7f2b7d768"}
Sep 17 16:34:50.247664 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:50.247595    2585 scope.go:117] "RemoveContainer" containerID="c227097908a637446f035dce21aa81c10984d5fbdc85c7eab94404547b5a6619"
Sep 17 16:34:50.256081 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:50.256064    2585 scope.go:117] "RemoveContainer" containerID="c40454f446cf3934a346bea554e3fe848dbe931e825c4c7d14b80b7a71dda332"
Sep 17 16:34:50.263794 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:50.263771    2585 scope.go:117] "RemoveContainer" containerID="7fbc4506405e4077a4f29f6c0475bcc5c0aecba2700f21729889f525f53e869f"
Sep 17 16:34:50.271907 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:50.271877    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hhvtx"]
Sep 17 16:34:50.273147 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:50.273125    2585 scope.go:117] "RemoveContainer" containerID="c227097908a637446f035dce21aa81c10984d5fbdc85c7eab94404547b5a6619"
Sep 17 16:34:50.273534 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:34:50.273516    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c227097908a637446f035dce21aa81c10984d5fbdc85c7eab94404547b5a6619\": container with ID starting with c227097908a637446f035dce21aa81c10984d5fbdc85c7eab94404547b5a6619 not found: ID does not exist" containerID="c227097908a637446f035dce21aa81c10984d5fbdc85c7eab94404547b5a6619"
Sep 17 16:34:50.273636 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:50.273547    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c227097908a637446f035dce21aa81c10984d5fbdc85c7eab94404547b5a6619"} err="failed to get container status \"c227097908a637446f035dce21aa81c10984d5fbdc85c7eab94404547b5a6619\": rpc error: code = NotFound desc = could not find container \"c227097908a637446f035dce21aa81c10984d5fbdc85c7eab94404547b5a6619\": container with ID starting with c227097908a637446f035dce21aa81c10984d5fbdc85c7eab94404547b5a6619 not found: ID does not exist"
Sep 17 16:34:50.273636 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:50.273571    2585 scope.go:117] "RemoveContainer" containerID="c40454f446cf3934a346bea554e3fe848dbe931e825c4c7d14b80b7a71dda332"
Sep 17 16:34:50.273852 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:34:50.273836    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c40454f446cf3934a346bea554e3fe848dbe931e825c4c7d14b80b7a71dda332\": container with ID starting with c40454f446cf3934a346bea554e3fe848dbe931e825c4c7d14b80b7a71dda332 not found: ID does not exist" containerID="c40454f446cf3934a346bea554e3fe848dbe931e825c4c7d14b80b7a71dda332"
Sep 17 16:34:50.273920 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:50.273859    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c40454f446cf3934a346bea554e3fe848dbe931e825c4c7d14b80b7a71dda332"} err="failed to get container status \"c40454f446cf3934a346bea554e3fe848dbe931e825c4c7d14b80b7a71dda332\": rpc error: code = NotFound desc = could not find container \"c40454f446cf3934a346bea554e3fe848dbe931e825c4c7d14b80b7a71dda332\": container with ID starting with c40454f446cf3934a346bea554e3fe848dbe931e825c4c7d14b80b7a71dda332 not found: ID does not exist"
Sep 17 16:34:50.273920 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:50.273878    2585 scope.go:117] "RemoveContainer" containerID="7fbc4506405e4077a4f29f6c0475bcc5c0aecba2700f21729889f525f53e869f"
Sep 17 16:34:50.274007 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:50.273986    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-hhvtx"]
Sep 17 16:34:50.274142 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:34:50.274123    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fbc4506405e4077a4f29f6c0475bcc5c0aecba2700f21729889f525f53e869f\": container with ID starting with 7fbc4506405e4077a4f29f6c0475bcc5c0aecba2700f21729889f525f53e869f not found: ID does not exist" containerID="7fbc4506405e4077a4f29f6c0475bcc5c0aecba2700f21729889f525f53e869f"
Sep 17 16:34:50.274188 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:50.274148    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fbc4506405e4077a4f29f6c0475bcc5c0aecba2700f21729889f525f53e869f"} err="failed to get container status \"7fbc4506405e4077a4f29f6c0475bcc5c0aecba2700f21729889f525f53e869f\": rpc error: code = NotFound desc = could not find container \"7fbc4506405e4077a4f29f6c0475bcc5c0aecba2700f21729889f525f53e869f\": container with ID starting with 7fbc4506405e4077a4f29f6c0475bcc5c0aecba2700f21729889f525f53e869f not found: ID does not exist"
Sep 17 16:34:50.512552 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:50.512462    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76e07fe9-066f-4da0-aca2-bc2a7f64cc82" path="/var/lib/kubelet/pods/76e07fe9-066f-4da0-aca2-bc2a7f64cc82/volumes"
Sep 17 16:34:51.838218 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:51.838176    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-5m945" podUID="e65398b0-e29f-48c6-86cb-3a33bcf8c3aa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.49:8643/healthz\": dial tcp 10.134.0.49:8643: connect: connection refused"
Sep 17 16:34:51.842531 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:51.842504    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-5m945" podUID="e65398b0-e29f-48c6-86cb-3a33bcf8c3aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 17 16:34:53.262943 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:53.262912    2585 generic.go:356] "Generic (PLEG): container finished" podID="7bc509df-381e-4e44-b59e-01e4b4cc1325" containerID="15506425acaf09bae724716a1e2ab34fd0b43148a25ea6c68cfedc323357eda1" exitCode=0
Sep 17 16:34:53.263353 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:53.262954    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d7cvs" event={"ID":"7bc509df-381e-4e44-b59e-01e4b4cc1325","Type":"ContainerDied","Data":"15506425acaf09bae724716a1e2ab34fd0b43148a25ea6c68cfedc323357eda1"}
Sep 17 16:34:53.954933 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:53.954905    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-5m945"
Sep 17 16:34:54.086528 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.086446    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e65398b0-e29f-48c6-86cb-3a33bcf8c3aa-cabundle-cert\") pod \"e65398b0-e29f-48c6-86cb-3a33bcf8c3aa\" (UID: \"e65398b0-e29f-48c6-86cb-3a33bcf8c3aa\") "
Sep 17 16:34:54.086528 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.086501    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e65398b0-e29f-48c6-86cb-3a33bcf8c3aa-kube-rbac-proxy-sar-config\") pod \"e65398b0-e29f-48c6-86cb-3a33bcf8c3aa\" (UID: \"e65398b0-e29f-48c6-86cb-3a33bcf8c3aa\") "
Sep 17 16:34:54.086758 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.086543    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w77km\" (UniqueName: \"kubernetes.io/projected/e65398b0-e29f-48c6-86cb-3a33bcf8c3aa-kube-api-access-w77km\") pod \"e65398b0-e29f-48c6-86cb-3a33bcf8c3aa\" (UID: \"e65398b0-e29f-48c6-86cb-3a33bcf8c3aa\") "
Sep 17 16:34:54.086758 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.086631    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e65398b0-e29f-48c6-86cb-3a33bcf8c3aa-kserve-provision-location\") pod \"e65398b0-e29f-48c6-86cb-3a33bcf8c3aa\" (UID: \"e65398b0-e29f-48c6-86cb-3a33bcf8c3aa\") "
Sep 17 16:34:54.086758 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.086656    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e65398b0-e29f-48c6-86cb-3a33bcf8c3aa-proxy-tls\") pod \"e65398b0-e29f-48c6-86cb-3a33bcf8c3aa\" (UID: \"e65398b0-e29f-48c6-86cb-3a33bcf8c3aa\") "
Sep 17 16:34:54.086995 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.086906    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e65398b0-e29f-48c6-86cb-3a33bcf8c3aa-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e65398b0-e29f-48c6-86cb-3a33bcf8c3aa" (UID: "e65398b0-e29f-48c6-86cb-3a33bcf8c3aa"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:34:54.087074 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.086865    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e65398b0-e29f-48c6-86cb-3a33bcf8c3aa-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e65398b0-e29f-48c6-86cb-3a33bcf8c3aa" (UID: "e65398b0-e29f-48c6-86cb-3a33bcf8c3aa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:34:54.087074 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.087025    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e65398b0-e29f-48c6-86cb-3a33bcf8c3aa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e65398b0-e29f-48c6-86cb-3a33bcf8c3aa" (UID: "e65398b0-e29f-48c6-86cb-3a33bcf8c3aa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:34:54.088834 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.088808    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e65398b0-e29f-48c6-86cb-3a33bcf8c3aa-kube-api-access-w77km" (OuterVolumeSpecName: "kube-api-access-w77km") pod "e65398b0-e29f-48c6-86cb-3a33bcf8c3aa" (UID: "e65398b0-e29f-48c6-86cb-3a33bcf8c3aa"). InnerVolumeSpecName "kube-api-access-w77km". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:34:54.088972 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.088950    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e65398b0-e29f-48c6-86cb-3a33bcf8c3aa-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e65398b0-e29f-48c6-86cb-3a33bcf8c3aa" (UID: "e65398b0-e29f-48c6-86cb-3a33bcf8c3aa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:34:54.182867 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.182831    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx"]
Sep 17 16:34:54.183180 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.183168    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76e07fe9-066f-4da0-aca2-bc2a7f64cc82" containerName="kube-rbac-proxy"
Sep 17 16:34:54.183219 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.183182    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="76e07fe9-066f-4da0-aca2-bc2a7f64cc82" containerName="kube-rbac-proxy"
Sep 17 16:34:54.183219 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.183192    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76e07fe9-066f-4da0-aca2-bc2a7f64cc82" containerName="storage-initializer"
Sep 17 16:34:54.183219 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.183198    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="76e07fe9-066f-4da0-aca2-bc2a7f64cc82" containerName="storage-initializer"
Sep 17 16:34:54.183219 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.183213    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e65398b0-e29f-48c6-86cb-3a33bcf8c3aa" containerName="kube-rbac-proxy"
Sep 17 16:34:54.183219 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.183218    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e65398b0-e29f-48c6-86cb-3a33bcf8c3aa" containerName="kube-rbac-proxy"
Sep 17 16:34:54.183364 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.183226    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e65398b0-e29f-48c6-86cb-3a33bcf8c3aa" containerName="storage-initializer"
Sep 17 16:34:54.183364 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.183231    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e65398b0-e29f-48c6-86cb-3a33bcf8c3aa" containerName="storage-initializer"
Sep 17 16:34:54.183364 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.183237    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76e07fe9-066f-4da0-aca2-bc2a7f64cc82" containerName="kserve-container"
Sep 17 16:34:54.183364 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.183243    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="76e07fe9-066f-4da0-aca2-bc2a7f64cc82" containerName="kserve-container"
Sep 17 16:34:54.183364 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.183252    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e65398b0-e29f-48c6-86cb-3a33bcf8c3aa" containerName="kserve-container"
Sep 17 16:34:54.183364 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.183257    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e65398b0-e29f-48c6-86cb-3a33bcf8c3aa" containerName="kserve-container"
Sep 17 16:34:54.183364 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.183321    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="76e07fe9-066f-4da0-aca2-bc2a7f64cc82" containerName="kube-rbac-proxy"
Sep 17 16:34:54.183364 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.183329    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="e65398b0-e29f-48c6-86cb-3a33bcf8c3aa" containerName="kube-rbac-proxy"
Sep 17 16:34:54.183364 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.183336    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="76e07fe9-066f-4da0-aca2-bc2a7f64cc82" containerName="kserve-container"
Sep 17 16:34:54.183364 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.183342    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="e65398b0-e29f-48c6-86cb-3a33bcf8c3aa" containerName="kserve-container"
Sep 17 16:34:54.186574 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.186553    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx"
Sep 17 16:34:54.187262 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.187245    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e65398b0-e29f-48c6-86cb-3a33bcf8c3aa-kserve-provision-location\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:34:54.187349 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.187265    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e65398b0-e29f-48c6-86cb-3a33bcf8c3aa-proxy-tls\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:34:54.187349 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.187275    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e65398b0-e29f-48c6-86cb-3a33bcf8c3aa-cabundle-cert\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:34:54.187349 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.187283    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e65398b0-e29f-48c6-86cb-3a33bcf8c3aa-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:34:54.187349 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.187291    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w77km\" (UniqueName: \"kubernetes.io/projected/e65398b0-e29f-48c6-86cb-3a33bcf8c3aa-kube-api-access-w77km\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:34:54.192263 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.192239    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Sep 17 16:34:54.192810 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.192790    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 17 16:34:54.203626 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.203596    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx"]
Sep 17 16:34:54.269505 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.269464    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d7cvs" event={"ID":"7bc509df-381e-4e44-b59e-01e4b4cc1325","Type":"ContainerStarted","Data":"9cd4e8c3299fea1948e4316d6171bd1b70a4011608778e9406738eec40c4b5b0"}
Sep 17 16:34:54.269935 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.269538    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d7cvs" event={"ID":"7bc509df-381e-4e44-b59e-01e4b4cc1325","Type":"ContainerStarted","Data":"d127c249504ae4f0e2b7e17ca403c36fda7529fe60ffe9287d4953376bac80bf"}
Sep 17 16:34:54.269935 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.269725    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d7cvs"
Sep 17 16:34:54.271546 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.271522    2585 generic.go:356] "Generic (PLEG): container finished" podID="e65398b0-e29f-48c6-86cb-3a33bcf8c3aa" containerID="2e2b304511339f48eb9751598c7bfc20f69faa73615df7c5a34562a2b1ec76c3" exitCode=0
Sep 17 16:34:54.271670 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.271564    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-5m945"
Sep 17 16:34:54.271670 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.271570    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-5m945" event={"ID":"e65398b0-e29f-48c6-86cb-3a33bcf8c3aa","Type":"ContainerDied","Data":"2e2b304511339f48eb9751598c7bfc20f69faa73615df7c5a34562a2b1ec76c3"}
Sep 17 16:34:54.271670 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.271610    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-5m945" event={"ID":"e65398b0-e29f-48c6-86cb-3a33bcf8c3aa","Type":"ContainerDied","Data":"adc3ae82d0e54d0b2bf3284ee4ff7ab95f6816b16bc97f68661e60334fa59544"}
Sep 17 16:34:54.271670 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.271635    2585 scope.go:117] "RemoveContainer" containerID="81e7e9e3f9058a60cb55bef4e0a1a1e23340aa1da64fd4415786087f5b96d246"
Sep 17 16:34:54.280598 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.280563    2585 scope.go:117] "RemoveContainer" containerID="2e2b304511339f48eb9751598c7bfc20f69faa73615df7c5a34562a2b1ec76c3"
Sep 17 16:34:54.287794 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.287772    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp6pn\" (UniqueName: \"kubernetes.io/projected/d08bf548-bf5b-41eb-b946-4d35e6c193f8-kube-api-access-kp6pn\") pod \"isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx\" (UID: \"d08bf548-bf5b-41eb-b946-4d35e6c193f8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx"
Sep 17 16:34:54.287912 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.287824    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d08bf548-bf5b-41eb-b946-4d35e6c193f8-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx\" (UID: \"d08bf548-bf5b-41eb-b946-4d35e6c193f8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx"
Sep 17 16:34:54.288048 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.287910    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d08bf548-bf5b-41eb-b946-4d35e6c193f8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx\" (UID: \"d08bf548-bf5b-41eb-b946-4d35e6c193f8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx"
Sep 17 16:34:54.288048 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.287946    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d08bf548-bf5b-41eb-b946-4d35e6c193f8-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx\" (UID: \"d08bf548-bf5b-41eb-b946-4d35e6c193f8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx"
Sep 17 16:34:54.288048 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.287977    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d08bf548-bf5b-41eb-b946-4d35e6c193f8-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx\" (UID: \"d08bf548-bf5b-41eb-b946-4d35e6c193f8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx"
Sep 17 16:34:54.288048 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.287981    2585 scope.go:117] "RemoveContainer" containerID="68e8522e8bff7cb2f23671ace1b2c5a7bd7463d7da1f28621f2e3240693bd4f5"
Sep 17 16:34:54.296647 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.296621    2585 scope.go:117] "RemoveContainer" containerID="81e7e9e3f9058a60cb55bef4e0a1a1e23340aa1da64fd4415786087f5b96d246"
Sep 17 16:34:54.296968 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:34:54.296947    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81e7e9e3f9058a60cb55bef4e0a1a1e23340aa1da64fd4415786087f5b96d246\": container with ID starting with 81e7e9e3f9058a60cb55bef4e0a1a1e23340aa1da64fd4415786087f5b96d246 not found: ID does not exist" containerID="81e7e9e3f9058a60cb55bef4e0a1a1e23340aa1da64fd4415786087f5b96d246"
Sep 17 16:34:54.297054 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.296980    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81e7e9e3f9058a60cb55bef4e0a1a1e23340aa1da64fd4415786087f5b96d246"} err="failed to get container status \"81e7e9e3f9058a60cb55bef4e0a1a1e23340aa1da64fd4415786087f5b96d246\": rpc error: code = NotFound desc = could not find container \"81e7e9e3f9058a60cb55bef4e0a1a1e23340aa1da64fd4415786087f5b96d246\": container with ID starting with 81e7e9e3f9058a60cb55bef4e0a1a1e23340aa1da64fd4415786087f5b96d246 not found: ID does not exist"
Sep 17 16:34:54.297054 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.297004    2585 scope.go:117] "RemoveContainer" containerID="2e2b304511339f48eb9751598c7bfc20f69faa73615df7c5a34562a2b1ec76c3"
Sep 17 16:34:54.297332 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:34:54.297316    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e2b304511339f48eb9751598c7bfc20f69faa73615df7c5a34562a2b1ec76c3\": container with ID starting with 2e2b304511339f48eb9751598c7bfc20f69faa73615df7c5a34562a2b1ec76c3 not found: ID does not exist" containerID="2e2b304511339f48eb9751598c7bfc20f69faa73615df7c5a34562a2b1ec76c3"
Sep 17 16:34:54.297393 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.297342    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e2b304511339f48eb9751598c7bfc20f69faa73615df7c5a34562a2b1ec76c3"} err="failed to get container status \"2e2b304511339f48eb9751598c7bfc20f69faa73615df7c5a34562a2b1ec76c3\": rpc error: code = NotFound desc = could not find container \"2e2b304511339f48eb9751598c7bfc20f69faa73615df7c5a34562a2b1ec76c3\": container with ID starting with 2e2b304511339f48eb9751598c7bfc20f69faa73615df7c5a34562a2b1ec76c3 not found: ID does not exist"
Sep 17 16:34:54.297393 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.297365    2585 scope.go:117] "RemoveContainer" containerID="68e8522e8bff7cb2f23671ace1b2c5a7bd7463d7da1f28621f2e3240693bd4f5"
Sep 17 16:34:54.297698 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:34:54.297675    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68e8522e8bff7cb2f23671ace1b2c5a7bd7463d7da1f28621f2e3240693bd4f5\": container with ID starting with 68e8522e8bff7cb2f23671ace1b2c5a7bd7463d7da1f28621f2e3240693bd4f5 not found: ID does not exist" containerID="68e8522e8bff7cb2f23671ace1b2c5a7bd7463d7da1f28621f2e3240693bd4f5"
Sep 17 16:34:54.297756 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.297709    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68e8522e8bff7cb2f23671ace1b2c5a7bd7463d7da1f28621f2e3240693bd4f5"} err="failed to get container status \"68e8522e8bff7cb2f23671ace1b2c5a7bd7463d7da1f28621f2e3240693bd4f5\": rpc error: code = NotFound desc = could not find container \"68e8522e8bff7cb2f23671ace1b2c5a7bd7463d7da1f28621f2e3240693bd4f5\": container with ID starting with 68e8522e8bff7cb2f23671ace1b2c5a7bd7463d7da1f28621f2e3240693bd4f5 not found: ID does not exist"
Sep 17 16:34:54.308900 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.308848    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d7cvs" podStartSLOduration=2.387128327 podStartE2EDuration="6.308833207s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.92170488s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:34:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:34:54.308478087 +0000 UTC m=+2450.078809885" watchObservedRunningTime="2026-09-17 16:34:54.308833207 +0000 UTC m=+2450.079165003"
Sep 17 16:34:54.325982 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.325950    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-swmqr"]
Sep 17 16:34:54.330864 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.330847    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-swmqr"
Sep 17 16:34:54.333501 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.333479    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 17 16:34:54.335085 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.335059    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-5m945"]
Sep 17 16:34:54.340567 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.340543    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-5m945"]
Sep 17 16:34:54.348485 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.348458    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-swmqr"]
Sep 17 16:34:54.388397 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.388360    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp6pn\" (UniqueName: \"kubernetes.io/projected/d08bf548-bf5b-41eb-b946-4d35e6c193f8-kube-api-access-kp6pn\") pod \"isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx\" (UID: \"d08bf548-bf5b-41eb-b946-4d35e6c193f8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx"
Sep 17 16:34:54.388610 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.388497    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d08bf548-bf5b-41eb-b946-4d35e6c193f8-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx\" (UID: \"d08bf548-bf5b-41eb-b946-4d35e6c193f8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx"
Sep 17 16:34:54.388610 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.388558    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d08bf548-bf5b-41eb-b946-4d35e6c193f8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx\" (UID: \"d08bf548-bf5b-41eb-b946-4d35e6c193f8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx"
Sep 17 16:34:54.388610 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.388584    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d08bf548-bf5b-41eb-b946-4d35e6c193f8-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx\" (UID: \"d08bf548-bf5b-41eb-b946-4d35e6c193f8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx"
Sep 17 16:34:54.388794 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.388610    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d08bf548-bf5b-41eb-b946-4d35e6c193f8-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx\" (UID: \"d08bf548-bf5b-41eb-b946-4d35e6c193f8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx"
Sep 17 16:34:54.388972 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:34:54.388948    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-serving-cert: secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Sep 17 16:34:54.389038 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:34:54.389022    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d08bf548-bf5b-41eb-b946-4d35e6c193f8-proxy-tls podName:d08bf548-bf5b-41eb-b946-4d35e6c193f8 nodeName:}" failed. No retries permitted until 2026-09-17 16:34:54.889001368 +0000 UTC m=+2450.659333156 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d08bf548-bf5b-41eb-b946-4d35e6c193f8-proxy-tls") pod "isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx" (UID: "d08bf548-bf5b-41eb-b946-4d35e6c193f8") : secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Sep 17 16:34:54.389091 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.389050    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d08bf548-bf5b-41eb-b946-4d35e6c193f8-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx\" (UID: \"d08bf548-bf5b-41eb-b946-4d35e6c193f8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx"
Sep 17 16:34:54.389258 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.389239    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d08bf548-bf5b-41eb-b946-4d35e6c193f8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx\" (UID: \"d08bf548-bf5b-41eb-b946-4d35e6c193f8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx"
Sep 17 16:34:54.389328 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.389307    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d08bf548-bf5b-41eb-b946-4d35e6c193f8-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx\" (UID: \"d08bf548-bf5b-41eb-b946-4d35e6c193f8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx"
Sep 17 16:34:54.398444 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.398406    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp6pn\" (UniqueName: \"kubernetes.io/projected/d08bf548-bf5b-41eb-b946-4d35e6c193f8-kube-api-access-kp6pn\") pod \"isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx\" (UID: \"d08bf548-bf5b-41eb-b946-4d35e6c193f8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx"
Sep 17 16:34:54.490207 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.490169    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2102846a-adb2-4077-96db-b535461139ba-kserve-provision-location\") pod \"isvc-canary-stop-predictor-7f68f9d4d-swmqr\" (UID: \"2102846a-adb2-4077-96db-b535461139ba\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-swmqr"
Sep 17 16:34:54.490380 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.490219    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2102846a-adb2-4077-96db-b535461139ba-proxy-tls\") pod \"isvc-canary-stop-predictor-7f68f9d4d-swmqr\" (UID: \"2102846a-adb2-4077-96db-b535461139ba\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-swmqr"
Sep 17 16:34:54.490380 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.490240    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2102846a-adb2-4077-96db-b535461139ba-cabundle-cert\") pod \"isvc-canary-stop-predictor-7f68f9d4d-swmqr\" (UID: \"2102846a-adb2-4077-96db-b535461139ba\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-swmqr"
Sep 17 16:34:54.490380 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.490301    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2102846a-adb2-4077-96db-b535461139ba-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-7f68f9d4d-swmqr\" (UID: \"2102846a-adb2-4077-96db-b535461139ba\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-swmqr"
Sep 17 16:34:54.490380 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.490341    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg8mv\" (UniqueName: \"kubernetes.io/projected/2102846a-adb2-4077-96db-b535461139ba-kube-api-access-mg8mv\") pod \"isvc-canary-stop-predictor-7f68f9d4d-swmqr\" (UID: \"2102846a-adb2-4077-96db-b535461139ba\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-swmqr"
Sep 17 16:34:54.513128 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.513095    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e65398b0-e29f-48c6-86cb-3a33bcf8c3aa" path="/var/lib/kubelet/pods/e65398b0-e29f-48c6-86cb-3a33bcf8c3aa/volumes"
Sep 17 16:34:54.591761 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.591676    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2102846a-adb2-4077-96db-b535461139ba-kserve-provision-location\") pod \"isvc-canary-stop-predictor-7f68f9d4d-swmqr\" (UID: \"2102846a-adb2-4077-96db-b535461139ba\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-swmqr"
Sep 17 16:34:54.591761 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.591721    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2102846a-adb2-4077-96db-b535461139ba-proxy-tls\") pod \"isvc-canary-stop-predictor-7f68f9d4d-swmqr\" (UID: \"2102846a-adb2-4077-96db-b535461139ba\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-swmqr"
Sep 17 16:34:54.591977 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.591837    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2102846a-adb2-4077-96db-b535461139ba-cabundle-cert\") pod \"isvc-canary-stop-predictor-7f68f9d4d-swmqr\" (UID: \"2102846a-adb2-4077-96db-b535461139ba\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-swmqr"
Sep 17 16:34:54.591977 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.591877    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2102846a-adb2-4077-96db-b535461139ba-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-7f68f9d4d-swmqr\" (UID: \"2102846a-adb2-4077-96db-b535461139ba\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-swmqr"
Sep 17 16:34:54.591977 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.591894    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg8mv\" (UniqueName: \"kubernetes.io/projected/2102846a-adb2-4077-96db-b535461139ba-kube-api-access-mg8mv\") pod \"isvc-canary-stop-predictor-7f68f9d4d-swmqr\" (UID: \"2102846a-adb2-4077-96db-b535461139ba\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-swmqr"
Sep 17 16:34:54.592128 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.592113    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2102846a-adb2-4077-96db-b535461139ba-kserve-provision-location\") pod \"isvc-canary-stop-predictor-7f68f9d4d-swmqr\" (UID: \"2102846a-adb2-4077-96db-b535461139ba\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-swmqr"
Sep 17 16:34:54.592484 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.592463    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2102846a-adb2-4077-96db-b535461139ba-cabundle-cert\") pod \"isvc-canary-stop-predictor-7f68f9d4d-swmqr\" (UID: \"2102846a-adb2-4077-96db-b535461139ba\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-swmqr"
Sep 17 16:34:54.592587 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.592512    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2102846a-adb2-4077-96db-b535461139ba-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-7f68f9d4d-swmqr\" (UID: \"2102846a-adb2-4077-96db-b535461139ba\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-swmqr"
Sep 17 16:34:54.594587 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.594560    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2102846a-adb2-4077-96db-b535461139ba-proxy-tls\") pod \"isvc-canary-stop-predictor-7f68f9d4d-swmqr\" (UID: \"2102846a-adb2-4077-96db-b535461139ba\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-swmqr"
Sep 17 16:34:54.606490 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.606466    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg8mv\" (UniqueName: \"kubernetes.io/projected/2102846a-adb2-4077-96db-b535461139ba-kube-api-access-mg8mv\") pod \"isvc-canary-stop-predictor-7f68f9d4d-swmqr\" (UID: \"2102846a-adb2-4077-96db-b535461139ba\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-swmqr"
Sep 17 16:34:54.641181 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.641146    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-swmqr"
Sep 17 16:34:54.894118 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.894045    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d08bf548-bf5b-41eb-b946-4d35e6c193f8-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx\" (UID: \"d08bf548-bf5b-41eb-b946-4d35e6c193f8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx"
Sep 17 16:34:54.896458 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.896414    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d08bf548-bf5b-41eb-b946-4d35e6c193f8-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx\" (UID: \"d08bf548-bf5b-41eb-b946-4d35e6c193f8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx"
Sep 17 16:34:54.927167 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:54.927135    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-swmqr"]
Sep 17 16:34:54.930685 ip-10-0-135-43 kubenswrapper[2585]: W0917 16:34:54.930651    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2102846a_adb2_4077_96db_b535461139ba.slice/crio-0c4d3f1ae71c5aef125470a3cae6eb8b3957566226dbb23d7a1e7dc67a392270 WatchSource:0}: Error finding container 0c4d3f1ae71c5aef125470a3cae6eb8b3957566226dbb23d7a1e7dc67a392270: Status 404 returned error can't find the container with id 0c4d3f1ae71c5aef125470a3cae6eb8b3957566226dbb23d7a1e7dc67a392270
Sep 17 16:34:55.097660 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:55.097618    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx"
Sep 17 16:34:55.277964 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:55.277925    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-swmqr" event={"ID":"2102846a-adb2-4077-96db-b535461139ba","Type":"ContainerStarted","Data":"5f249375971795f630c763c8179b05e010efe805fd90d2410cb78863286d3bb7"}
Sep 17 16:34:55.278403 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:55.278000    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-swmqr" event={"ID":"2102846a-adb2-4077-96db-b535461139ba","Type":"ContainerStarted","Data":"0c4d3f1ae71c5aef125470a3cae6eb8b3957566226dbb23d7a1e7dc67a392270"}
Sep 17 16:34:55.279162 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:55.279136    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d7cvs"
Sep 17 16:34:55.383860 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:55.383828    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx"]
Sep 17 16:34:55.386689 ip-10-0-135-43 kubenswrapper[2585]: W0917 16:34:55.386656    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd08bf548_bf5b_41eb_b946_4d35e6c193f8.slice/crio-86c405b5e8023afc7dcd007624f29a048300c6d39c2c5d49434ded3485a8b875 WatchSource:0}: Error finding container 86c405b5e8023afc7dcd007624f29a048300c6d39c2c5d49434ded3485a8b875: Status 404 returned error can't find the container with id 86c405b5e8023afc7dcd007624f29a048300c6d39c2c5d49434ded3485a8b875
Sep 17 16:34:56.284518 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:56.284479    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx" event={"ID":"d08bf548-bf5b-41eb-b946-4d35e6c193f8","Type":"ContainerStarted","Data":"a971510c816bef2f6c5a929eba7f052ab4cf6e5dae4889d1634211af04be8f30"}
Sep 17 16:34:56.284518 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:56.284539    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx" event={"ID":"d08bf548-bf5b-41eb-b946-4d35e6c193f8","Type":"ContainerStarted","Data":"86c405b5e8023afc7dcd007624f29a048300c6d39c2c5d49434ded3485a8b875"}
Sep 17 16:34:59.299086 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:59.299050    2585 generic.go:356] "Generic (PLEG): container finished" podID="2102846a-adb2-4077-96db-b535461139ba" containerID="5f249375971795f630c763c8179b05e010efe805fd90d2410cb78863286d3bb7" exitCode=0
Sep 17 16:34:59.299541 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:34:59.299122    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-swmqr" event={"ID":"2102846a-adb2-4077-96db-b535461139ba","Type":"ContainerDied","Data":"5f249375971795f630c763c8179b05e010efe805fd90d2410cb78863286d3bb7"}
Sep 17 16:35:00.306364 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:00.306323    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-swmqr" event={"ID":"2102846a-adb2-4077-96db-b535461139ba","Type":"ContainerStarted","Data":"638d79e1545b5bafda1a98c839e78f20af7364eaec47b944e6c3fdf597739a50"}
Sep 17 16:35:00.306844 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:00.306393    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-swmqr" event={"ID":"2102846a-adb2-4077-96db-b535461139ba","Type":"ContainerStarted","Data":"f3f836503b2c4f60b2db560725fc05dc6ac576f51028539dfc8bb03f432aef32"}
Sep 17 16:35:00.306844 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:00.306603    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-swmqr"
Sep 17 16:35:00.307981 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:00.307962    2585 generic.go:356] "Generic (PLEG): container finished" podID="d08bf548-bf5b-41eb-b946-4d35e6c193f8" containerID="a971510c816bef2f6c5a929eba7f052ab4cf6e5dae4889d1634211af04be8f30" exitCode=0
Sep 17 16:35:00.308034 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:00.308019    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx" event={"ID":"d08bf548-bf5b-41eb-b946-4d35e6c193f8","Type":"ContainerDied","Data":"a971510c816bef2f6c5a929eba7f052ab4cf6e5dae4889d1634211af04be8f30"}
Sep 17 16:35:00.330333 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:00.330281    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-swmqr" podStartSLOduration=2.216136865 podStartE2EDuration="6.330260255s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.11412339s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:34:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:35:00.32925844 +0000 UTC m=+2456.099590236" watchObservedRunningTime="2026-09-17 16:35:00.330260255 +0000 UTC m=+2456.100592053"
Sep 17 16:35:01.289668 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:01.289636    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d7cvs"
Sep 17 16:35:01.317744 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:01.317712    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx" event={"ID":"d08bf548-bf5b-41eb-b946-4d35e6c193f8","Type":"ContainerStarted","Data":"8bf391e022447c07b3370ef3e0f7e166a6bd6e9b6c2d663e248f62141389c543"}
Sep 17 16:35:01.318240 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:01.317781    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx" event={"ID":"d08bf548-bf5b-41eb-b946-4d35e6c193f8","Type":"ContainerStarted","Data":"cd1854e374daabfc9561a1e63942b1cad0f106e6d692f7b945f6ddd2b44dfd7e"}
Sep 17 16:35:01.318240 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:01.317990    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-swmqr"
Sep 17 16:35:01.318401 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:01.318379    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx"
Sep 17 16:35:01.318576 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:01.318558    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx"
Sep 17 16:35:01.319143 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:01.319100    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-swmqr" podUID="2102846a-adb2-4077-96db-b535461139ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 16:35:01.319700 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:01.319674    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx" podUID="d08bf548-bf5b-41eb-b946-4d35e6c193f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 17 16:35:01.337142 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:01.337093    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx" podStartSLOduration=3.397627999 podStartE2EDuration="7.337074537s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.939446538s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:34:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:35:01.336023551 +0000 UTC m=+2457.106355358" watchObservedRunningTime="2026-09-17 16:35:01.337074537 +0000 UTC m=+2457.107406336"
Sep 17 16:35:02.323439 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:02.323384    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-swmqr" podUID="2102846a-adb2-4077-96db-b535461139ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 16:35:02.323839 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:02.323439    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx" podUID="d08bf548-bf5b-41eb-b946-4d35e6c193f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 17 16:35:03.328043 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:03.328000    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx" podUID="d08bf548-bf5b-41eb-b946-4d35e6c193f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 17 16:35:07.327850 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:07.327822    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-swmqr"
Sep 17 16:35:07.328455 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:07.328407    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-swmqr" podUID="2102846a-adb2-4077-96db-b535461139ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 16:35:08.332731 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:08.332703    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx"
Sep 17 16:35:08.333138 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:08.333119    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx" podUID="d08bf548-bf5b-41eb-b946-4d35e6c193f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 17 16:35:17.329227 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:17.329187    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-swmqr" podUID="2102846a-adb2-4077-96db-b535461139ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 16:35:18.333383 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:18.333340    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx" podUID="d08bf548-bf5b-41eb-b946-4d35e6c193f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 17 16:35:27.328675 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:27.328635    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-swmqr" podUID="2102846a-adb2-4077-96db-b535461139ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 16:35:28.333864 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:28.333823    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx" podUID="d08bf548-bf5b-41eb-b946-4d35e6c193f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 17 16:35:31.314582 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:31.314546    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d7cvs"
Sep 17 16:35:37.329367 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:37.329323    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-swmqr" podUID="2102846a-adb2-4077-96db-b535461139ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 16:35:38.123700 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:38.123659    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d7cvs"]
Sep 17 16:35:38.123927 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:38.123900    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d7cvs" podUID="7bc509df-381e-4e44-b59e-01e4b4cc1325" containerName="kserve-container" containerID="cri-o://d127c249504ae4f0e2b7e17ca403c36fda7529fe60ffe9287d4953376bac80bf" gracePeriod=30
Sep 17 16:35:38.123998 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:38.123961    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d7cvs" podUID="7bc509df-381e-4e44-b59e-01e4b4cc1325" containerName="kube-rbac-proxy" containerID="cri-o://9cd4e8c3299fea1948e4316d6171bd1b70a4011608778e9406738eec40c4b5b0" gracePeriod=30
Sep 17 16:35:38.333232 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:38.333193    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx" podUID="d08bf548-bf5b-41eb-b946-4d35e6c193f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 17 16:35:38.496046 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:38.496012    2585 generic.go:356] "Generic (PLEG): container finished" podID="7bc509df-381e-4e44-b59e-01e4b4cc1325" containerID="9cd4e8c3299fea1948e4316d6171bd1b70a4011608778e9406738eec40c4b5b0" exitCode=2
Sep 17 16:35:38.496223 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:38.496087    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d7cvs" event={"ID":"7bc509df-381e-4e44-b59e-01e4b4cc1325","Type":"ContainerDied","Data":"9cd4e8c3299fea1948e4316d6171bd1b70a4011608778e9406738eec40c4b5b0"}
Sep 17 16:35:41.285927 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:41.285881    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d7cvs" podUID="7bc509df-381e-4e44-b59e-01e4b4cc1325" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.51:8643/healthz\": dial tcp 10.134.0.51:8643: connect: connection refused"
Sep 17 16:35:41.290288 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:41.290259    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d7cvs" podUID="7bc509df-381e-4e44-b59e-01e4b4cc1325" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.51:8080/v2/models/xgboost-v2-mlserver/ready\": dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 17 16:35:45.076046 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:45.076020    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d7cvs"
Sep 17 16:35:45.240645 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:45.240611    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bc509df-381e-4e44-b59e-01e4b4cc1325-proxy-tls\") pod \"7bc509df-381e-4e44-b59e-01e4b4cc1325\" (UID: \"7bc509df-381e-4e44-b59e-01e4b4cc1325\") "
Sep 17 16:35:45.240645 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:45.240648    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7bc509df-381e-4e44-b59e-01e4b4cc1325-cabundle-cert\") pod \"7bc509df-381e-4e44-b59e-01e4b4cc1325\" (UID: \"7bc509df-381e-4e44-b59e-01e4b4cc1325\") "
Sep 17 16:35:45.240886 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:45.240683    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bc509df-381e-4e44-b59e-01e4b4cc1325-kserve-provision-location\") pod \"7bc509df-381e-4e44-b59e-01e4b4cc1325\" (UID: \"7bc509df-381e-4e44-b59e-01e4b4cc1325\") "
Sep 17 16:35:45.240886 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:45.240700    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bc509df-381e-4e44-b59e-01e4b4cc1325-kube-rbac-proxy-sar-config\") pod \"7bc509df-381e-4e44-b59e-01e4b4cc1325\" (UID: \"7bc509df-381e-4e44-b59e-01e4b4cc1325\") "
Sep 17 16:35:45.240886 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:45.240835    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4w49l\" (UniqueName: \"kubernetes.io/projected/7bc509df-381e-4e44-b59e-01e4b4cc1325-kube-api-access-4w49l\") pod \"7bc509df-381e-4e44-b59e-01e4b4cc1325\" (UID: \"7bc509df-381e-4e44-b59e-01e4b4cc1325\") "
Sep 17 16:35:45.241038 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:45.241005    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bc509df-381e-4e44-b59e-01e4b4cc1325-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7bc509df-381e-4e44-b59e-01e4b4cc1325" (UID: "7bc509df-381e-4e44-b59e-01e4b4cc1325"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:35:45.241099 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:45.241030    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bc509df-381e-4e44-b59e-01e4b4cc1325-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7bc509df-381e-4e44-b59e-01e4b4cc1325" (UID: "7bc509df-381e-4e44-b59e-01e4b4cc1325"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:35:45.241099 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:45.241046    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bc509df-381e-4e44-b59e-01e4b4cc1325-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7bc509df-381e-4e44-b59e-01e4b4cc1325" (UID: "7bc509df-381e-4e44-b59e-01e4b4cc1325"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:35:45.241218 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:45.241204    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7bc509df-381e-4e44-b59e-01e4b4cc1325-cabundle-cert\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:35:45.241270 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:45.241221    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bc509df-381e-4e44-b59e-01e4b4cc1325-kserve-provision-location\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:35:45.241270 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:45.241230    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bc509df-381e-4e44-b59e-01e4b4cc1325-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:35:45.242871 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:45.242846    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bc509df-381e-4e44-b59e-01e4b4cc1325-kube-api-access-4w49l" (OuterVolumeSpecName: "kube-api-access-4w49l") pod "7bc509df-381e-4e44-b59e-01e4b4cc1325" (UID: "7bc509df-381e-4e44-b59e-01e4b4cc1325"). InnerVolumeSpecName "kube-api-access-4w49l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:35:45.242987 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:45.242965    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bc509df-381e-4e44-b59e-01e4b4cc1325-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7bc509df-381e-4e44-b59e-01e4b4cc1325" (UID: "7bc509df-381e-4e44-b59e-01e4b4cc1325"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:35:45.342145 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:45.342107    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bc509df-381e-4e44-b59e-01e4b4cc1325-proxy-tls\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:35:45.342145 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:45.342137    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w49l\" (UniqueName: \"kubernetes.io/projected/7bc509df-381e-4e44-b59e-01e4b4cc1325-kube-api-access-4w49l\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:35:45.531735 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:45.531645    2585 generic.go:356] "Generic (PLEG): container finished" podID="7bc509df-381e-4e44-b59e-01e4b4cc1325" containerID="d127c249504ae4f0e2b7e17ca403c36fda7529fe60ffe9287d4953376bac80bf" exitCode=0
Sep 17 16:35:45.531735 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:45.531707    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d7cvs"
Sep 17 16:35:45.531925 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:45.531726    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d7cvs" event={"ID":"7bc509df-381e-4e44-b59e-01e4b4cc1325","Type":"ContainerDied","Data":"d127c249504ae4f0e2b7e17ca403c36fda7529fe60ffe9287d4953376bac80bf"}
Sep 17 16:35:45.531925 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:45.531810    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d7cvs" event={"ID":"7bc509df-381e-4e44-b59e-01e4b4cc1325","Type":"ContainerDied","Data":"4fe9a2d1a1f7214d7705340d3e8ecc479e9fb223b48458a82e646b9dae463fd7"}
Sep 17 16:35:45.531925 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:45.531850    2585 scope.go:117] "RemoveContainer" containerID="9cd4e8c3299fea1948e4316d6171bd1b70a4011608778e9406738eec40c4b5b0"
Sep 17 16:35:45.541333 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:45.541312    2585 scope.go:117] "RemoveContainer" containerID="d127c249504ae4f0e2b7e17ca403c36fda7529fe60ffe9287d4953376bac80bf"
Sep 17 16:35:45.549701 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:45.549680    2585 scope.go:117] "RemoveContainer" containerID="15506425acaf09bae724716a1e2ab34fd0b43148a25ea6c68cfedc323357eda1"
Sep 17 16:35:45.556047 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:45.556019    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d7cvs"]
Sep 17 16:35:45.559724 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:45.559693    2585 scope.go:117] "RemoveContainer" containerID="9cd4e8c3299fea1948e4316d6171bd1b70a4011608778e9406738eec40c4b5b0"
Sep 17 16:35:45.560106 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:35:45.560084    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cd4e8c3299fea1948e4316d6171bd1b70a4011608778e9406738eec40c4b5b0\": container with ID starting with 9cd4e8c3299fea1948e4316d6171bd1b70a4011608778e9406738eec40c4b5b0 not found: ID does not exist" containerID="9cd4e8c3299fea1948e4316d6171bd1b70a4011608778e9406738eec40c4b5b0"
Sep 17 16:35:45.560200 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:45.560118    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cd4e8c3299fea1948e4316d6171bd1b70a4011608778e9406738eec40c4b5b0"} err="failed to get container status \"9cd4e8c3299fea1948e4316d6171bd1b70a4011608778e9406738eec40c4b5b0\": rpc error: code = NotFound desc = could not find container \"9cd4e8c3299fea1948e4316d6171bd1b70a4011608778e9406738eec40c4b5b0\": container with ID starting with 9cd4e8c3299fea1948e4316d6171bd1b70a4011608778e9406738eec40c4b5b0 not found: ID does not exist"
Sep 17 16:35:45.560200 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:45.560147    2585 scope.go:117] "RemoveContainer" containerID="d127c249504ae4f0e2b7e17ca403c36fda7529fe60ffe9287d4953376bac80bf"
Sep 17 16:35:45.560293 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:45.560219    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-d7cvs"]
Sep 17 16:35:45.560496 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:35:45.560475    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d127c249504ae4f0e2b7e17ca403c36fda7529fe60ffe9287d4953376bac80bf\": container with ID starting with d127c249504ae4f0e2b7e17ca403c36fda7529fe60ffe9287d4953376bac80bf not found: ID does not exist" containerID="d127c249504ae4f0e2b7e17ca403c36fda7529fe60ffe9287d4953376bac80bf"
Sep 17 16:35:45.560576 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:45.560504    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d127c249504ae4f0e2b7e17ca403c36fda7529fe60ffe9287d4953376bac80bf"} err="failed to get container status \"d127c249504ae4f0e2b7e17ca403c36fda7529fe60ffe9287d4953376bac80bf\": rpc error: code = NotFound desc = could not find container \"d127c249504ae4f0e2b7e17ca403c36fda7529fe60ffe9287d4953376bac80bf\": container with ID starting with d127c249504ae4f0e2b7e17ca403c36fda7529fe60ffe9287d4953376bac80bf not found: ID does not exist"
Sep 17 16:35:45.560576 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:45.560527    2585 scope.go:117] "RemoveContainer" containerID="15506425acaf09bae724716a1e2ab34fd0b43148a25ea6c68cfedc323357eda1"
Sep 17 16:35:45.560824 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:35:45.560801    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15506425acaf09bae724716a1e2ab34fd0b43148a25ea6c68cfedc323357eda1\": container with ID starting with 15506425acaf09bae724716a1e2ab34fd0b43148a25ea6c68cfedc323357eda1 not found: ID does not exist" containerID="15506425acaf09bae724716a1e2ab34fd0b43148a25ea6c68cfedc323357eda1"
Sep 17 16:35:45.560924 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:45.560827    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15506425acaf09bae724716a1e2ab34fd0b43148a25ea6c68cfedc323357eda1"} err="failed to get container status \"15506425acaf09bae724716a1e2ab34fd0b43148a25ea6c68cfedc323357eda1\": rpc error: code = NotFound desc = could not find container \"15506425acaf09bae724716a1e2ab34fd0b43148a25ea6c68cfedc323357eda1\": container with ID starting with 15506425acaf09bae724716a1e2ab34fd0b43148a25ea6c68cfedc323357eda1 not found: ID does not exist"
Sep 17 16:35:46.513268 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:46.513231    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bc509df-381e-4e44-b59e-01e4b4cc1325" path="/var/lib/kubelet/pods/7bc509df-381e-4e44-b59e-01e4b4cc1325/volumes"
Sep 17 16:35:47.328752 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:47.328712    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-swmqr" podUID="2102846a-adb2-4077-96db-b535461139ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 16:35:48.333213 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:48.333176    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx" podUID="d08bf548-bf5b-41eb-b946-4d35e6c193f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 17 16:35:57.329181 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:57.329134    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-swmqr" podUID="2102846a-adb2-4077-96db-b535461139ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 16:35:58.334206 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:35:58.334164    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx" podUID="d08bf548-bf5b-41eb-b946-4d35e6c193f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 17 16:36:07.329129 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:07.329087    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-swmqr" podUID="2102846a-adb2-4077-96db-b535461139ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 16:36:08.333525 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:08.333482    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx" podUID="d08bf548-bf5b-41eb-b946-4d35e6c193f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 17 16:36:11.508578 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:11.508524    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-swmqr" podUID="2102846a-adb2-4077-96db-b535461139ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 16:36:18.334143 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:18.334111    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx"
Sep 17 16:36:18.471507 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:18.471459    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-swmqr"]
Sep 17 16:36:18.471761 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:18.471732    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-swmqr" podUID="2102846a-adb2-4077-96db-b535461139ba" containerName="kserve-container" containerID="cri-o://f3f836503b2c4f60b2db560725fc05dc6ac576f51028539dfc8bb03f432aef32" gracePeriod=30
Sep 17 16:36:18.471927 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:18.471771    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-swmqr" podUID="2102846a-adb2-4077-96db-b535461139ba" containerName="kube-rbac-proxy" containerID="cri-o://638d79e1545b5bafda1a98c839e78f20af7364eaec47b944e6c3fdf597739a50" gracePeriod=30
Sep 17 16:36:18.472021 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:18.471951    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-swmqr" podUID="2102846a-adb2-4077-96db-b535461139ba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 17 16:36:18.691692 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:18.691659    2585 generic.go:356] "Generic (PLEG): container finished" podID="2102846a-adb2-4077-96db-b535461139ba" containerID="638d79e1545b5bafda1a98c839e78f20af7364eaec47b944e6c3fdf597739a50" exitCode=2
Sep 17 16:36:18.691867 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:18.691735    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-swmqr" event={"ID":"2102846a-adb2-4077-96db-b535461139ba","Type":"ContainerDied","Data":"638d79e1545b5bafda1a98c839e78f20af7364eaec47b944e6c3fdf597739a50"}
Sep 17 16:36:22.324262 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:22.324217    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-swmqr" podUID="2102846a-adb2-4077-96db-b535461139ba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.53:8643/healthz\": dial tcp 10.134.0.53:8643: connect: connection refused"
Sep 17 16:36:24.126473 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:24.126444    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-swmqr"
Sep 17 16:36:24.140128 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:24.140091    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx"]
Sep 17 16:36:24.140345 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:24.140323    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx" podUID="d08bf548-bf5b-41eb-b946-4d35e6c193f8" containerName="kserve-container" containerID="cri-o://cd1854e374daabfc9561a1e63942b1cad0f106e6d692f7b945f6ddd2b44dfd7e" gracePeriod=30
Sep 17 16:36:24.140461 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:24.140415    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx" podUID="d08bf548-bf5b-41eb-b946-4d35e6c193f8" containerName="kube-rbac-proxy" containerID="cri-o://8bf391e022447c07b3370ef3e0f7e166a6bd6e9b6c2d663e248f62141389c543" gracePeriod=30
Sep 17 16:36:24.264657 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:24.264630    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2102846a-adb2-4077-96db-b535461139ba-kserve-provision-location\") pod \"2102846a-adb2-4077-96db-b535461139ba\" (UID: \"2102846a-adb2-4077-96db-b535461139ba\") "
Sep 17 16:36:24.264776 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:24.264685    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2102846a-adb2-4077-96db-b535461139ba-cabundle-cert\") pod \"2102846a-adb2-4077-96db-b535461139ba\" (UID: \"2102846a-adb2-4077-96db-b535461139ba\") "
Sep 17 16:36:24.264776 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:24.264718    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2102846a-adb2-4077-96db-b535461139ba-kube-rbac-proxy-sar-config\") pod \"2102846a-adb2-4077-96db-b535461139ba\" (UID: \"2102846a-adb2-4077-96db-b535461139ba\") "
Sep 17 16:36:24.264776 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:24.264752    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2102846a-adb2-4077-96db-b535461139ba-proxy-tls\") pod \"2102846a-adb2-4077-96db-b535461139ba\" (UID: \"2102846a-adb2-4077-96db-b535461139ba\") "
Sep 17 16:36:24.264776 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:24.264771    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mg8mv\" (UniqueName: \"kubernetes.io/projected/2102846a-adb2-4077-96db-b535461139ba-kube-api-access-mg8mv\") pod \"2102846a-adb2-4077-96db-b535461139ba\" (UID: \"2102846a-adb2-4077-96db-b535461139ba\") "
Sep 17 16:36:24.264998 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:24.264972    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2102846a-adb2-4077-96db-b535461139ba-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2102846a-adb2-4077-96db-b535461139ba" (UID: "2102846a-adb2-4077-96db-b535461139ba"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:36:24.265110 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:24.265088    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2102846a-adb2-4077-96db-b535461139ba-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2102846a-adb2-4077-96db-b535461139ba" (UID: "2102846a-adb2-4077-96db-b535461139ba"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:36:24.265180 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:24.265105    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2102846a-adb2-4077-96db-b535461139ba-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2102846a-adb2-4077-96db-b535461139ba" (UID: "2102846a-adb2-4077-96db-b535461139ba"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:36:24.266827 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:24.266801    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2102846a-adb2-4077-96db-b535461139ba-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2102846a-adb2-4077-96db-b535461139ba" (UID: "2102846a-adb2-4077-96db-b535461139ba"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:36:24.266948 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:24.266925    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2102846a-adb2-4077-96db-b535461139ba-kube-api-access-mg8mv" (OuterVolumeSpecName: "kube-api-access-mg8mv") pod "2102846a-adb2-4077-96db-b535461139ba" (UID: "2102846a-adb2-4077-96db-b535461139ba"). InnerVolumeSpecName "kube-api-access-mg8mv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:36:24.365720 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:24.365624    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2102846a-adb2-4077-96db-b535461139ba-cabundle-cert\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:36:24.365720 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:24.365658    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2102846a-adb2-4077-96db-b535461139ba-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:36:24.365720 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:24.365667    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2102846a-adb2-4077-96db-b535461139ba-proxy-tls\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:36:24.365720 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:24.365677    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg8mv\" (UniqueName: \"kubernetes.io/projected/2102846a-adb2-4077-96db-b535461139ba-kube-api-access-mg8mv\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:36:24.365720 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:24.365685    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2102846a-adb2-4077-96db-b535461139ba-kserve-provision-location\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:36:24.721925 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:24.721894    2585 generic.go:356] "Generic (PLEG): container finished" podID="d08bf548-bf5b-41eb-b946-4d35e6c193f8" containerID="8bf391e022447c07b3370ef3e0f7e166a6bd6e9b6c2d663e248f62141389c543" exitCode=2
Sep 17 16:36:24.722111 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:24.721967    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx" event={"ID":"d08bf548-bf5b-41eb-b946-4d35e6c193f8","Type":"ContainerDied","Data":"8bf391e022447c07b3370ef3e0f7e166a6bd6e9b6c2d663e248f62141389c543"}
Sep 17 16:36:24.723873 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:24.723851    2585 generic.go:356] "Generic (PLEG): container finished" podID="2102846a-adb2-4077-96db-b535461139ba" containerID="f3f836503b2c4f60b2db560725fc05dc6ac576f51028539dfc8bb03f432aef32" exitCode=139
Sep 17 16:36:24.723990 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:24.723899    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-swmqr" event={"ID":"2102846a-adb2-4077-96db-b535461139ba","Type":"ContainerDied","Data":"f3f836503b2c4f60b2db560725fc05dc6ac576f51028539dfc8bb03f432aef32"}
Sep 17 16:36:24.723990 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:24.723913    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-swmqr"
Sep 17 16:36:24.723990 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:24.723950    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-swmqr" event={"ID":"2102846a-adb2-4077-96db-b535461139ba","Type":"ContainerDied","Data":"0c4d3f1ae71c5aef125470a3cae6eb8b3957566226dbb23d7a1e7dc67a392270"}
Sep 17 16:36:24.723990 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:24.723977    2585 scope.go:117] "RemoveContainer" containerID="638d79e1545b5bafda1a98c839e78f20af7364eaec47b944e6c3fdf597739a50"
Sep 17 16:36:24.732653 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:24.732636    2585 scope.go:117] "RemoveContainer" containerID="f3f836503b2c4f60b2db560725fc05dc6ac576f51028539dfc8bb03f432aef32"
Sep 17 16:36:24.740252 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:24.740233    2585 scope.go:117] "RemoveContainer" containerID="5f249375971795f630c763c8179b05e010efe805fd90d2410cb78863286d3bb7"
Sep 17 16:36:24.745131 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:24.745105    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-swmqr"]
Sep 17 16:36:24.749231 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:24.749202    2585 scope.go:117] "RemoveContainer" containerID="638d79e1545b5bafda1a98c839e78f20af7364eaec47b944e6c3fdf597739a50"
Sep 17 16:36:24.749627 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:36:24.749599    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"638d79e1545b5bafda1a98c839e78f20af7364eaec47b944e6c3fdf597739a50\": container with ID starting with 638d79e1545b5bafda1a98c839e78f20af7364eaec47b944e6c3fdf597739a50 not found: ID does not exist" containerID="638d79e1545b5bafda1a98c839e78f20af7364eaec47b944e6c3fdf597739a50"
Sep 17 16:36:24.749734 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:24.749636    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"638d79e1545b5bafda1a98c839e78f20af7364eaec47b944e6c3fdf597739a50"} err="failed to get container status \"638d79e1545b5bafda1a98c839e78f20af7364eaec47b944e6c3fdf597739a50\": rpc error: code = NotFound desc = could not find container \"638d79e1545b5bafda1a98c839e78f20af7364eaec47b944e6c3fdf597739a50\": container with ID starting with 638d79e1545b5bafda1a98c839e78f20af7364eaec47b944e6c3fdf597739a50 not found: ID does not exist"
Sep 17 16:36:24.749734 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:24.749658    2585 scope.go:117] "RemoveContainer" containerID="f3f836503b2c4f60b2db560725fc05dc6ac576f51028539dfc8bb03f432aef32"
Sep 17 16:36:24.749956 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:36:24.749937    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3f836503b2c4f60b2db560725fc05dc6ac576f51028539dfc8bb03f432aef32\": container with ID starting with f3f836503b2c4f60b2db560725fc05dc6ac576f51028539dfc8bb03f432aef32 not found: ID does not exist" containerID="f3f836503b2c4f60b2db560725fc05dc6ac576f51028539dfc8bb03f432aef32"
Sep 17 16:36:24.750016 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:24.749962    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3f836503b2c4f60b2db560725fc05dc6ac576f51028539dfc8bb03f432aef32"} err="failed to get container status \"f3f836503b2c4f60b2db560725fc05dc6ac576f51028539dfc8bb03f432aef32\": rpc error: code = NotFound desc = could not find container \"f3f836503b2c4f60b2db560725fc05dc6ac576f51028539dfc8bb03f432aef32\": container with ID starting with f3f836503b2c4f60b2db560725fc05dc6ac576f51028539dfc8bb03f432aef32 not found: ID does not exist"
Sep 17 16:36:24.750016 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:24.749982    2585 scope.go:117] "RemoveContainer" containerID="5f249375971795f630c763c8179b05e010efe805fd90d2410cb78863286d3bb7"
Sep 17 16:36:24.750208 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:36:24.750184    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f249375971795f630c763c8179b05e010efe805fd90d2410cb78863286d3bb7\": container with ID starting with 5f249375971795f630c763c8179b05e010efe805fd90d2410cb78863286d3bb7 not found: ID does not exist" containerID="5f249375971795f630c763c8179b05e010efe805fd90d2410cb78863286d3bb7"
Sep 17 16:36:24.750271 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:24.750213    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f249375971795f630c763c8179b05e010efe805fd90d2410cb78863286d3bb7"} err="failed to get container status \"5f249375971795f630c763c8179b05e010efe805fd90d2410cb78863286d3bb7\": rpc error: code = NotFound desc = could not find container \"5f249375971795f630c763c8179b05e010efe805fd90d2410cb78863286d3bb7\": container with ID starting with 5f249375971795f630c763c8179b05e010efe805fd90d2410cb78863286d3bb7 not found: ID does not exist"
Sep 17 16:36:24.750438 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:24.750400    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-swmqr"]
Sep 17 16:36:26.513033 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:26.512999    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2102846a-adb2-4077-96db-b535461139ba" path="/var/lib/kubelet/pods/2102846a-adb2-4077-96db-b535461139ba/volumes"
Sep 17 16:36:28.329280 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:28.329237    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx" podUID="d08bf548-bf5b-41eb-b946-4d35e6c193f8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.52:8643/healthz\": dial tcp 10.134.0.52:8643: connect: connection refused"
Sep 17 16:36:28.333645 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:28.333608    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx" podUID="d08bf548-bf5b-41eb-b946-4d35e6c193f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 17 16:36:29.123111 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:29.123084    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx"
Sep 17 16:36:29.207706 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:29.207670    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d08bf548-bf5b-41eb-b946-4d35e6c193f8-kube-rbac-proxy-sar-config\") pod \"d08bf548-bf5b-41eb-b946-4d35e6c193f8\" (UID: \"d08bf548-bf5b-41eb-b946-4d35e6c193f8\") "
Sep 17 16:36:29.207904 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:29.207718    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d08bf548-bf5b-41eb-b946-4d35e6c193f8-proxy-tls\") pod \"d08bf548-bf5b-41eb-b946-4d35e6c193f8\" (UID: \"d08bf548-bf5b-41eb-b946-4d35e6c193f8\") "
Sep 17 16:36:29.207904 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:29.207752    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d08bf548-bf5b-41eb-b946-4d35e6c193f8-cabundle-cert\") pod \"d08bf548-bf5b-41eb-b946-4d35e6c193f8\" (UID: \"d08bf548-bf5b-41eb-b946-4d35e6c193f8\") "
Sep 17 16:36:29.207904 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:29.207868    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d08bf548-bf5b-41eb-b946-4d35e6c193f8-kserve-provision-location\") pod \"d08bf548-bf5b-41eb-b946-4d35e6c193f8\" (UID: \"d08bf548-bf5b-41eb-b946-4d35e6c193f8\") "
Sep 17 16:36:29.208092 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:29.207978    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kp6pn\" (UniqueName: \"kubernetes.io/projected/d08bf548-bf5b-41eb-b946-4d35e6c193f8-kube-api-access-kp6pn\") pod \"d08bf548-bf5b-41eb-b946-4d35e6c193f8\" (UID: \"d08bf548-bf5b-41eb-b946-4d35e6c193f8\") "
Sep 17 16:36:29.208149 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:29.208107    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d08bf548-bf5b-41eb-b946-4d35e6c193f8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d08bf548-bf5b-41eb-b946-4d35e6c193f8" (UID: "d08bf548-bf5b-41eb-b946-4d35e6c193f8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:36:29.208223 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:29.208193    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d08bf548-bf5b-41eb-b946-4d35e6c193f8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d08bf548-bf5b-41eb-b946-4d35e6c193f8" (UID: "d08bf548-bf5b-41eb-b946-4d35e6c193f8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:36:29.208280 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:29.208206    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d08bf548-bf5b-41eb-b946-4d35e6c193f8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d08bf548-bf5b-41eb-b946-4d35e6c193f8" (UID: "d08bf548-bf5b-41eb-b946-4d35e6c193f8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:36:29.208320 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:29.208282    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d08bf548-bf5b-41eb-b946-4d35e6c193f8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:36:29.209935 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:29.209910    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d08bf548-bf5b-41eb-b946-4d35e6c193f8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d08bf548-bf5b-41eb-b946-4d35e6c193f8" (UID: "d08bf548-bf5b-41eb-b946-4d35e6c193f8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:36:29.210036 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:29.210014    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d08bf548-bf5b-41eb-b946-4d35e6c193f8-kube-api-access-kp6pn" (OuterVolumeSpecName: "kube-api-access-kp6pn") pod "d08bf548-bf5b-41eb-b946-4d35e6c193f8" (UID: "d08bf548-bf5b-41eb-b946-4d35e6c193f8"). InnerVolumeSpecName "kube-api-access-kp6pn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:36:29.309303 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:29.309249    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d08bf548-bf5b-41eb-b946-4d35e6c193f8-kserve-provision-location\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:36:29.309303 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:29.309296    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kp6pn\" (UniqueName: \"kubernetes.io/projected/d08bf548-bf5b-41eb-b946-4d35e6c193f8-kube-api-access-kp6pn\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:36:29.309303 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:29.309309    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d08bf548-bf5b-41eb-b946-4d35e6c193f8-proxy-tls\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:36:29.309676 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:29.309324    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d08bf548-bf5b-41eb-b946-4d35e6c193f8-cabundle-cert\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:36:29.751524 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:29.751485    2585 generic.go:356] "Generic (PLEG): container finished" podID="d08bf548-bf5b-41eb-b946-4d35e6c193f8" containerID="cd1854e374daabfc9561a1e63942b1cad0f106e6d692f7b945f6ddd2b44dfd7e" exitCode=0
Sep 17 16:36:29.752021 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:29.751548    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx"
Sep 17 16:36:29.752021 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:29.751561    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx" event={"ID":"d08bf548-bf5b-41eb-b946-4d35e6c193f8","Type":"ContainerDied","Data":"cd1854e374daabfc9561a1e63942b1cad0f106e6d692f7b945f6ddd2b44dfd7e"}
Sep 17 16:36:29.752021 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:29.751627    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx" event={"ID":"d08bf548-bf5b-41eb-b946-4d35e6c193f8","Type":"ContainerDied","Data":"86c405b5e8023afc7dcd007624f29a048300c6d39c2c5d49434ded3485a8b875"}
Sep 17 16:36:29.752021 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:29.751653    2585 scope.go:117] "RemoveContainer" containerID="8bf391e022447c07b3370ef3e0f7e166a6bd6e9b6c2d663e248f62141389c543"
Sep 17 16:36:29.760132 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:29.760116    2585 scope.go:117] "RemoveContainer" containerID="cd1854e374daabfc9561a1e63942b1cad0f106e6d692f7b945f6ddd2b44dfd7e"
Sep 17 16:36:29.767753 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:29.767737    2585 scope.go:117] "RemoveContainer" containerID="a971510c816bef2f6c5a929eba7f052ab4cf6e5dae4889d1634211af04be8f30"
Sep 17 16:36:29.775800 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:29.775773    2585 scope.go:117] "RemoveContainer" containerID="8bf391e022447c07b3370ef3e0f7e166a6bd6e9b6c2d663e248f62141389c543"
Sep 17 16:36:29.775877 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:29.775802    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx"]
Sep 17 16:36:29.776146 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:36:29.776127    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bf391e022447c07b3370ef3e0f7e166a6bd6e9b6c2d663e248f62141389c543\": container with ID starting with 8bf391e022447c07b3370ef3e0f7e166a6bd6e9b6c2d663e248f62141389c543 not found: ID does not exist" containerID="8bf391e022447c07b3370ef3e0f7e166a6bd6e9b6c2d663e248f62141389c543"
Sep 17 16:36:29.776215 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:29.776154    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bf391e022447c07b3370ef3e0f7e166a6bd6e9b6c2d663e248f62141389c543"} err="failed to get container status \"8bf391e022447c07b3370ef3e0f7e166a6bd6e9b6c2d663e248f62141389c543\": rpc error: code = NotFound desc = could not find container \"8bf391e022447c07b3370ef3e0f7e166a6bd6e9b6c2d663e248f62141389c543\": container with ID starting with 8bf391e022447c07b3370ef3e0f7e166a6bd6e9b6c2d663e248f62141389c543 not found: ID does not exist"
Sep 17 16:36:29.776215 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:29.776172    2585 scope.go:117] "RemoveContainer" containerID="cd1854e374daabfc9561a1e63942b1cad0f106e6d692f7b945f6ddd2b44dfd7e"
Sep 17 16:36:29.776454 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:36:29.776414    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd1854e374daabfc9561a1e63942b1cad0f106e6d692f7b945f6ddd2b44dfd7e\": container with ID starting with cd1854e374daabfc9561a1e63942b1cad0f106e6d692f7b945f6ddd2b44dfd7e not found: ID does not exist" containerID="cd1854e374daabfc9561a1e63942b1cad0f106e6d692f7b945f6ddd2b44dfd7e"
Sep 17 16:36:29.776508 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:29.776458    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd1854e374daabfc9561a1e63942b1cad0f106e6d692f7b945f6ddd2b44dfd7e"} err="failed to get container status \"cd1854e374daabfc9561a1e63942b1cad0f106e6d692f7b945f6ddd2b44dfd7e\": rpc error: code = NotFound desc = could not find container \"cd1854e374daabfc9561a1e63942b1cad0f106e6d692f7b945f6ddd2b44dfd7e\": container with ID starting with cd1854e374daabfc9561a1e63942b1cad0f106e6d692f7b945f6ddd2b44dfd7e not found: ID does not exist"
Sep 17 16:36:29.776508 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:29.776471    2585 scope.go:117] "RemoveContainer" containerID="a971510c816bef2f6c5a929eba7f052ab4cf6e5dae4889d1634211af04be8f30"
Sep 17 16:36:29.776727 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:36:29.776711    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a971510c816bef2f6c5a929eba7f052ab4cf6e5dae4889d1634211af04be8f30\": container with ID starting with a971510c816bef2f6c5a929eba7f052ab4cf6e5dae4889d1634211af04be8f30 not found: ID does not exist" containerID="a971510c816bef2f6c5a929eba7f052ab4cf6e5dae4889d1634211af04be8f30"
Sep 17 16:36:29.776778 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:29.776729    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a971510c816bef2f6c5a929eba7f052ab4cf6e5dae4889d1634211af04be8f30"} err="failed to get container status \"a971510c816bef2f6c5a929eba7f052ab4cf6e5dae4889d1634211af04be8f30\": rpc error: code = NotFound desc = could not find container \"a971510c816bef2f6c5a929eba7f052ab4cf6e5dae4889d1634211af04be8f30\": container with ID starting with a971510c816bef2f6c5a929eba7f052ab4cf6e5dae4889d1634211af04be8f30 not found: ID does not exist"
Sep 17 16:36:29.780379 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:29.780352    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-tlvxx"]
Sep 17 16:36:30.514043 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:30.514009    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d08bf548-bf5b-41eb-b946-4d35e6c193f8" path="/var/lib/kubelet/pods/d08bf548-bf5b-41eb-b946-4d35e6c193f8/volumes"
Sep 17 16:36:34.595776 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:34.595738    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-zc48r"]
Sep 17 16:36:34.596244 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:34.596147    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d08bf548-bf5b-41eb-b946-4d35e6c193f8" containerName="kserve-container"
Sep 17 16:36:34.596244 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:34.596160    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="d08bf548-bf5b-41eb-b946-4d35e6c193f8" containerName="kserve-container"
Sep 17 16:36:34.596244 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:34.596170    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bc509df-381e-4e44-b59e-01e4b4cc1325" containerName="kube-rbac-proxy"
Sep 17 16:36:34.596244 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:34.596175    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bc509df-381e-4e44-b59e-01e4b4cc1325" containerName="kube-rbac-proxy"
Sep 17 16:36:34.596244 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:34.596183    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d08bf548-bf5b-41eb-b946-4d35e6c193f8" containerName="storage-initializer"
Sep 17 16:36:34.596244 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:34.596190    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="d08bf548-bf5b-41eb-b946-4d35e6c193f8" containerName="storage-initializer"
Sep 17 16:36:34.596244 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:34.596200    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2102846a-adb2-4077-96db-b535461139ba" containerName="kube-rbac-proxy"
Sep 17 16:36:34.596244 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:34.596206    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="2102846a-adb2-4077-96db-b535461139ba" containerName="kube-rbac-proxy"
Sep 17 16:36:34.596244 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:34.596215    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bc509df-381e-4e44-b59e-01e4b4cc1325" containerName="kserve-container"
Sep 17 16:36:34.596244 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:34.596220    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bc509df-381e-4e44-b59e-01e4b4cc1325" containerName="kserve-container"
Sep 17 16:36:34.596244 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:34.596228    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2102846a-adb2-4077-96db-b535461139ba" containerName="storage-initializer"
Sep 17 16:36:34.596244 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:34.596234    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="2102846a-adb2-4077-96db-b535461139ba" containerName="storage-initializer"
Sep 17 16:36:34.596244 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:34.596239    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d08bf548-bf5b-41eb-b946-4d35e6c193f8" containerName="kube-rbac-proxy"
Sep 17 16:36:34.596244 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:34.596244    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="d08bf548-bf5b-41eb-b946-4d35e6c193f8" containerName="kube-rbac-proxy"
Sep 17 16:36:34.596244 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:34.596252    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2102846a-adb2-4077-96db-b535461139ba" containerName="kserve-container"
Sep 17 16:36:34.596763 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:34.596257    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="2102846a-adb2-4077-96db-b535461139ba" containerName="kserve-container"
Sep 17 16:36:34.596763 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:34.596268    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bc509df-381e-4e44-b59e-01e4b4cc1325" containerName="storage-initializer"
Sep 17 16:36:34.596763 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:34.596273    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bc509df-381e-4e44-b59e-01e4b4cc1325" containerName="storage-initializer"
Sep 17 16:36:34.596763 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:34.596340    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="d08bf548-bf5b-41eb-b946-4d35e6c193f8" containerName="kserve-container"
Sep 17 16:36:34.596763 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:34.596348    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="2102846a-adb2-4077-96db-b535461139ba" containerName="kube-rbac-proxy"
Sep 17 16:36:34.596763 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:34.596355    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="2102846a-adb2-4077-96db-b535461139ba" containerName="kserve-container"
Sep 17 16:36:34.596763 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:34.596360    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bc509df-381e-4e44-b59e-01e4b4cc1325" containerName="kube-rbac-proxy"
Sep 17 16:36:34.596763 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:34.596366    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bc509df-381e-4e44-b59e-01e4b4cc1325" containerName="kserve-container"
Sep 17 16:36:34.596763 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:34.596371    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="d08bf548-bf5b-41eb-b946-4d35e6c193f8" containerName="kube-rbac-proxy"
Sep 17 16:36:34.599505 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:34.599484    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-zc48r"
Sep 17 16:36:34.602371 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:34.602348    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 16:36:34.602530 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:34.602381    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
Sep 17 16:36:34.602530 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:34.602475    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xkc4g\""
Sep 17 16:36:34.602629 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:34.602607    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 16:36:34.603510 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:34.603486    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 16:36:34.603661 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:34.603576    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:36:34.603661 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:34.603627    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-predictor-serving-cert\""
Sep 17 16:36:34.620756 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:34.620720    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-zc48r"]
Sep 17 16:36:34.758616 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:34.758578    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa9881e7-0486-4427-819b-be0e9c282452-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-zc48r\" (UID: \"fa9881e7-0486-4427-819b-be0e9c282452\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-zc48r"
Sep 17 16:36:34.758616 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:34.758618    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa9881e7-0486-4427-819b-be0e9c282452-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-zc48r\" (UID: \"fa9881e7-0486-4427-819b-be0e9c282452\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-zc48r"
Sep 17 16:36:34.758894 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:34.758652    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl8sp\" (UniqueName: \"kubernetes.io/projected/fa9881e7-0486-4427-819b-be0e9c282452-kube-api-access-vl8sp\") pod \"isvc-lightgbm-predictor-54f85ddf88-zc48r\" (UID: \"fa9881e7-0486-4427-819b-be0e9c282452\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-zc48r"
Sep 17 16:36:34.758894 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:34.758668    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa9881e7-0486-4427-819b-be0e9c282452-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-zc48r\" (UID: \"fa9881e7-0486-4427-819b-be0e9c282452\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-zc48r"
Sep 17 16:36:34.758894 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:34.758800    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa9881e7-0486-4427-819b-be0e9c282452-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-zc48r\" (UID: \"fa9881e7-0486-4427-819b-be0e9c282452\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-zc48r"
Sep 17 16:36:34.860284 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:34.860182    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa9881e7-0486-4427-819b-be0e9c282452-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-zc48r\" (UID: \"fa9881e7-0486-4427-819b-be0e9c282452\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-zc48r"
Sep 17 16:36:34.860284 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:34.860233    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa9881e7-0486-4427-819b-be0e9c282452-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-zc48r\" (UID: \"fa9881e7-0486-4427-819b-be0e9c282452\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-zc48r"
Sep 17 16:36:34.860284 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:34.860263    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa9881e7-0486-4427-819b-be0e9c282452-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-zc48r\" (UID: \"fa9881e7-0486-4427-819b-be0e9c282452\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-zc48r"
Sep 17 16:36:34.860652 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:34.860306    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl8sp\" (UniqueName: \"kubernetes.io/projected/fa9881e7-0486-4427-819b-be0e9c282452-kube-api-access-vl8sp\") pod \"isvc-lightgbm-predictor-54f85ddf88-zc48r\" (UID: \"fa9881e7-0486-4427-819b-be0e9c282452\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-zc48r"
Sep 17 16:36:34.860652 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:34.860333    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa9881e7-0486-4427-819b-be0e9c282452-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-zc48r\" (UID: \"fa9881e7-0486-4427-819b-be0e9c282452\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-zc48r"
Sep 17 16:36:34.860787 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:34.860759    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa9881e7-0486-4427-819b-be0e9c282452-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-zc48r\" (UID: \"fa9881e7-0486-4427-819b-be0e9c282452\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-zc48r"
Sep 17 16:36:34.860963 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:34.860945    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa9881e7-0486-4427-819b-be0e9c282452-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-zc48r\" (UID: \"fa9881e7-0486-4427-819b-be0e9c282452\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-zc48r"
Sep 17 16:36:34.861008 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:34.860946    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa9881e7-0486-4427-819b-be0e9c282452-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-zc48r\" (UID: \"fa9881e7-0486-4427-819b-be0e9c282452\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-zc48r"
Sep 17 16:36:34.862778 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:34.862756    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa9881e7-0486-4427-819b-be0e9c282452-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-zc48r\" (UID: \"fa9881e7-0486-4427-819b-be0e9c282452\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-zc48r"
Sep 17 16:36:34.870158 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:34.870133    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl8sp\" (UniqueName: \"kubernetes.io/projected/fa9881e7-0486-4427-819b-be0e9c282452-kube-api-access-vl8sp\") pod \"isvc-lightgbm-predictor-54f85ddf88-zc48r\" (UID: \"fa9881e7-0486-4427-819b-be0e9c282452\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-zc48r"
Sep 17 16:36:34.910946 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:34.910908    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-zc48r"
Sep 17 16:36:35.189634 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:35.189543    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-zc48r"]
Sep 17 16:36:35.192553 ip-10-0-135-43 kubenswrapper[2585]: W0917 16:36:35.192522    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa9881e7_0486_4427_819b_be0e9c282452.slice/crio-9f599ac5ef9a7b5c3de2a9dacc33ce9f9298ed332fb1c22fd631a17a2c6dd636 WatchSource:0}: Error finding container 9f599ac5ef9a7b5c3de2a9dacc33ce9f9298ed332fb1c22fd631a17a2c6dd636: Status 404 returned error can't find the container with id 9f599ac5ef9a7b5c3de2a9dacc33ce9f9298ed332fb1c22fd631a17a2c6dd636
Sep 17 16:36:35.789154 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:35.789122    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-zc48r" event={"ID":"fa9881e7-0486-4427-819b-be0e9c282452","Type":"ContainerStarted","Data":"4abdb86c787a2d9e27d646afb4ad82d92e6190bbcddcb2a3a1da196ed26c2e2f"}
Sep 17 16:36:35.789559 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:35.789178    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-zc48r" event={"ID":"fa9881e7-0486-4427-819b-be0e9c282452","Type":"ContainerStarted","Data":"9f599ac5ef9a7b5c3de2a9dacc33ce9f9298ed332fb1c22fd631a17a2c6dd636"}
Sep 17 16:36:39.808619 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:39.808584    2585 generic.go:356] "Generic (PLEG): container finished" podID="fa9881e7-0486-4427-819b-be0e9c282452" containerID="4abdb86c787a2d9e27d646afb4ad82d92e6190bbcddcb2a3a1da196ed26c2e2f" exitCode=0
Sep 17 16:36:39.809008 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:39.808657    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-zc48r" event={"ID":"fa9881e7-0486-4427-819b-be0e9c282452","Type":"ContainerDied","Data":"4abdb86c787a2d9e27d646afb4ad82d92e6190bbcddcb2a3a1da196ed26c2e2f"}
Sep 17 16:36:39.809960 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:39.809945    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 16:36:46.853807 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:46.853770    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-zc48r" event={"ID":"fa9881e7-0486-4427-819b-be0e9c282452","Type":"ContainerStarted","Data":"df22a7fcf867a0f6277d93279ef9715e283cbdead73da18714116087ea8fedc7"}
Sep 17 16:36:46.854217 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:46.853833    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-zc48r" event={"ID":"fa9881e7-0486-4427-819b-be0e9c282452","Type":"ContainerStarted","Data":"dc6d652f977bd9d72748756e10964e18a5b06f4b14fa27194c6ea1baf97741ac"}
Sep 17 16:36:46.854217 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:46.854098    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-zc48r"
Sep 17 16:36:46.854340 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:46.854253    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-zc48r"
Sep 17 16:36:46.855554 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:46.855529    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-zc48r" podUID="fa9881e7-0486-4427-819b-be0e9c282452" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 16:36:46.882222 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:46.882174    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-zc48r" podStartSLOduration=2.687430006 podStartE2EDuration="12.882156676s" totalImagesPullingTime="6.325406836s" totalInitContainerRuntime="3.869319834s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:36:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:36:46.880565455 +0000 UTC m=+2562.650897253" watchObservedRunningTime="2026-09-17 16:36:46.882156676 +0000 UTC m=+2562.652488474"
Sep 17 16:36:47.859508 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:47.859470    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-zc48r" podUID="fa9881e7-0486-4427-819b-be0e9c282452" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 16:36:52.864722 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:52.864692    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-zc48r"
Sep 17 16:36:52.865212 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:36:52.865181    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-zc48r" podUID="fa9881e7-0486-4427-819b-be0e9c282452" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 16:37:02.865855 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:37:02.865814    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-zc48r" podUID="fa9881e7-0486-4427-819b-be0e9c282452" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 16:37:12.866017 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:37:12.865978    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-zc48r" podUID="fa9881e7-0486-4427-819b-be0e9c282452" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 16:37:19.082828 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:37:19.082708    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-blcfm"]
Sep 17 16:37:19.125242 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:37:19.125209    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-blcfm"]
Sep 17 16:37:19.125460 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:37:19.125347    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-blcfm"
Sep 17 16:37:19.128406 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:37:19.128377    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 17 16:37:19.128562 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:37:19.128379    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
Sep 17 16:37:19.246675 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:37:19.246634    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4e4c43b-b97f-4ee7-85d8-b99d34ee8889-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-blcfm\" (UID: \"a4e4c43b-b97f-4ee7-85d8-b99d34ee8889\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-blcfm"
Sep 17 16:37:19.246853 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:37:19.246710    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4e4c43b-b97f-4ee7-85d8-b99d34ee8889-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-blcfm\" (UID: \"a4e4c43b-b97f-4ee7-85d8-b99d34ee8889\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-blcfm"
Sep 17 16:37:19.246853 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:37:19.246747    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhdjx\" (UniqueName: \"kubernetes.io/projected/a4e4c43b-b97f-4ee7-85d8-b99d34ee8889-kube-api-access-vhdjx\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-blcfm\" (UID: \"a4e4c43b-b97f-4ee7-85d8-b99d34ee8889\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-blcfm"
Sep 17 16:37:19.246853 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:37:19.246767    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4e4c43b-b97f-4ee7-85d8-b99d34ee8889-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-blcfm\" (UID: \"a4e4c43b-b97f-4ee7-85d8-b99d34ee8889\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-blcfm"
Sep 17 16:37:19.246853 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:37:19.246789    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4e4c43b-b97f-4ee7-85d8-b99d34ee8889-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-blcfm\" (UID: \"a4e4c43b-b97f-4ee7-85d8-b99d34ee8889\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-blcfm"
Sep 17 16:37:19.347775 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:37:19.347669    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4e4c43b-b97f-4ee7-85d8-b99d34ee8889-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-blcfm\" (UID: \"a4e4c43b-b97f-4ee7-85d8-b99d34ee8889\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-blcfm"
Sep 17 16:37:19.347775 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:37:19.347723    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4e4c43b-b97f-4ee7-85d8-b99d34ee8889-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-blcfm\" (UID: \"a4e4c43b-b97f-4ee7-85d8-b99d34ee8889\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-blcfm"
Sep 17 16:37:19.347775 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:37:19.347747    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhdjx\" (UniqueName: \"kubernetes.io/projected/a4e4c43b-b97f-4ee7-85d8-b99d34ee8889-kube-api-access-vhdjx\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-blcfm\" (UID: \"a4e4c43b-b97f-4ee7-85d8-b99d34ee8889\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-blcfm"
Sep 17 16:37:19.347775 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:37:19.347765    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4e4c43b-b97f-4ee7-85d8-b99d34ee8889-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-blcfm\" (UID: \"a4e4c43b-b97f-4ee7-85d8-b99d34ee8889\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-blcfm"
Sep 17 16:37:19.348128 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:37:19.347790    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4e4c43b-b97f-4ee7-85d8-b99d34ee8889-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-blcfm\" (UID: \"a4e4c43b-b97f-4ee7-85d8-b99d34ee8889\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-blcfm"
Sep 17 16:37:19.348261 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:37:19.348237    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4e4c43b-b97f-4ee7-85d8-b99d34ee8889-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-blcfm\" (UID: \"a4e4c43b-b97f-4ee7-85d8-b99d34ee8889\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-blcfm"
Sep 17 16:37:19.348481 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:37:19.348454    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4e4c43b-b97f-4ee7-85d8-b99d34ee8889-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-blcfm\" (UID: \"a4e4c43b-b97f-4ee7-85d8-b99d34ee8889\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-blcfm"
Sep 17 16:37:19.348558 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:37:19.348500    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4e4c43b-b97f-4ee7-85d8-b99d34ee8889-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-blcfm\" (UID: \"a4e4c43b-b97f-4ee7-85d8-b99d34ee8889\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-blcfm"
Sep 17 16:37:19.350310 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:37:19.350294    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4e4c43b-b97f-4ee7-85d8-b99d34ee8889-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-blcfm\" (UID: \"a4e4c43b-b97f-4ee7-85d8-b99d34ee8889\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-blcfm"
Sep 17 16:37:19.357160 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:37:19.357134    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhdjx\" (UniqueName: \"kubernetes.io/projected/a4e4c43b-b97f-4ee7-85d8-b99d34ee8889-kube-api-access-vhdjx\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-blcfm\" (UID: \"a4e4c43b-b97f-4ee7-85d8-b99d34ee8889\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-blcfm"
Sep 17 16:37:19.436136 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:37:19.436101    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-blcfm"
Sep 17 16:37:19.745718 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:37:19.745680    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-blcfm"]
Sep 17 16:37:19.748725 ip-10-0-135-43 kubenswrapper[2585]: W0917 16:37:19.748699    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4e4c43b_b97f_4ee7_85d8_b99d34ee8889.slice/crio-2f0090eead9e5f7f6582ed2871cfde010e23ca1795c164917df173c0085a45e9 WatchSource:0}: Error finding container 2f0090eead9e5f7f6582ed2871cfde010e23ca1795c164917df173c0085a45e9: Status 404 returned error can't find the container with id 2f0090eead9e5f7f6582ed2871cfde010e23ca1795c164917df173c0085a45e9
Sep 17 16:37:20.007821 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:37:20.007724    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-blcfm" event={"ID":"a4e4c43b-b97f-4ee7-85d8-b99d34ee8889","Type":"ContainerStarted","Data":"0407d841d5450b613b41db42e642e4f4c29dbdf8864a5b0f46e11832f638bfdc"}
Sep 17 16:37:20.007821 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:37:20.007784    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-blcfm" event={"ID":"a4e4c43b-b97f-4ee7-85d8-b99d34ee8889","Type":"ContainerStarted","Data":"2f0090eead9e5f7f6582ed2871cfde010e23ca1795c164917df173c0085a45e9"}
Sep 17 16:37:22.866081 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:37:22.866037    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-zc48r" podUID="fa9881e7-0486-4427-819b-be0e9c282452" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 16:37:24.026819 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:37:24.026781    2585 generic.go:356] "Generic (PLEG): container finished" podID="a4e4c43b-b97f-4ee7-85d8-b99d34ee8889" containerID="0407d841d5450b613b41db42e642e4f4c29dbdf8864a5b0f46e11832f638bfdc" exitCode=0
Sep 17 16:37:24.027309 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:37:24.026841    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-blcfm" event={"ID":"a4e4c43b-b97f-4ee7-85d8-b99d34ee8889","Type":"ContainerDied","Data":"0407d841d5450b613b41db42e642e4f4c29dbdf8864a5b0f46e11832f638bfdc"}
Sep 17 16:37:25.033518 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:37:25.033481    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-blcfm" event={"ID":"a4e4c43b-b97f-4ee7-85d8-b99d34ee8889","Type":"ContainerStarted","Data":"ccd68cb09b2cfd2480ae1e475b915996c02688dc5ca206fbeeff91ffd5a9e5d0"}
Sep 17 16:37:25.033992 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:37:25.033557    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-blcfm" event={"ID":"a4e4c43b-b97f-4ee7-85d8-b99d34ee8889","Type":"ContainerStarted","Data":"e0726a5c8874dca08b8201815bd22cf47eb0efd183df2eec18156cbac1aba895"}
Sep 17 16:37:25.033992 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:37:25.033776    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-blcfm"
Sep 17 16:37:25.059654 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:37:25.059601    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-blcfm" podStartSLOduration=2.018627751 podStartE2EDuration="6.059582201s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.04095445s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:37:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:37:25.056920075 +0000 UTC m=+2600.827251874" watchObservedRunningTime="2026-09-17 16:37:25.059582201 +0000 UTC m=+2600.829913998"
Sep 17 16:37:26.038338 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:37:26.038308    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-blcfm"
Sep 17 16:37:32.047569 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:37:32.047539    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-blcfm"
Sep 17 16:37:32.865979 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:37:32.865939    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-zc48r" podUID="fa9881e7-0486-4427-819b-be0e9c282452" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 16:37:42.865610 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:37:42.865568    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-zc48r" podUID="fa9881e7-0486-4427-819b-be0e9c282452" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 16:37:52.865489 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:37:52.865445    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-zc48r" podUID="fa9881e7-0486-4427-819b-be0e9c282452" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 16:38:02.413221 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:02.413166    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-blcfm" podUID="a4e4c43b-b97f-4ee7-85d8-b99d34ee8889" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 17 16:38:02.865885 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:02.865845    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-zc48r" podUID="fa9881e7-0486-4427-819b-be0e9c282452" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 16:38:12.050434 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:12.050384    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-blcfm"
Sep 17 16:38:12.865583 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:12.865545    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-zc48r" podUID="fa9881e7-0486-4427-819b-be0e9c282452" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 17 16:38:19.184751 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:19.184715    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-blcfm"]
Sep 17 16:38:19.185619 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:19.184983    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-blcfm" podUID="a4e4c43b-b97f-4ee7-85d8-b99d34ee8889" containerName="kserve-container" containerID="cri-o://e0726a5c8874dca08b8201815bd22cf47eb0efd183df2eec18156cbac1aba895" gracePeriod=30
Sep 17 16:38:19.185619 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:19.185048    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-blcfm" podUID="a4e4c43b-b97f-4ee7-85d8-b99d34ee8889" containerName="kube-rbac-proxy" containerID="cri-o://ccd68cb09b2cfd2480ae1e475b915996c02688dc5ca206fbeeff91ffd5a9e5d0" gracePeriod=30
Sep 17 16:38:20.298022 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:20.297986    2585 generic.go:356] "Generic (PLEG): container finished" podID="a4e4c43b-b97f-4ee7-85d8-b99d34ee8889" containerID="ccd68cb09b2cfd2480ae1e475b915996c02688dc5ca206fbeeff91ffd5a9e5d0" exitCode=2
Sep 17 16:38:20.298412 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:20.298040    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-blcfm" event={"ID":"a4e4c43b-b97f-4ee7-85d8-b99d34ee8889","Type":"ContainerDied","Data":"ccd68cb09b2cfd2480ae1e475b915996c02688dc5ca206fbeeff91ffd5a9e5d0"}
Sep 17 16:38:21.508171 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:21.508138    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-zc48r"
Sep 17 16:38:22.043431 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:22.043388    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-blcfm" podUID="a4e4c43b-b97f-4ee7-85d8-b99d34ee8889" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.55:8643/healthz\": dial tcp 10.134.0.55:8643: connect: connection refused"
Sep 17 16:38:25.538780 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:25.538738    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-zc48r"]
Sep 17 16:38:25.539260 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:25.538960    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-zc48r" podUID="fa9881e7-0486-4427-819b-be0e9c282452" containerName="kserve-container" containerID="cri-o://dc6d652f977bd9d72748756e10964e18a5b06f4b14fa27194c6ea1baf97741ac" gracePeriod=30
Sep 17 16:38:25.539260 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:25.539004    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-zc48r" podUID="fa9881e7-0486-4427-819b-be0e9c282452" containerName="kube-rbac-proxy" containerID="cri-o://df22a7fcf867a0f6277d93279ef9715e283cbdead73da18714116087ea8fedc7" gracePeriod=30
Sep 17 16:38:26.332920 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:26.332889    2585 generic.go:356] "Generic (PLEG): container finished" podID="fa9881e7-0486-4427-819b-be0e9c282452" containerID="df22a7fcf867a0f6277d93279ef9715e283cbdead73da18714116087ea8fedc7" exitCode=2
Sep 17 16:38:26.333094 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:26.332946    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-zc48r" event={"ID":"fa9881e7-0486-4427-819b-be0e9c282452","Type":"ContainerDied","Data":"df22a7fcf867a0f6277d93279ef9715e283cbdead73da18714116087ea8fedc7"}
Sep 17 16:38:27.027360 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:27.027334    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-blcfm"
Sep 17 16:38:27.095350 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:27.095253    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhdjx\" (UniqueName: \"kubernetes.io/projected/a4e4c43b-b97f-4ee7-85d8-b99d34ee8889-kube-api-access-vhdjx\") pod \"a4e4c43b-b97f-4ee7-85d8-b99d34ee8889\" (UID: \"a4e4c43b-b97f-4ee7-85d8-b99d34ee8889\") "
Sep 17 16:38:27.095350 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:27.095312    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4e4c43b-b97f-4ee7-85d8-b99d34ee8889-kserve-provision-location\") pod \"a4e4c43b-b97f-4ee7-85d8-b99d34ee8889\" (UID: \"a4e4c43b-b97f-4ee7-85d8-b99d34ee8889\") "
Sep 17 16:38:27.095350 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:27.095334    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4e4c43b-b97f-4ee7-85d8-b99d34ee8889-proxy-tls\") pod \"a4e4c43b-b97f-4ee7-85d8-b99d34ee8889\" (UID: \"a4e4c43b-b97f-4ee7-85d8-b99d34ee8889\") "
Sep 17 16:38:27.095350 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:27.095352    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4e4c43b-b97f-4ee7-85d8-b99d34ee8889-cabundle-cert\") pod \"a4e4c43b-b97f-4ee7-85d8-b99d34ee8889\" (UID: \"a4e4c43b-b97f-4ee7-85d8-b99d34ee8889\") "
Sep 17 16:38:27.095737 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:27.095415    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4e4c43b-b97f-4ee7-85d8-b99d34ee8889-kube-rbac-proxy-sar-config\") pod \"a4e4c43b-b97f-4ee7-85d8-b99d34ee8889\" (UID: \"a4e4c43b-b97f-4ee7-85d8-b99d34ee8889\") "
Sep 17 16:38:27.095816 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:27.095786    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4e4c43b-b97f-4ee7-85d8-b99d34ee8889-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a4e4c43b-b97f-4ee7-85d8-b99d34ee8889" (UID: "a4e4c43b-b97f-4ee7-85d8-b99d34ee8889"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:38:27.095862 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:27.095819    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4e4c43b-b97f-4ee7-85d8-b99d34ee8889-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a4e4c43b-b97f-4ee7-85d8-b99d34ee8889" (UID: "a4e4c43b-b97f-4ee7-85d8-b99d34ee8889"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:38:27.095962 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:27.095930    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4e4c43b-b97f-4ee7-85d8-b99d34ee8889-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a4e4c43b-b97f-4ee7-85d8-b99d34ee8889" (UID: "a4e4c43b-b97f-4ee7-85d8-b99d34ee8889"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:38:27.097430 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:27.097392    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4e4c43b-b97f-4ee7-85d8-b99d34ee8889-kube-api-access-vhdjx" (OuterVolumeSpecName: "kube-api-access-vhdjx") pod "a4e4c43b-b97f-4ee7-85d8-b99d34ee8889" (UID: "a4e4c43b-b97f-4ee7-85d8-b99d34ee8889"). InnerVolumeSpecName "kube-api-access-vhdjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:38:27.097514 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:27.097464    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4e4c43b-b97f-4ee7-85d8-b99d34ee8889-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a4e4c43b-b97f-4ee7-85d8-b99d34ee8889" (UID: "a4e4c43b-b97f-4ee7-85d8-b99d34ee8889"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:38:27.196405 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:27.196370    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4e4c43b-b97f-4ee7-85d8-b99d34ee8889-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:38:27.196405 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:27.196400    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhdjx\" (UniqueName: \"kubernetes.io/projected/a4e4c43b-b97f-4ee7-85d8-b99d34ee8889-kube-api-access-vhdjx\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:38:27.196405 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:27.196409    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4e4c43b-b97f-4ee7-85d8-b99d34ee8889-kserve-provision-location\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:38:27.196405 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:27.196440    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4e4c43b-b97f-4ee7-85d8-b99d34ee8889-proxy-tls\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:38:27.196716 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:27.196451    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4e4c43b-b97f-4ee7-85d8-b99d34ee8889-cabundle-cert\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:38:27.339104 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:27.339069    2585 generic.go:356] "Generic (PLEG): container finished" podID="a4e4c43b-b97f-4ee7-85d8-b99d34ee8889" containerID="e0726a5c8874dca08b8201815bd22cf47eb0efd183df2eec18156cbac1aba895" exitCode=0
Sep 17 16:38:27.339282 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:27.339139    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-blcfm"
Sep 17 16:38:27.339282 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:27.339138    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-blcfm" event={"ID":"a4e4c43b-b97f-4ee7-85d8-b99d34ee8889","Type":"ContainerDied","Data":"e0726a5c8874dca08b8201815bd22cf47eb0efd183df2eec18156cbac1aba895"}
Sep 17 16:38:27.339282 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:27.339273    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-blcfm" event={"ID":"a4e4c43b-b97f-4ee7-85d8-b99d34ee8889","Type":"ContainerDied","Data":"2f0090eead9e5f7f6582ed2871cfde010e23ca1795c164917df173c0085a45e9"}
Sep 17 16:38:27.339399 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:27.339313    2585 scope.go:117] "RemoveContainer" containerID="ccd68cb09b2cfd2480ae1e475b915996c02688dc5ca206fbeeff91ffd5a9e5d0"
Sep 17 16:38:27.348228 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:27.348208    2585 scope.go:117] "RemoveContainer" containerID="e0726a5c8874dca08b8201815bd22cf47eb0efd183df2eec18156cbac1aba895"
Sep 17 16:38:27.356663 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:27.356641    2585 scope.go:117] "RemoveContainer" containerID="0407d841d5450b613b41db42e642e4f4c29dbdf8864a5b0f46e11832f638bfdc"
Sep 17 16:38:27.365355 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:27.365331    2585 scope.go:117] "RemoveContainer" containerID="ccd68cb09b2cfd2480ae1e475b915996c02688dc5ca206fbeeff91ffd5a9e5d0"
Sep 17 16:38:27.365694 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:38:27.365674    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccd68cb09b2cfd2480ae1e475b915996c02688dc5ca206fbeeff91ffd5a9e5d0\": container with ID starting with ccd68cb09b2cfd2480ae1e475b915996c02688dc5ca206fbeeff91ffd5a9e5d0 not found: ID does not exist" containerID="ccd68cb09b2cfd2480ae1e475b915996c02688dc5ca206fbeeff91ffd5a9e5d0"
Sep 17 16:38:27.365752 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:27.365702    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccd68cb09b2cfd2480ae1e475b915996c02688dc5ca206fbeeff91ffd5a9e5d0"} err="failed to get container status \"ccd68cb09b2cfd2480ae1e475b915996c02688dc5ca206fbeeff91ffd5a9e5d0\": rpc error: code = NotFound desc = could not find container \"ccd68cb09b2cfd2480ae1e475b915996c02688dc5ca206fbeeff91ffd5a9e5d0\": container with ID starting with ccd68cb09b2cfd2480ae1e475b915996c02688dc5ca206fbeeff91ffd5a9e5d0 not found: ID does not exist"
Sep 17 16:38:27.365752 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:27.365721    2585 scope.go:117] "RemoveContainer" containerID="e0726a5c8874dca08b8201815bd22cf47eb0efd183df2eec18156cbac1aba895"
Sep 17 16:38:27.366016 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:38:27.366000    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0726a5c8874dca08b8201815bd22cf47eb0efd183df2eec18156cbac1aba895\": container with ID starting with e0726a5c8874dca08b8201815bd22cf47eb0efd183df2eec18156cbac1aba895 not found: ID does not exist" containerID="e0726a5c8874dca08b8201815bd22cf47eb0efd183df2eec18156cbac1aba895"
Sep 17 16:38:27.366061 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:27.366021    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0726a5c8874dca08b8201815bd22cf47eb0efd183df2eec18156cbac1aba895"} err="failed to get container status \"e0726a5c8874dca08b8201815bd22cf47eb0efd183df2eec18156cbac1aba895\": rpc error: code = NotFound desc = could not find container \"e0726a5c8874dca08b8201815bd22cf47eb0efd183df2eec18156cbac1aba895\": container with ID starting with e0726a5c8874dca08b8201815bd22cf47eb0efd183df2eec18156cbac1aba895 not found: ID does not exist"
Sep 17 16:38:27.366061 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:27.366034    2585 scope.go:117] "RemoveContainer" containerID="0407d841d5450b613b41db42e642e4f4c29dbdf8864a5b0f46e11832f638bfdc"
Sep 17 16:38:27.366263 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:27.366243    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-blcfm"]
Sep 17 16:38:27.366302 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:38:27.366251    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0407d841d5450b613b41db42e642e4f4c29dbdf8864a5b0f46e11832f638bfdc\": container with ID starting with 0407d841d5450b613b41db42e642e4f4c29dbdf8864a5b0f46e11832f638bfdc not found: ID does not exist" containerID="0407d841d5450b613b41db42e642e4f4c29dbdf8864a5b0f46e11832f638bfdc"
Sep 17 16:38:27.366335 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:27.366301    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0407d841d5450b613b41db42e642e4f4c29dbdf8864a5b0f46e11832f638bfdc"} err="failed to get container status \"0407d841d5450b613b41db42e642e4f4c29dbdf8864a5b0f46e11832f638bfdc\": rpc error: code = NotFound desc = could not find container \"0407d841d5450b613b41db42e642e4f4c29dbdf8864a5b0f46e11832f638bfdc\": container with ID starting with 0407d841d5450b613b41db42e642e4f4c29dbdf8864a5b0f46e11832f638bfdc not found: ID does not exist"
Sep 17 16:38:27.375088 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:27.375058    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-blcfm"]
Sep 17 16:38:27.859749 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:27.859697    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-zc48r" podUID="fa9881e7-0486-4427-819b-be0e9c282452" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.54:8643/healthz\": dial tcp 10.134.0.54:8643: connect: connection refused"
Sep 17 16:38:28.512308 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:28.512267    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4e4c43b-b97f-4ee7-85d8-b99d34ee8889" path="/var/lib/kubelet/pods/a4e4c43b-b97f-4ee7-85d8-b99d34ee8889/volumes"
Sep 17 16:38:30.718923 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:30.718876    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp"]
Sep 17 16:38:30.719465 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:30.719446    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4e4c43b-b97f-4ee7-85d8-b99d34ee8889" containerName="storage-initializer"
Sep 17 16:38:30.719523 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:30.719468    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4e4c43b-b97f-4ee7-85d8-b99d34ee8889" containerName="storage-initializer"
Sep 17 16:38:30.719523 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:30.719481    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4e4c43b-b97f-4ee7-85d8-b99d34ee8889" containerName="kube-rbac-proxy"
Sep 17 16:38:30.719523 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:30.719489    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4e4c43b-b97f-4ee7-85d8-b99d34ee8889" containerName="kube-rbac-proxy"
Sep 17 16:38:30.719523 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:30.719499    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4e4c43b-b97f-4ee7-85d8-b99d34ee8889" containerName="kserve-container"
Sep 17 16:38:30.719523 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:30.719509    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4e4c43b-b97f-4ee7-85d8-b99d34ee8889" containerName="kserve-container"
Sep 17 16:38:30.719688 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:30.719589    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4e4c43b-b97f-4ee7-85d8-b99d34ee8889" containerName="kserve-container"
Sep 17 16:38:30.719688 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:30.719607    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4e4c43b-b97f-4ee7-85d8-b99d34ee8889" containerName="kube-rbac-proxy"
Sep 17 16:38:30.724811 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:30.724787    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp"
Sep 17 16:38:30.727398 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:30.727373    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
Sep 17 16:38:30.727549 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:30.727376    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
Sep 17 16:38:30.733969 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:30.733932    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp"]
Sep 17 16:38:30.826635 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:30.826597    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp\" (UID: \"e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp"
Sep 17 16:38:30.826839 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:30.826652    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp\" (UID: \"e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp"
Sep 17 16:38:30.826839 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:30.826703    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp\" (UID: \"e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp"
Sep 17 16:38:30.826839 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:30.826735    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp\" (UID: \"e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp"
Sep 17 16:38:30.826839 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:30.826790    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8c4d\" (UniqueName: \"kubernetes.io/projected/e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f-kube-api-access-n8c4d\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp\" (UID: \"e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp"
Sep 17 16:38:30.927586 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:30.927543    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp\" (UID: \"e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp"
Sep 17 16:38:30.927795 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:30.927598    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8c4d\" (UniqueName: \"kubernetes.io/projected/e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f-kube-api-access-n8c4d\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp\" (UID: \"e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp"
Sep 17 16:38:30.927795 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:30.927674    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp\" (UID: \"e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp"
Sep 17 16:38:30.927795 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:30.927715    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp\" (UID: \"e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp"
Sep 17 16:38:30.927795 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:30.927747    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp\" (UID: \"e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp"
Sep 17 16:38:30.928140 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:30.928116    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp\" (UID: \"e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp"
Sep 17 16:38:30.928356 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:30.928339    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp\" (UID: \"e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp"
Sep 17 16:38:30.928395 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:30.928349    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp\" (UID: \"e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp"
Sep 17 16:38:30.930185 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:30.930167    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp\" (UID: \"e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp"
Sep 17 16:38:30.937280 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:30.937256    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8c4d\" (UniqueName: \"kubernetes.io/projected/e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f-kube-api-access-n8c4d\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp\" (UID: \"e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp"
Sep 17 16:38:31.037175 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:31.037137    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp"
Sep 17 16:38:31.313990 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:31.313960    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp"]
Sep 17 16:38:31.316484 ip-10-0-135-43 kubenswrapper[2585]: W0917 16:38:31.316451    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0c3d4be_dca6_4ccc_b0a5_f81e5a75349f.slice/crio-3d38eb7d1f90f09c450e1a060120234d335518103095ca3b12a2915172f8f022 WatchSource:0}: Error finding container 3d38eb7d1f90f09c450e1a060120234d335518103095ca3b12a2915172f8f022: Status 404 returned error can't find the container with id 3d38eb7d1f90f09c450e1a060120234d335518103095ca3b12a2915172f8f022
Sep 17 16:38:31.360360 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:31.360330    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp" event={"ID":"e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f","Type":"ContainerStarted","Data":"3d38eb7d1f90f09c450e1a060120234d335518103095ca3b12a2915172f8f022"}
Sep 17 16:38:31.362845 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:31.362819    2585 generic.go:356] "Generic (PLEG): container finished" podID="fa9881e7-0486-4427-819b-be0e9c282452" containerID="dc6d652f977bd9d72748756e10964e18a5b06f4b14fa27194c6ea1baf97741ac" exitCode=0
Sep 17 16:38:31.362971 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:31.362888    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-zc48r" event={"ID":"fa9881e7-0486-4427-819b-be0e9c282452","Type":"ContainerDied","Data":"dc6d652f977bd9d72748756e10964e18a5b06f4b14fa27194c6ea1baf97741ac"}
Sep 17 16:38:31.484759 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:31.484732    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-zc48r"
Sep 17 16:38:31.632390 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:31.632302    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa9881e7-0486-4427-819b-be0e9c282452-proxy-tls\") pod \"fa9881e7-0486-4427-819b-be0e9c282452\" (UID: \"fa9881e7-0486-4427-819b-be0e9c282452\") "
Sep 17 16:38:31.632390 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:31.632363    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vl8sp\" (UniqueName: \"kubernetes.io/projected/fa9881e7-0486-4427-819b-be0e9c282452-kube-api-access-vl8sp\") pod \"fa9881e7-0486-4427-819b-be0e9c282452\" (UID: \"fa9881e7-0486-4427-819b-be0e9c282452\") "
Sep 17 16:38:31.632682 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:31.632464    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa9881e7-0486-4427-819b-be0e9c282452-kserve-provision-location\") pod \"fa9881e7-0486-4427-819b-be0e9c282452\" (UID: \"fa9881e7-0486-4427-819b-be0e9c282452\") "
Sep 17 16:38:31.632682 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:31.632504    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa9881e7-0486-4427-819b-be0e9c282452-cabundle-cert\") pod \"fa9881e7-0486-4427-819b-be0e9c282452\" (UID: \"fa9881e7-0486-4427-819b-be0e9c282452\") "
Sep 17 16:38:31.632682 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:31.632585    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa9881e7-0486-4427-819b-be0e9c282452-kube-rbac-proxy-sar-config\") pod \"fa9881e7-0486-4427-819b-be0e9c282452\" (UID: \"fa9881e7-0486-4427-819b-be0e9c282452\") "
Sep 17 16:38:31.632848 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:31.632805    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa9881e7-0486-4427-819b-be0e9c282452-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fa9881e7-0486-4427-819b-be0e9c282452" (UID: "fa9881e7-0486-4427-819b-be0e9c282452"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:38:31.632949 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:31.632925    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa9881e7-0486-4427-819b-be0e9c282452-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fa9881e7-0486-4427-819b-be0e9c282452" (UID: "fa9881e7-0486-4427-819b-be0e9c282452"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:38:31.633020 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:31.632989    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa9881e7-0486-4427-819b-be0e9c282452-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fa9881e7-0486-4427-819b-be0e9c282452" (UID: "fa9881e7-0486-4427-819b-be0e9c282452"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:38:31.634607 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:31.634582    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa9881e7-0486-4427-819b-be0e9c282452-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fa9881e7-0486-4427-819b-be0e9c282452" (UID: "fa9881e7-0486-4427-819b-be0e9c282452"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:38:31.634607 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:31.634597    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa9881e7-0486-4427-819b-be0e9c282452-kube-api-access-vl8sp" (OuterVolumeSpecName: "kube-api-access-vl8sp") pod "fa9881e7-0486-4427-819b-be0e9c282452" (UID: "fa9881e7-0486-4427-819b-be0e9c282452"). InnerVolumeSpecName "kube-api-access-vl8sp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:38:31.734222 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:31.734182    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa9881e7-0486-4427-819b-be0e9c282452-proxy-tls\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:38:31.734222 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:31.734216    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vl8sp\" (UniqueName: \"kubernetes.io/projected/fa9881e7-0486-4427-819b-be0e9c282452-kube-api-access-vl8sp\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:38:31.734222 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:31.734226    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa9881e7-0486-4427-819b-be0e9c282452-kserve-provision-location\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:38:31.734658 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:31.734235    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa9881e7-0486-4427-819b-be0e9c282452-cabundle-cert\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:38:31.734658 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:31.734243    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa9881e7-0486-4427-819b-be0e9c282452-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:38:32.375216 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:32.375184    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-zc48r"
Sep 17 16:38:32.375216 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:32.375188    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-zc48r" event={"ID":"fa9881e7-0486-4427-819b-be0e9c282452","Type":"ContainerDied","Data":"9f599ac5ef9a7b5c3de2a9dacc33ce9f9298ed332fb1c22fd631a17a2c6dd636"}
Sep 17 16:38:32.375509 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:32.375280    2585 scope.go:117] "RemoveContainer" containerID="df22a7fcf867a0f6277d93279ef9715e283cbdead73da18714116087ea8fedc7"
Sep 17 16:38:32.376931 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:32.376900    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp" event={"ID":"e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f","Type":"ContainerStarted","Data":"c21f0ba249c74df1525692c1ea69f65cea476d6abb4ecffd668705096a3967f8"}
Sep 17 16:38:32.384290 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:32.384273    2585 scope.go:117] "RemoveContainer" containerID="dc6d652f977bd9d72748756e10964e18a5b06f4b14fa27194c6ea1baf97741ac"
Sep 17 16:38:32.391985 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:32.391967    2585 scope.go:117] "RemoveContainer" containerID="4abdb86c787a2d9e27d646afb4ad82d92e6190bbcddcb2a3a1da196ed26c2e2f"
Sep 17 16:38:32.439832 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:32.439794    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-zc48r"]
Sep 17 16:38:32.447282 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:32.447255    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-zc48r"]
Sep 17 16:38:32.513662 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:32.513627    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa9881e7-0486-4427-819b-be0e9c282452" path="/var/lib/kubelet/pods/fa9881e7-0486-4427-819b-be0e9c282452/volumes"
Sep 17 16:38:36.398673 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:36.398643    2585 generic.go:356] "Generic (PLEG): container finished" podID="e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f" containerID="c21f0ba249c74df1525692c1ea69f65cea476d6abb4ecffd668705096a3967f8" exitCode=0
Sep 17 16:38:36.399122 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:36.398710    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp" event={"ID":"e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f","Type":"ContainerDied","Data":"c21f0ba249c74df1525692c1ea69f65cea476d6abb4ecffd668705096a3967f8"}
Sep 17 16:38:37.406084 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:37.406047    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp" event={"ID":"e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f","Type":"ContainerStarted","Data":"cbb2e4af4dfe7b67318e6b2ea37866fc775ccb5655bd798201601e71eecb980f"}
Sep 17 16:38:37.406611 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:37.406154    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp" event={"ID":"e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f","Type":"ContainerStarted","Data":"6f7bbb966bce01c4cc5c26c666eedfcb7cb3dd96cf91f9b1f3a3bd3fb5ef54ed"}
Sep 17 16:38:37.406611 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:37.406384    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp"
Sep 17 16:38:37.432609 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:37.432546    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp" podStartSLOduration=3.350261688 podStartE2EDuration="7.432526112s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.082264424s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:38:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:38:37.431288479 +0000 UTC m=+2673.201620276" watchObservedRunningTime="2026-09-17 16:38:37.432526112 +0000 UTC m=+2673.202857910"
Sep 17 16:38:38.411339 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:38.411307    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp"
Sep 17 16:38:38.412923 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:38.412891    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp" podUID="e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 17 16:38:39.417436 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:39.417385    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp" podUID="e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 17 16:38:44.422566 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:44.422535    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp"
Sep 17 16:38:44.423042 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:44.423013    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp" podUID="e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 17 16:38:54.423639 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:38:54.423548    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp" podUID="e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 17 16:39:04.423401 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:39:04.423355    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp" podUID="e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 17 16:39:14.423114 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:39:14.423076    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp" podUID="e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 17 16:39:24.423662 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:39:24.423626    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp" podUID="e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 17 16:39:34.423633 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:39:34.423589    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp" podUID="e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 17 16:39:44.423229 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:39:44.423184    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp" podUID="e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 17 16:39:47.798333 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:39:47.798302    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2b47p_a1850a4d-4d01-4a71-8636-b5918f1426c2/ovn-acl-logging/0.log"
Sep 17 16:39:47.803724 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:39:47.803702    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2b47p_a1850a4d-4d01-4a71-8636-b5918f1426c2/ovn-acl-logging/0.log"
Sep 17 16:39:50.250209 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:39:50.250175    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-db86d8dd4-66krn"]
Sep 17 16:39:50.250613 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:39:50.250528    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa9881e7-0486-4427-819b-be0e9c282452" containerName="kserve-container"
Sep 17 16:39:50.250613 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:39:50.250540    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa9881e7-0486-4427-819b-be0e9c282452" containerName="kserve-container"
Sep 17 16:39:50.250613 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:39:50.250557    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa9881e7-0486-4427-819b-be0e9c282452" containerName="storage-initializer"
Sep 17 16:39:50.250613 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:39:50.250563    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa9881e7-0486-4427-819b-be0e9c282452" containerName="storage-initializer"
Sep 17 16:39:50.250613 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:39:50.250571    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa9881e7-0486-4427-819b-be0e9c282452" containerName="kube-rbac-proxy"
Sep 17 16:39:50.250613 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:39:50.250576    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa9881e7-0486-4427-819b-be0e9c282452" containerName="kube-rbac-proxy"
Sep 17 16:39:50.250817 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:39:50.250630    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa9881e7-0486-4427-819b-be0e9c282452" containerName="kube-rbac-proxy"
Sep 17 16:39:50.250817 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:39:50.250640    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa9881e7-0486-4427-819b-be0e9c282452" containerName="kserve-container"
Sep 17 16:39:50.253688 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:39:50.253667    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-db86d8dd4-66krn"
Sep 17 16:39:50.258128 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:39:50.258107    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-predictor-serving-cert\""
Sep 17 16:39:50.258330 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:39:50.258314    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
Sep 17 16:39:50.258535 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:39:50.258518    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 17 16:39:50.273081 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:39:50.273060    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-db86d8dd4-66krn"]
Sep 17 16:39:50.379731 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:39:50.379692    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2-proxy-tls\") pod \"isvc-sklearn-s3-predictor-db86d8dd4-66krn\" (UID: \"5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-db86d8dd4-66krn"
Sep 17 16:39:50.379921 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:39:50.379743    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2knnz\" (UniqueName: \"kubernetes.io/projected/5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2-kube-api-access-2knnz\") pod \"isvc-sklearn-s3-predictor-db86d8dd4-66krn\" (UID: \"5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-db86d8dd4-66krn"
Sep 17 16:39:50.379921 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:39:50.379780    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-db86d8dd4-66krn\" (UID: \"5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-db86d8dd4-66krn"
Sep 17 16:39:50.379921 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:39:50.379815    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-db86d8dd4-66krn\" (UID: \"5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-db86d8dd4-66krn"
Sep 17 16:39:50.481033 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:39:50.481001    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2-proxy-tls\") pod \"isvc-sklearn-s3-predictor-db86d8dd4-66krn\" (UID: \"5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-db86d8dd4-66krn"
Sep 17 16:39:50.481235 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:39:50.481041    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2knnz\" (UniqueName: \"kubernetes.io/projected/5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2-kube-api-access-2knnz\") pod \"isvc-sklearn-s3-predictor-db86d8dd4-66krn\" (UID: \"5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-db86d8dd4-66krn"
Sep 17 16:39:50.481235 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:39:50.481062    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-db86d8dd4-66krn\" (UID: \"5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-db86d8dd4-66krn"
Sep 17 16:39:50.481235 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:39:50.481088    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-db86d8dd4-66krn\" (UID: \"5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-db86d8dd4-66krn"
Sep 17 16:39:50.481235 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:39:50.481155    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-predictor-serving-cert: secret "isvc-sklearn-s3-predictor-serving-cert" not found
Sep 17 16:39:50.481235 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:39:50.481231    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2-proxy-tls podName:5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2 nodeName:}" failed. No retries permitted until 2026-09-17 16:39:50.981210891 +0000 UTC m=+2746.751542667 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2-proxy-tls") pod "isvc-sklearn-s3-predictor-db86d8dd4-66krn" (UID: "5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2") : secret "isvc-sklearn-s3-predictor-serving-cert" not found
Sep 17 16:39:50.481646 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:39:50.481509    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-db86d8dd4-66krn\" (UID: \"5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-db86d8dd4-66krn"
Sep 17 16:39:50.481714 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:39:50.481696    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-db86d8dd4-66krn\" (UID: \"5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-db86d8dd4-66krn"
Sep 17 16:39:50.493648 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:39:50.493623    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2knnz\" (UniqueName: \"kubernetes.io/projected/5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2-kube-api-access-2knnz\") pod \"isvc-sklearn-s3-predictor-db86d8dd4-66krn\" (UID: \"5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-db86d8dd4-66krn"
Sep 17 16:39:50.985626 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:39:50.985586    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2-proxy-tls\") pod \"isvc-sklearn-s3-predictor-db86d8dd4-66krn\" (UID: \"5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-db86d8dd4-66krn"
Sep 17 16:39:50.988151 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:39:50.988126    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2-proxy-tls\") pod \"isvc-sklearn-s3-predictor-db86d8dd4-66krn\" (UID: \"5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-db86d8dd4-66krn"
Sep 17 16:39:51.164383 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:39:51.164346    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-db86d8dd4-66krn"
Sep 17 16:39:51.447814 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:39:51.447788    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-db86d8dd4-66krn"]
Sep 17 16:39:51.451052 ip-10-0-135-43 kubenswrapper[2585]: W0917 16:39:51.451023    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c7dc87c_41fc_4000_97ad_d3a09d9ae8c2.slice/crio-11a538fffbcd079c1361f59b7841e506f2ecf6bd05217f91931a116baa33c17a WatchSource:0}: Error finding container 11a538fffbcd079c1361f59b7841e506f2ecf6bd05217f91931a116baa33c17a: Status 404 returned error can't find the container with id 11a538fffbcd079c1361f59b7841e506f2ecf6bd05217f91931a116baa33c17a
Sep 17 16:39:51.747975 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:39:51.747876    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-db86d8dd4-66krn" event={"ID":"5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2","Type":"ContainerStarted","Data":"d821c7532312db8bb4a9805fed80ba08456571165c5cb1fa302a5e7e2e705bde"}
Sep 17 16:39:51.747975 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:39:51.747940    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-db86d8dd4-66krn" event={"ID":"5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2","Type":"ContainerStarted","Data":"11a538fffbcd079c1361f59b7841e506f2ecf6bd05217f91931a116baa33c17a"}
Sep 17 16:39:53.759022 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:39:53.758977    2585 generic.go:356] "Generic (PLEG): container finished" podID="5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2" containerID="d821c7532312db8bb4a9805fed80ba08456571165c5cb1fa302a5e7e2e705bde" exitCode=0
Sep 17 16:39:53.759412 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:39:53.759049    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-db86d8dd4-66krn" event={"ID":"5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2","Type":"ContainerDied","Data":"d821c7532312db8bb4a9805fed80ba08456571165c5cb1fa302a5e7e2e705bde"}
Sep 17 16:39:54.423624 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:39:54.423586    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp" podUID="e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 17 16:39:54.765625 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:39:54.765592    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-db86d8dd4-66krn" event={"ID":"5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2","Type":"ContainerStarted","Data":"a903df8b7c6963507f277d8f1e4ed9fd38ae7f00b921492c93a2d67a24bbec96"}
Sep 17 16:39:54.766092 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:39:54.765649    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-db86d8dd4-66krn" event={"ID":"5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2","Type":"ContainerStarted","Data":"64f4aef979594c484782c732b4f51da76732335878366acdf56ce7125aceedf5"}
Sep 17 16:39:54.766092 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:39:54.765909    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-db86d8dd4-66krn"
Sep 17 16:39:55.770189 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:39:55.770157    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-db86d8dd4-66krn"
Sep 17 16:39:55.771409 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:39:55.771384    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-db86d8dd4-66krn" podUID="5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 17 16:39:56.774720 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:39:56.774676    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-db86d8dd4-66krn" podUID="5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 17 16:40:01.779573 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:01.779544    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-db86d8dd4-66krn"
Sep 17 16:40:01.780046 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:01.780021    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-db86d8dd4-66krn" podUID="5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 17 16:40:01.803055 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:01.802997    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-db86d8dd4-66krn" podStartSLOduration=10.438188745 podStartE2EDuration="11.802982362s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.364793617s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:39:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:39:54.800241143 +0000 UTC m=+2750.570572939" watchObservedRunningTime="2026-09-17 16:40:01.802982362 +0000 UTC m=+2757.573314159"
Sep 17 16:40:04.423643 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:04.423603    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp" podUID="e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 17 16:40:06.509776 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:06.509725    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp" podUID="e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 17 16:40:11.780865 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:11.780818    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-db86d8dd4-66krn" podUID="5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 17 16:40:16.512158 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:16.512129    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp"
Sep 17 16:40:21.780629 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:21.780532    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-db86d8dd4-66krn" podUID="5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 17 16:40:22.337611 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:22.337569    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp"]
Sep 17 16:40:22.337887 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:22.337844    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp" podUID="e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f" containerName="kserve-container" containerID="cri-o://6f7bbb966bce01c4cc5c26c666eedfcb7cb3dd96cf91f9b1f3a3bd3fb5ef54ed" gracePeriod=30
Sep 17 16:40:22.337980 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:22.337880    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp" podUID="e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f" containerName="kube-rbac-proxy" containerID="cri-o://cbb2e4af4dfe7b67318e6b2ea37866fc775ccb5655bd798201601e71eecb980f" gracePeriod=30
Sep 17 16:40:22.896566 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:22.896530    2585 generic.go:356] "Generic (PLEG): container finished" podID="e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f" containerID="cbb2e4af4dfe7b67318e6b2ea37866fc775ccb5655bd798201601e71eecb980f" exitCode=2
Sep 17 16:40:22.896975 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:22.896602    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp" event={"ID":"e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f","Type":"ContainerDied","Data":"cbb2e4af4dfe7b67318e6b2ea37866fc775ccb5655bd798201601e71eecb980f"}
Sep 17 16:40:24.418581 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:24.418531    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp" podUID="e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.56:8643/healthz\": dial tcp 10.134.0.56:8643: connect: connection refused"
Sep 17 16:40:26.513663 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:26.513605    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp" podUID="e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 17 16:40:27.572759 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:27.572719    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s6ffb"]
Sep 17 16:40:27.576860 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:27.576832    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s6ffb"
Sep 17 16:40:27.579497 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:27.579472    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 17 16:40:27.579644 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:27.579472    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
Sep 17 16:40:27.589963 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:27.589926    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s6ffb"]
Sep 17 16:40:27.711895 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:27.711845    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/530388b1-7699-4531-860d-b422b7b544c7-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s6ffb\" (UID: \"530388b1-7699-4531-860d-b422b7b544c7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s6ffb"
Sep 17 16:40:27.712100 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:27.711955    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/530388b1-7699-4531-860d-b422b7b544c7-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s6ffb\" (UID: \"530388b1-7699-4531-860d-b422b7b544c7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s6ffb"
Sep 17 16:40:27.712100 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:27.711990    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvdn9\" (UniqueName: \"kubernetes.io/projected/530388b1-7699-4531-860d-b422b7b544c7-kube-api-access-kvdn9\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s6ffb\" (UID: \"530388b1-7699-4531-860d-b422b7b544c7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s6ffb"
Sep 17 16:40:27.712100 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:27.712060    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/530388b1-7699-4531-860d-b422b7b544c7-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s6ffb\" (UID: \"530388b1-7699-4531-860d-b422b7b544c7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s6ffb"
Sep 17 16:40:27.712224 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:27.712149    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/530388b1-7699-4531-860d-b422b7b544c7-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s6ffb\" (UID: \"530388b1-7699-4531-860d-b422b7b544c7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s6ffb"
Sep 17 16:40:27.813124 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:27.813084    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/530388b1-7699-4531-860d-b422b7b544c7-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s6ffb\" (UID: \"530388b1-7699-4531-860d-b422b7b544c7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s6ffb"
Sep 17 16:40:27.813317 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:27.813145    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/530388b1-7699-4531-860d-b422b7b544c7-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s6ffb\" (UID: \"530388b1-7699-4531-860d-b422b7b544c7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s6ffb"
Sep 17 16:40:27.813317 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:27.813182    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/530388b1-7699-4531-860d-b422b7b544c7-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s6ffb\" (UID: \"530388b1-7699-4531-860d-b422b7b544c7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s6ffb"
Sep 17 16:40:27.813317 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:27.813202    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvdn9\" (UniqueName: \"kubernetes.io/projected/530388b1-7699-4531-860d-b422b7b544c7-kube-api-access-kvdn9\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s6ffb\" (UID: \"530388b1-7699-4531-860d-b422b7b544c7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s6ffb"
Sep 17 16:40:27.813317 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:27.813234    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/530388b1-7699-4531-860d-b422b7b544c7-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s6ffb\" (UID: \"530388b1-7699-4531-860d-b422b7b544c7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s6ffb"
Sep 17 16:40:27.813619 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:27.813592    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/530388b1-7699-4531-860d-b422b7b544c7-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s6ffb\" (UID: \"530388b1-7699-4531-860d-b422b7b544c7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s6ffb"
Sep 17 16:40:27.814061 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:27.814022    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/530388b1-7699-4531-860d-b422b7b544c7-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s6ffb\" (UID: \"530388b1-7699-4531-860d-b422b7b544c7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s6ffb"
Sep 17 16:40:27.814197 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:27.814100    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/530388b1-7699-4531-860d-b422b7b544c7-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s6ffb\" (UID: \"530388b1-7699-4531-860d-b422b7b544c7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s6ffb"
Sep 17 16:40:27.815833 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:27.815808    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/530388b1-7699-4531-860d-b422b7b544c7-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s6ffb\" (UID: \"530388b1-7699-4531-860d-b422b7b544c7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s6ffb"
Sep 17 16:40:27.823970 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:27.823904    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvdn9\" (UniqueName: \"kubernetes.io/projected/530388b1-7699-4531-860d-b422b7b544c7-kube-api-access-kvdn9\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s6ffb\" (UID: \"530388b1-7699-4531-860d-b422b7b544c7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s6ffb"
Sep 17 16:40:27.887637 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:27.887592    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s6ffb"
Sep 17 16:40:28.166109 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:28.166082    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s6ffb"]
Sep 17 16:40:28.168539 ip-10-0-135-43 kubenswrapper[2585]: W0917 16:40:28.168506    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod530388b1_7699_4531_860d_b422b7b544c7.slice/crio-e541bc501066efbde53f8aedc7ce912a822110b3142208fe4145d8a94c3b0c75 WatchSource:0}: Error finding container e541bc501066efbde53f8aedc7ce912a822110b3142208fe4145d8a94c3b0c75: Status 404 returned error can't find the container with id e541bc501066efbde53f8aedc7ce912a822110b3142208fe4145d8a94c3b0c75
Sep 17 16:40:28.582291 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:28.582263    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp"
Sep 17 16:40:28.720828 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:28.720737    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f-cabundle-cert\") pod \"e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f\" (UID: \"e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f\") "
Sep 17 16:40:28.720828 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:28.720773    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f-kube-rbac-proxy-sar-config\") pod \"e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f\" (UID: \"e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f\") "
Sep 17 16:40:28.720828 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:28.720801    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f-kserve-provision-location\") pod \"e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f\" (UID: \"e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f\") "
Sep 17 16:40:28.721130 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:28.720960    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8c4d\" (UniqueName: \"kubernetes.io/projected/e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f-kube-api-access-n8c4d\") pod \"e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f\" (UID: \"e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f\") "
Sep 17 16:40:28.721130 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:28.721015    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f-proxy-tls\") pod \"e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f\" (UID: \"e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f\") "
Sep 17 16:40:28.721231 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:28.721128    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f" (UID: "e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:40:28.721231 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:28.721137    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f" (UID: "e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:40:28.721231 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:28.721140    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f" (UID: "e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:40:28.721377 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:28.721315    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f-cabundle-cert\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:40:28.721377 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:28.721336    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:40:28.721377 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:28.721351    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f-kserve-provision-location\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:40:28.723087 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:28.723065    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f" (UID: "e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:40:28.723231 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:28.723206    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f-kube-api-access-n8c4d" (OuterVolumeSpecName: "kube-api-access-n8c4d") pod "e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f" (UID: "e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f"). InnerVolumeSpecName "kube-api-access-n8c4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:40:28.822376 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:28.822329    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8c4d\" (UniqueName: \"kubernetes.io/projected/e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f-kube-api-access-n8c4d\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:40:28.822376 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:28.822375    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f-proxy-tls\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:40:28.926944 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:28.926908    2585 generic.go:356] "Generic (PLEG): container finished" podID="e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f" containerID="6f7bbb966bce01c4cc5c26c666eedfcb7cb3dd96cf91f9b1f3a3bd3fb5ef54ed" exitCode=0
Sep 17 16:40:28.927145 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:28.926972    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp"
Sep 17 16:40:28.927145 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:28.927001    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp" event={"ID":"e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f","Type":"ContainerDied","Data":"6f7bbb966bce01c4cc5c26c666eedfcb7cb3dd96cf91f9b1f3a3bd3fb5ef54ed"}
Sep 17 16:40:28.927145 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:28.927075    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp" event={"ID":"e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f","Type":"ContainerDied","Data":"3d38eb7d1f90f09c450e1a060120234d335518103095ca3b12a2915172f8f022"}
Sep 17 16:40:28.927145 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:28.927111    2585 scope.go:117] "RemoveContainer" containerID="cbb2e4af4dfe7b67318e6b2ea37866fc775ccb5655bd798201601e71eecb980f"
Sep 17 16:40:28.928933 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:28.928905    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s6ffb" event={"ID":"530388b1-7699-4531-860d-b422b7b544c7","Type":"ContainerStarted","Data":"9b3ffc761074f6d6edf75658d25c6d9a3dac1b885e097e2e6443f8112953838a"}
Sep 17 16:40:28.929067 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:28.928968    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s6ffb" event={"ID":"530388b1-7699-4531-860d-b422b7b544c7","Type":"ContainerStarted","Data":"e541bc501066efbde53f8aedc7ce912a822110b3142208fe4145d8a94c3b0c75"}
Sep 17 16:40:28.936655 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:28.936637    2585 scope.go:117] "RemoveContainer" containerID="6f7bbb966bce01c4cc5c26c666eedfcb7cb3dd96cf91f9b1f3a3bd3fb5ef54ed"
Sep 17 16:40:28.945629 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:28.945609    2585 scope.go:117] "RemoveContainer" containerID="c21f0ba249c74df1525692c1ea69f65cea476d6abb4ecffd668705096a3967f8"
Sep 17 16:40:28.954842 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:28.954821    2585 scope.go:117] "RemoveContainer" containerID="cbb2e4af4dfe7b67318e6b2ea37866fc775ccb5655bd798201601e71eecb980f"
Sep 17 16:40:28.955179 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:40:28.955162    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbb2e4af4dfe7b67318e6b2ea37866fc775ccb5655bd798201601e71eecb980f\": container with ID starting with cbb2e4af4dfe7b67318e6b2ea37866fc775ccb5655bd798201601e71eecb980f not found: ID does not exist" containerID="cbb2e4af4dfe7b67318e6b2ea37866fc775ccb5655bd798201601e71eecb980f"
Sep 17 16:40:28.955225 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:28.955195    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbb2e4af4dfe7b67318e6b2ea37866fc775ccb5655bd798201601e71eecb980f"} err="failed to get container status \"cbb2e4af4dfe7b67318e6b2ea37866fc775ccb5655bd798201601e71eecb980f\": rpc error: code = NotFound desc = could not find container \"cbb2e4af4dfe7b67318e6b2ea37866fc775ccb5655bd798201601e71eecb980f\": container with ID starting with cbb2e4af4dfe7b67318e6b2ea37866fc775ccb5655bd798201601e71eecb980f not found: ID does not exist"
Sep 17 16:40:28.955225 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:28.955214    2585 scope.go:117] "RemoveContainer" containerID="6f7bbb966bce01c4cc5c26c666eedfcb7cb3dd96cf91f9b1f3a3bd3fb5ef54ed"
Sep 17 16:40:28.955492 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:40:28.955473    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f7bbb966bce01c4cc5c26c666eedfcb7cb3dd96cf91f9b1f3a3bd3fb5ef54ed\": container with ID starting with 6f7bbb966bce01c4cc5c26c666eedfcb7cb3dd96cf91f9b1f3a3bd3fb5ef54ed not found: ID does not exist" containerID="6f7bbb966bce01c4cc5c26c666eedfcb7cb3dd96cf91f9b1f3a3bd3fb5ef54ed"
Sep 17 16:40:28.955602 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:28.955493    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f7bbb966bce01c4cc5c26c666eedfcb7cb3dd96cf91f9b1f3a3bd3fb5ef54ed"} err="failed to get container status \"6f7bbb966bce01c4cc5c26c666eedfcb7cb3dd96cf91f9b1f3a3bd3fb5ef54ed\": rpc error: code = NotFound desc = could not find container \"6f7bbb966bce01c4cc5c26c666eedfcb7cb3dd96cf91f9b1f3a3bd3fb5ef54ed\": container with ID starting with 6f7bbb966bce01c4cc5c26c666eedfcb7cb3dd96cf91f9b1f3a3bd3fb5ef54ed not found: ID does not exist"
Sep 17 16:40:28.955602 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:28.955508    2585 scope.go:117] "RemoveContainer" containerID="c21f0ba249c74df1525692c1ea69f65cea476d6abb4ecffd668705096a3967f8"
Sep 17 16:40:28.955827 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:40:28.955805    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c21f0ba249c74df1525692c1ea69f65cea476d6abb4ecffd668705096a3967f8\": container with ID starting with c21f0ba249c74df1525692c1ea69f65cea476d6abb4ecffd668705096a3967f8 not found: ID does not exist" containerID="c21f0ba249c74df1525692c1ea69f65cea476d6abb4ecffd668705096a3967f8"
Sep 17 16:40:28.955869 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:28.955834    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c21f0ba249c74df1525692c1ea69f65cea476d6abb4ecffd668705096a3967f8"} err="failed to get container status \"c21f0ba249c74df1525692c1ea69f65cea476d6abb4ecffd668705096a3967f8\": rpc error: code = NotFound desc = could not find container \"c21f0ba249c74df1525692c1ea69f65cea476d6abb4ecffd668705096a3967f8\": container with ID starting with c21f0ba249c74df1525692c1ea69f65cea476d6abb4ecffd668705096a3967f8 not found: ID does not exist"
Sep 17 16:40:28.989311 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:28.989222    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp"]
Sep 17 16:40:28.994017 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:28.993987    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-d26rp"]
Sep 17 16:40:30.520352 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:30.515761    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f" path="/var/lib/kubelet/pods/e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f/volumes"
Sep 17 16:40:31.780235 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:31.780197    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-db86d8dd4-66krn" podUID="5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 17 16:40:32.950369 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:32.950336    2585 generic.go:356] "Generic (PLEG): container finished" podID="530388b1-7699-4531-860d-b422b7b544c7" containerID="9b3ffc761074f6d6edf75658d25c6d9a3dac1b885e097e2e6443f8112953838a" exitCode=0
Sep 17 16:40:32.950800 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:32.950439    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s6ffb" event={"ID":"530388b1-7699-4531-860d-b422b7b544c7","Type":"ContainerDied","Data":"9b3ffc761074f6d6edf75658d25c6d9a3dac1b885e097e2e6443f8112953838a"}
Sep 17 16:40:33.958490 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:33.958452    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s6ffb" event={"ID":"530388b1-7699-4531-860d-b422b7b544c7","Type":"ContainerStarted","Data":"e476e5b83da1d3a46bf330b4a504ea48dead7b718907c2aeab45a7d83195d5ea"}
Sep 17 16:40:33.958888 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:33.958516    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s6ffb" event={"ID":"530388b1-7699-4531-860d-b422b7b544c7","Type":"ContainerStarted","Data":"dd70288501112300290b35bf56aaaa2b22bbcad717521cdf6711a18f929c3df2"}
Sep 17 16:40:33.958888 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:33.958722    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s6ffb"
Sep 17 16:40:33.958888 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:33.958769    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s6ffb"
Sep 17 16:40:33.982518 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:33.982457    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s6ffb" podStartSLOduration=3.205348321 podStartE2EDuration="6.982440365s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.777092044s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:40:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:40:33.981232321 +0000 UTC m=+2789.751564130" watchObservedRunningTime="2026-09-17 16:40:33.982440365 +0000 UTC m=+2789.752772161"
Sep 17 16:40:39.968957 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:39.968926    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s6ffb"
Sep 17 16:40:41.780654 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:41.780615    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-db86d8dd4-66krn" podUID="5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 17 16:40:51.780305 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:40:51.780260    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-db86d8dd4-66krn" podUID="5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 17 16:41:01.780903 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:01.780862    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-db86d8dd4-66krn" podUID="5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 17 16:41:09.973028 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:09.972994    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s6ffb"
Sep 17 16:41:11.780586 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:11.780550    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-db86d8dd4-66krn"
Sep 17 16:41:17.785142 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:17.785101    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s6ffb"]
Sep 17 16:41:17.785744 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:17.785402    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s6ffb" podUID="530388b1-7699-4531-860d-b422b7b544c7" containerName="kserve-container" containerID="cri-o://dd70288501112300290b35bf56aaaa2b22bbcad717521cdf6711a18f929c3df2" gracePeriod=30
Sep 17 16:41:17.785744 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:17.785453    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s6ffb" podUID="530388b1-7699-4531-860d-b422b7b544c7" containerName="kube-rbac-proxy" containerID="cri-o://e476e5b83da1d3a46bf330b4a504ea48dead7b718907c2aeab45a7d83195d5ea" gracePeriod=30
Sep 17 16:41:18.169348 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:18.169256    2585 generic.go:356] "Generic (PLEG): container finished" podID="530388b1-7699-4531-860d-b422b7b544c7" containerID="e476e5b83da1d3a46bf330b4a504ea48dead7b718907c2aeab45a7d83195d5ea" exitCode=2
Sep 17 16:41:18.169348 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:18.169335    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s6ffb" event={"ID":"530388b1-7699-4531-860d-b422b7b544c7","Type":"ContainerDied","Data":"e476e5b83da1d3a46bf330b4a504ea48dead7b718907c2aeab45a7d83195d5ea"}
Sep 17 16:41:18.939954 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:18.939920    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s6ffb"
Sep 17 16:41:19.048669 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:19.048578    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/530388b1-7699-4531-860d-b422b7b544c7-kserve-provision-location\") pod \"530388b1-7699-4531-860d-b422b7b544c7\" (UID: \"530388b1-7699-4531-860d-b422b7b544c7\") "
Sep 17 16:41:19.048669 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:19.048625    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvdn9\" (UniqueName: \"kubernetes.io/projected/530388b1-7699-4531-860d-b422b7b544c7-kube-api-access-kvdn9\") pod \"530388b1-7699-4531-860d-b422b7b544c7\" (UID: \"530388b1-7699-4531-860d-b422b7b544c7\") "
Sep 17 16:41:19.048669 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:19.048652    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/530388b1-7699-4531-860d-b422b7b544c7-kube-rbac-proxy-sar-config\") pod \"530388b1-7699-4531-860d-b422b7b544c7\" (UID: \"530388b1-7699-4531-860d-b422b7b544c7\") "
Sep 17 16:41:19.048919 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:19.048678    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/530388b1-7699-4531-860d-b422b7b544c7-proxy-tls\") pod \"530388b1-7699-4531-860d-b422b7b544c7\" (UID: \"530388b1-7699-4531-860d-b422b7b544c7\") "
Sep 17 16:41:19.048919 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:19.048705    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/530388b1-7699-4531-860d-b422b7b544c7-cabundle-cert\") pod \"530388b1-7699-4531-860d-b422b7b544c7\" (UID: \"530388b1-7699-4531-860d-b422b7b544c7\") "
Sep 17 16:41:19.049117 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:19.049089    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/530388b1-7699-4531-860d-b422b7b544c7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "530388b1-7699-4531-860d-b422b7b544c7" (UID: "530388b1-7699-4531-860d-b422b7b544c7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:41:19.049178 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:19.049087    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/530388b1-7699-4531-860d-b422b7b544c7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "530388b1-7699-4531-860d-b422b7b544c7" (UID: "530388b1-7699-4531-860d-b422b7b544c7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:41:19.049218 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:19.049197    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/530388b1-7699-4531-860d-b422b7b544c7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "530388b1-7699-4531-860d-b422b7b544c7" (UID: "530388b1-7699-4531-860d-b422b7b544c7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:41:19.050892 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:19.050869    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/530388b1-7699-4531-860d-b422b7b544c7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "530388b1-7699-4531-860d-b422b7b544c7" (UID: "530388b1-7699-4531-860d-b422b7b544c7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:41:19.051000 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:19.050918    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/530388b1-7699-4531-860d-b422b7b544c7-kube-api-access-kvdn9" (OuterVolumeSpecName: "kube-api-access-kvdn9") pod "530388b1-7699-4531-860d-b422b7b544c7" (UID: "530388b1-7699-4531-860d-b422b7b544c7"). InnerVolumeSpecName "kube-api-access-kvdn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:41:19.149505 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:19.149467    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/530388b1-7699-4531-860d-b422b7b544c7-cabundle-cert\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:41:19.149505 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:19.149498    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/530388b1-7699-4531-860d-b422b7b544c7-kserve-provision-location\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:41:19.149505 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:19.149507    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvdn9\" (UniqueName: \"kubernetes.io/projected/530388b1-7699-4531-860d-b422b7b544c7-kube-api-access-kvdn9\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:41:19.149505 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:19.149515    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/530388b1-7699-4531-860d-b422b7b544c7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:41:19.149781 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:19.149524    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/530388b1-7699-4531-860d-b422b7b544c7-proxy-tls\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:41:19.176443 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:19.176393    2585 generic.go:356] "Generic (PLEG): container finished" podID="530388b1-7699-4531-860d-b422b7b544c7" containerID="dd70288501112300290b35bf56aaaa2b22bbcad717521cdf6711a18f929c3df2" exitCode=0
Sep 17 16:41:19.176627 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:19.176472    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s6ffb"
Sep 17 16:41:19.176627 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:19.176475    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s6ffb" event={"ID":"530388b1-7699-4531-860d-b422b7b544c7","Type":"ContainerDied","Data":"dd70288501112300290b35bf56aaaa2b22bbcad717521cdf6711a18f929c3df2"}
Sep 17 16:41:19.176627 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:19.176537    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s6ffb" event={"ID":"530388b1-7699-4531-860d-b422b7b544c7","Type":"ContainerDied","Data":"e541bc501066efbde53f8aedc7ce912a822110b3142208fe4145d8a94c3b0c75"}
Sep 17 16:41:19.176627 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:19.176592    2585 scope.go:117] "RemoveContainer" containerID="e476e5b83da1d3a46bf330b4a504ea48dead7b718907c2aeab45a7d83195d5ea"
Sep 17 16:41:19.186340 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:19.186321    2585 scope.go:117] "RemoveContainer" containerID="dd70288501112300290b35bf56aaaa2b22bbcad717521cdf6711a18f929c3df2"
Sep 17 16:41:19.194461 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:19.194437    2585 scope.go:117] "RemoveContainer" containerID="9b3ffc761074f6d6edf75658d25c6d9a3dac1b885e097e2e6443f8112953838a"
Sep 17 16:41:19.202131 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:19.202067    2585 scope.go:117] "RemoveContainer" containerID="e476e5b83da1d3a46bf330b4a504ea48dead7b718907c2aeab45a7d83195d5ea"
Sep 17 16:41:19.202632 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:41:19.202600    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e476e5b83da1d3a46bf330b4a504ea48dead7b718907c2aeab45a7d83195d5ea\": container with ID starting with e476e5b83da1d3a46bf330b4a504ea48dead7b718907c2aeab45a7d83195d5ea not found: ID does not exist" containerID="e476e5b83da1d3a46bf330b4a504ea48dead7b718907c2aeab45a7d83195d5ea"
Sep 17 16:41:19.202779 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:19.202644    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e476e5b83da1d3a46bf330b4a504ea48dead7b718907c2aeab45a7d83195d5ea"} err="failed to get container status \"e476e5b83da1d3a46bf330b4a504ea48dead7b718907c2aeab45a7d83195d5ea\": rpc error: code = NotFound desc = could not find container \"e476e5b83da1d3a46bf330b4a504ea48dead7b718907c2aeab45a7d83195d5ea\": container with ID starting with e476e5b83da1d3a46bf330b4a504ea48dead7b718907c2aeab45a7d83195d5ea not found: ID does not exist"
Sep 17 16:41:19.202779 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:19.202669    2585 scope.go:117] "RemoveContainer" containerID="dd70288501112300290b35bf56aaaa2b22bbcad717521cdf6711a18f929c3df2"
Sep 17 16:41:19.203139 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:41:19.203111    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd70288501112300290b35bf56aaaa2b22bbcad717521cdf6711a18f929c3df2\": container with ID starting with dd70288501112300290b35bf56aaaa2b22bbcad717521cdf6711a18f929c3df2 not found: ID does not exist" containerID="dd70288501112300290b35bf56aaaa2b22bbcad717521cdf6711a18f929c3df2"
Sep 17 16:41:19.203276 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:19.203146    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd70288501112300290b35bf56aaaa2b22bbcad717521cdf6711a18f929c3df2"} err="failed to get container status \"dd70288501112300290b35bf56aaaa2b22bbcad717521cdf6711a18f929c3df2\": rpc error: code = NotFound desc = could not find container \"dd70288501112300290b35bf56aaaa2b22bbcad717521cdf6711a18f929c3df2\": container with ID starting with dd70288501112300290b35bf56aaaa2b22bbcad717521cdf6711a18f929c3df2 not found: ID does not exist"
Sep 17 16:41:19.203476 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:19.203455    2585 scope.go:117] "RemoveContainer" containerID="9b3ffc761074f6d6edf75658d25c6d9a3dac1b885e097e2e6443f8112953838a"
Sep 17 16:41:19.203796 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:41:19.203775    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b3ffc761074f6d6edf75658d25c6d9a3dac1b885e097e2e6443f8112953838a\": container with ID starting with 9b3ffc761074f6d6edf75658d25c6d9a3dac1b885e097e2e6443f8112953838a not found: ID does not exist" containerID="9b3ffc761074f6d6edf75658d25c6d9a3dac1b885e097e2e6443f8112953838a"
Sep 17 16:41:19.203883 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:19.203801    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b3ffc761074f6d6edf75658d25c6d9a3dac1b885e097e2e6443f8112953838a"} err="failed to get container status \"9b3ffc761074f6d6edf75658d25c6d9a3dac1b885e097e2e6443f8112953838a\": rpc error: code = NotFound desc = could not find container \"9b3ffc761074f6d6edf75658d25c6d9a3dac1b885e097e2e6443f8112953838a\": container with ID starting with 9b3ffc761074f6d6edf75658d25c6d9a3dac1b885e097e2e6443f8112953838a not found: ID does not exist"
Sep 17 16:41:19.205300 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:19.205278    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s6ffb"]
Sep 17 16:41:19.213127 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:19.213104    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-s6ffb"]
Sep 17 16:41:20.455111 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:20.455075    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-db86d8dd4-66krn"]
Sep 17 16:41:20.455576 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:20.455324    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-db86d8dd4-66krn" podUID="5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2" containerName="kserve-container" containerID="cri-o://64f4aef979594c484782c732b4f51da76732335878366acdf56ce7125aceedf5" gracePeriod=30
Sep 17 16:41:20.455576 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:20.455346    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-db86d8dd4-66krn" podUID="5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2" containerName="kube-rbac-proxy" containerID="cri-o://a903df8b7c6963507f277d8f1e4ed9fd38ae7f00b921492c93a2d67a24bbec96" gracePeriod=30
Sep 17 16:41:20.512808 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:20.512773    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="530388b1-7699-4531-860d-b422b7b544c7" path="/var/lib/kubelet/pods/530388b1-7699-4531-860d-b422b7b544c7/volumes"
Sep 17 16:41:20.641480 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:20.641439    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd"]
Sep 17 16:41:20.641915 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:20.641885    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f" containerName="kube-rbac-proxy"
Sep 17 16:41:20.641915 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:20.641909    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f" containerName="kube-rbac-proxy"
Sep 17 16:41:20.642088 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:20.641936    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="530388b1-7699-4531-860d-b422b7b544c7" containerName="kube-rbac-proxy"
Sep 17 16:41:20.642088 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:20.641942    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="530388b1-7699-4531-860d-b422b7b544c7" containerName="kube-rbac-proxy"
Sep 17 16:41:20.642088 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:20.641956    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f" containerName="kserve-container"
Sep 17 16:41:20.642088 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:20.641969    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f" containerName="kserve-container"
Sep 17 16:41:20.642088 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:20.641979    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f" containerName="storage-initializer"
Sep 17 16:41:20.642088 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:20.641984    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f" containerName="storage-initializer"
Sep 17 16:41:20.642088 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:20.641991    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="530388b1-7699-4531-860d-b422b7b544c7" containerName="storage-initializer"
Sep 17 16:41:20.642088 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:20.641996    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="530388b1-7699-4531-860d-b422b7b544c7" containerName="storage-initializer"
Sep 17 16:41:20.642088 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:20.642005    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="530388b1-7699-4531-860d-b422b7b544c7" containerName="kserve-container"
Sep 17 16:41:20.642088 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:20.642010    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="530388b1-7699-4531-860d-b422b7b544c7" containerName="kserve-container"
Sep 17 16:41:20.642088 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:20.642078    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f" containerName="kube-rbac-proxy"
Sep 17 16:41:20.642088 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:20.642085    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0c3d4be-dca6-4ccc-b0a5-f81e5a75349f" containerName="kserve-container"
Sep 17 16:41:20.642088 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:20.642091    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="530388b1-7699-4531-860d-b422b7b544c7" containerName="kserve-container"
Sep 17 16:41:20.642557 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:20.642100    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="530388b1-7699-4531-860d-b422b7b544c7" containerName="kube-rbac-proxy"
Sep 17 16:41:20.645367 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:20.645347    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd"
Sep 17 16:41:20.650913 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:20.650887    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
Sep 17 16:41:20.651069 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:20.650902    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
Sep 17 16:41:20.651156 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:20.651071    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:41:20.668451 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:20.663196    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd"]
Sep 17 16:41:20.763737 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:20.763646    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5abde47-a7e5-4a2c-8726-b6eba48a54b6-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd\" (UID: \"f5abde47-a7e5-4a2c-8726-b6eba48a54b6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd"
Sep 17 16:41:20.763737 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:20.763691    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhqmx\" (UniqueName: \"kubernetes.io/projected/f5abde47-a7e5-4a2c-8726-b6eba48a54b6-kube-api-access-nhqmx\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd\" (UID: \"f5abde47-a7e5-4a2c-8726-b6eba48a54b6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd"
Sep 17 16:41:20.763737 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:20.763712    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5abde47-a7e5-4a2c-8726-b6eba48a54b6-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd\" (UID: \"f5abde47-a7e5-4a2c-8726-b6eba48a54b6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd"
Sep 17 16:41:20.763737 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:20.763730    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5abde47-a7e5-4a2c-8726-b6eba48a54b6-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd\" (UID: \"f5abde47-a7e5-4a2c-8726-b6eba48a54b6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd"
Sep 17 16:41:20.763996 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:20.763788    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5abde47-a7e5-4a2c-8726-b6eba48a54b6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd\" (UID: \"f5abde47-a7e5-4a2c-8726-b6eba48a54b6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd"
Sep 17 16:41:20.865115 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:20.865074    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5abde47-a7e5-4a2c-8726-b6eba48a54b6-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd\" (UID: \"f5abde47-a7e5-4a2c-8726-b6eba48a54b6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd"
Sep 17 16:41:20.865115 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:20.865119    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhqmx\" (UniqueName: \"kubernetes.io/projected/f5abde47-a7e5-4a2c-8726-b6eba48a54b6-kube-api-access-nhqmx\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd\" (UID: \"f5abde47-a7e5-4a2c-8726-b6eba48a54b6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd"
Sep 17 16:41:20.865387 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:20.865139    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5abde47-a7e5-4a2c-8726-b6eba48a54b6-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd\" (UID: \"f5abde47-a7e5-4a2c-8726-b6eba48a54b6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd"
Sep 17 16:41:20.865387 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:20.865160    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5abde47-a7e5-4a2c-8726-b6eba48a54b6-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd\" (UID: \"f5abde47-a7e5-4a2c-8726-b6eba48a54b6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd"
Sep 17 16:41:20.865387 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:20.865177    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5abde47-a7e5-4a2c-8726-b6eba48a54b6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd\" (UID: \"f5abde47-a7e5-4a2c-8726-b6eba48a54b6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd"
Sep 17 16:41:20.865387 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:41:20.865296    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-global-pass-predictor-serving-cert" not found
Sep 17 16:41:20.865387 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:41:20.865391    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5abde47-a7e5-4a2c-8726-b6eba48a54b6-proxy-tls podName:f5abde47-a7e5-4a2c-8726-b6eba48a54b6 nodeName:}" failed. No retries permitted until 2026-09-17 16:41:21.365366137 +0000 UTC m=+2837.135697914 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f5abde47-a7e5-4a2c-8726-b6eba48a54b6-proxy-tls") pod "isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd" (UID: "f5abde47-a7e5-4a2c-8726-b6eba48a54b6") : secret "isvc-sklearn-s3-tls-global-pass-predictor-serving-cert" not found
Sep 17 16:41:20.865727 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:20.865702    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5abde47-a7e5-4a2c-8726-b6eba48a54b6-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd\" (UID: \"f5abde47-a7e5-4a2c-8726-b6eba48a54b6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd"
Sep 17 16:41:20.865941 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:20.865915    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5abde47-a7e5-4a2c-8726-b6eba48a54b6-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd\" (UID: \"f5abde47-a7e5-4a2c-8726-b6eba48a54b6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd"
Sep 17 16:41:20.866036 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:20.865955    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5abde47-a7e5-4a2c-8726-b6eba48a54b6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd\" (UID: \"f5abde47-a7e5-4a2c-8726-b6eba48a54b6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd"
Sep 17 16:41:20.877209 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:20.877180    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhqmx\" (UniqueName: \"kubernetes.io/projected/f5abde47-a7e5-4a2c-8726-b6eba48a54b6-kube-api-access-nhqmx\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd\" (UID: \"f5abde47-a7e5-4a2c-8726-b6eba48a54b6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd"
Sep 17 16:41:21.191510 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:21.191403    2585 generic.go:356] "Generic (PLEG): container finished" podID="5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2" containerID="a903df8b7c6963507f277d8f1e4ed9fd38ae7f00b921492c93a2d67a24bbec96" exitCode=2
Sep 17 16:41:21.191510 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:21.191480    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-db86d8dd4-66krn" event={"ID":"5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2","Type":"ContainerDied","Data":"a903df8b7c6963507f277d8f1e4ed9fd38ae7f00b921492c93a2d67a24bbec96"}
Sep 17 16:41:21.368922 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:21.368875    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5abde47-a7e5-4a2c-8726-b6eba48a54b6-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd\" (UID: \"f5abde47-a7e5-4a2c-8726-b6eba48a54b6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd"
Sep 17 16:41:21.371447 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:21.371394    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5abde47-a7e5-4a2c-8726-b6eba48a54b6-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd\" (UID: \"f5abde47-a7e5-4a2c-8726-b6eba48a54b6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd"
Sep 17 16:41:21.556156 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:21.556111    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd"
Sep 17 16:41:21.775682 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:21.775639    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-db86d8dd4-66krn" podUID="5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.57:8643/healthz\": dial tcp 10.134.0.57:8643: connect: connection refused"
Sep 17 16:41:21.780035 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:21.780007    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-db86d8dd4-66krn" podUID="5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 17 16:41:21.839537 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:21.839448    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd"]
Sep 17 16:41:21.844784 ip-10-0-135-43 kubenswrapper[2585]: W0917 16:41:21.844753    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5abde47_a7e5_4a2c_8726_b6eba48a54b6.slice/crio-add57e155c853057b837f591976e4360d396c4bfdccc0212102353040baa2f1a WatchSource:0}: Error finding container add57e155c853057b837f591976e4360d396c4bfdccc0212102353040baa2f1a: Status 404 returned error can't find the container with id add57e155c853057b837f591976e4360d396c4bfdccc0212102353040baa2f1a
Sep 17 16:41:22.197986 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:22.197949    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd" event={"ID":"f5abde47-a7e5-4a2c-8726-b6eba48a54b6","Type":"ContainerStarted","Data":"4577e0d61b961ab8405baaaba4bdbab6b989d5912fdcc0cd61458712a8119923"}
Sep 17 16:41:22.198161 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:22.198018    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd" event={"ID":"f5abde47-a7e5-4a2c-8726-b6eba48a54b6","Type":"ContainerStarted","Data":"add57e155c853057b837f591976e4360d396c4bfdccc0212102353040baa2f1a"}
Sep 17 16:41:23.204193 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:23.204159    2585 generic.go:356] "Generic (PLEG): container finished" podID="f5abde47-a7e5-4a2c-8726-b6eba48a54b6" containerID="4577e0d61b961ab8405baaaba4bdbab6b989d5912fdcc0cd61458712a8119923" exitCode=0
Sep 17 16:41:23.204687 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:23.204198    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd" event={"ID":"f5abde47-a7e5-4a2c-8726-b6eba48a54b6","Type":"ContainerDied","Data":"4577e0d61b961ab8405baaaba4bdbab6b989d5912fdcc0cd61458712a8119923"}
Sep 17 16:41:24.211123 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:24.211090    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd" event={"ID":"f5abde47-a7e5-4a2c-8726-b6eba48a54b6","Type":"ContainerStarted","Data":"078884b492c64828e868152eb2d49943d5d67486f7c8714a9e23e3e8f457ca45"}
Sep 17 16:41:24.213517 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:24.211156    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd" event={"ID":"f5abde47-a7e5-4a2c-8726-b6eba48a54b6","Type":"ContainerStarted","Data":"a8531bbf6dbe54c0108ff1d30a68e331005a9237845ab52bb6a578e080cccd3c"}
Sep 17 16:41:24.213517 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:24.211308    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd"
Sep 17 16:41:24.213517 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:24.211379    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd"
Sep 17 16:41:24.213517 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:24.212823    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd" podUID="f5abde47-a7e5-4a2c-8726-b6eba48a54b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 17 16:41:24.233707 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:24.233656    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd" podStartSLOduration=2.999329388 podStartE2EDuration="4.23364078s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.234311392s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:41:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:41:24.232109872 +0000 UTC m=+2840.002441718" watchObservedRunningTime="2026-09-17 16:41:24.23364078 +0000 UTC m=+2840.003972636"
Sep 17 16:41:25.224274 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:25.224221    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd" podUID="f5abde47-a7e5-4a2c-8726-b6eba48a54b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 17 16:41:25.803627 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:25.803597    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-db86d8dd4-66krn"
Sep 17 16:41:25.908207 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:25.908096    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2-proxy-tls\") pod \"5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2\" (UID: \"5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2\") "
Sep 17 16:41:25.908207 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:25.908194    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2-kube-rbac-proxy-sar-config\") pod \"5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2\" (UID: \"5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2\") "
Sep 17 16:41:25.908495 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:25.908245    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2-kserve-provision-location\") pod \"5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2\" (UID: \"5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2\") "
Sep 17 16:41:25.908495 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:25.908312    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2knnz\" (UniqueName: \"kubernetes.io/projected/5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2-kube-api-access-2knnz\") pod \"5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2\" (UID: \"5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2\") "
Sep 17 16:41:25.908615 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:25.908582    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2" (UID: "5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:41:25.908655 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:25.908606    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2" (UID: "5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:41:25.910313 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:25.910283    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2" (UID: "5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:41:25.910430 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:25.910402    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2-kube-api-access-2knnz" (OuterVolumeSpecName: "kube-api-access-2knnz") pod "5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2" (UID: "5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2"). InnerVolumeSpecName "kube-api-access-2knnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:41:26.008982 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:26.008945    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:41:26.008982 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:26.008975    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2-kserve-provision-location\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:41:26.008982 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:26.008984    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2knnz\" (UniqueName: \"kubernetes.io/projected/5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2-kube-api-access-2knnz\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:41:26.009235 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:26.008997    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2-proxy-tls\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:41:26.230131 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:26.230098    2585 generic.go:356] "Generic (PLEG): container finished" podID="5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2" containerID="64f4aef979594c484782c732b4f51da76732335878366acdf56ce7125aceedf5" exitCode=0
Sep 17 16:41:26.230628 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:26.230146    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-db86d8dd4-66krn"
Sep 17 16:41:26.230628 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:26.230145    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-db86d8dd4-66krn" event={"ID":"5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2","Type":"ContainerDied","Data":"64f4aef979594c484782c732b4f51da76732335878366acdf56ce7125aceedf5"}
Sep 17 16:41:26.230628 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:26.230322    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-db86d8dd4-66krn" event={"ID":"5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2","Type":"ContainerDied","Data":"11a538fffbcd079c1361f59b7841e506f2ecf6bd05217f91931a116baa33c17a"}
Sep 17 16:41:26.230628 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:26.230371    2585 scope.go:117] "RemoveContainer" containerID="a903df8b7c6963507f277d8f1e4ed9fd38ae7f00b921492c93a2d67a24bbec96"
Sep 17 16:41:26.239671 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:26.239649    2585 scope.go:117] "RemoveContainer" containerID="64f4aef979594c484782c732b4f51da76732335878366acdf56ce7125aceedf5"
Sep 17 16:41:26.249385 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:26.249366    2585 scope.go:117] "RemoveContainer" containerID="d821c7532312db8bb4a9805fed80ba08456571165c5cb1fa302a5e7e2e705bde"
Sep 17 16:41:26.257195 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:26.257175    2585 scope.go:117] "RemoveContainer" containerID="a903df8b7c6963507f277d8f1e4ed9fd38ae7f00b921492c93a2d67a24bbec96"
Sep 17 16:41:26.257469 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:41:26.257449    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a903df8b7c6963507f277d8f1e4ed9fd38ae7f00b921492c93a2d67a24bbec96\": container with ID starting with a903df8b7c6963507f277d8f1e4ed9fd38ae7f00b921492c93a2d67a24bbec96 not found: ID does not exist" containerID="a903df8b7c6963507f277d8f1e4ed9fd38ae7f00b921492c93a2d67a24bbec96"
Sep 17 16:41:26.257552 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:26.257483    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a903df8b7c6963507f277d8f1e4ed9fd38ae7f00b921492c93a2d67a24bbec96"} err="failed to get container status \"a903df8b7c6963507f277d8f1e4ed9fd38ae7f00b921492c93a2d67a24bbec96\": rpc error: code = NotFound desc = could not find container \"a903df8b7c6963507f277d8f1e4ed9fd38ae7f00b921492c93a2d67a24bbec96\": container with ID starting with a903df8b7c6963507f277d8f1e4ed9fd38ae7f00b921492c93a2d67a24bbec96 not found: ID does not exist"
Sep 17 16:41:26.257552 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:26.257509    2585 scope.go:117] "RemoveContainer" containerID="64f4aef979594c484782c732b4f51da76732335878366acdf56ce7125aceedf5"
Sep 17 16:41:26.257800 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:41:26.257782    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64f4aef979594c484782c732b4f51da76732335878366acdf56ce7125aceedf5\": container with ID starting with 64f4aef979594c484782c732b4f51da76732335878366acdf56ce7125aceedf5 not found: ID does not exist" containerID="64f4aef979594c484782c732b4f51da76732335878366acdf56ce7125aceedf5"
Sep 17 16:41:26.257843 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:26.257807    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64f4aef979594c484782c732b4f51da76732335878366acdf56ce7125aceedf5"} err="failed to get container status \"64f4aef979594c484782c732b4f51da76732335878366acdf56ce7125aceedf5\": rpc error: code = NotFound desc = could not find container \"64f4aef979594c484782c732b4f51da76732335878366acdf56ce7125aceedf5\": container with ID starting with 64f4aef979594c484782c732b4f51da76732335878366acdf56ce7125aceedf5 not found: ID does not exist"
Sep 17 16:41:26.257843 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:26.257821    2585 scope.go:117] "RemoveContainer" containerID="d821c7532312db8bb4a9805fed80ba08456571165c5cb1fa302a5e7e2e705bde"
Sep 17 16:41:26.258079 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:41:26.258062    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d821c7532312db8bb4a9805fed80ba08456571165c5cb1fa302a5e7e2e705bde\": container with ID starting with d821c7532312db8bb4a9805fed80ba08456571165c5cb1fa302a5e7e2e705bde not found: ID does not exist" containerID="d821c7532312db8bb4a9805fed80ba08456571165c5cb1fa302a5e7e2e705bde"
Sep 17 16:41:26.258122 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:26.258086    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d821c7532312db8bb4a9805fed80ba08456571165c5cb1fa302a5e7e2e705bde"} err="failed to get container status \"d821c7532312db8bb4a9805fed80ba08456571165c5cb1fa302a5e7e2e705bde\": rpc error: code = NotFound desc = could not find container \"d821c7532312db8bb4a9805fed80ba08456571165c5cb1fa302a5e7e2e705bde\": container with ID starting with d821c7532312db8bb4a9805fed80ba08456571165c5cb1fa302a5e7e2e705bde not found: ID does not exist"
Sep 17 16:41:26.264242 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:26.264218    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-db86d8dd4-66krn"]
Sep 17 16:41:26.277853 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:26.277825    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-db86d8dd4-66krn"]
Sep 17 16:41:26.512325 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:26.512233    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2" path="/var/lib/kubelet/pods/5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2/volumes"
Sep 17 16:41:30.228660 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:30.228628    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd"
Sep 17 16:41:30.229206 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:30.229174    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd" podUID="f5abde47-a7e5-4a2c-8726-b6eba48a54b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 17 16:41:40.229487 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:40.229441    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd" podUID="f5abde47-a7e5-4a2c-8726-b6eba48a54b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 17 16:41:50.229355 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:50.229257    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd" podUID="f5abde47-a7e5-4a2c-8726-b6eba48a54b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 17 16:41:58.246228 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:58.246189    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pl42n"]
Sep 17 16:41:58.246653 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:58.246546    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2" containerName="kube-rbac-proxy"
Sep 17 16:41:58.246653 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:58.246558    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2" containerName="kube-rbac-proxy"
Sep 17 16:41:58.246653 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:58.246581    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2" containerName="kserve-container"
Sep 17 16:41:58.246653 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:58.246587    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2" containerName="kserve-container"
Sep 17 16:41:58.246653 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:58.246595    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2" containerName="storage-initializer"
Sep 17 16:41:58.246653 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:58.246601    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2" containerName="storage-initializer"
Sep 17 16:41:58.246653 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:58.246653    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2" containerName="kserve-container"
Sep 17 16:41:58.246880 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:58.246660    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c7dc87c-41fc-4000-97ad-d3a09d9ae8c2" containerName="kube-rbac-proxy"
Sep 17 16:41:58.250080 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:58.250056    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pl42n"
Sep 17 16:41:58.254410 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:58.254388    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
Sep 17 16:41:58.254550 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:58.254406    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 16:41:58.255664 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:58.255638    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 17 16:41:58.270694 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:58.270663    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pl42n"]
Sep 17 16:41:58.394530 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:58.394494    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/021304af-7a6f-4a9d-b654-7541e237feb5-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-pl42n\" (UID: \"021304af-7a6f-4a9d-b654-7541e237feb5\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pl42n"
Sep 17 16:41:58.394729 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:58.394557    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/021304af-7a6f-4a9d-b654-7541e237feb5-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-pl42n\" (UID: \"021304af-7a6f-4a9d-b654-7541e237feb5\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pl42n"
Sep 17 16:41:58.394729 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:58.394589    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/021304af-7a6f-4a9d-b654-7541e237feb5-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-pl42n\" (UID: \"021304af-7a6f-4a9d-b654-7541e237feb5\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pl42n"
Sep 17 16:41:58.394729 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:58.394614    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbxkc\" (UniqueName: \"kubernetes.io/projected/021304af-7a6f-4a9d-b654-7541e237feb5-kube-api-access-dbxkc\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-pl42n\" (UID: \"021304af-7a6f-4a9d-b654-7541e237feb5\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pl42n"
Sep 17 16:41:58.394729 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:58.394635    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/021304af-7a6f-4a9d-b654-7541e237feb5-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-pl42n\" (UID: \"021304af-7a6f-4a9d-b654-7541e237feb5\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pl42n"
Sep 17 16:41:58.495885 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:58.495849    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/021304af-7a6f-4a9d-b654-7541e237feb5-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-pl42n\" (UID: \"021304af-7a6f-4a9d-b654-7541e237feb5\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pl42n"
Sep 17 16:41:58.496069 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:58.495895    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbxkc\" (UniqueName: \"kubernetes.io/projected/021304af-7a6f-4a9d-b654-7541e237feb5-kube-api-access-dbxkc\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-pl42n\" (UID: \"021304af-7a6f-4a9d-b654-7541e237feb5\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pl42n"
Sep 17 16:41:58.496069 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:58.496011    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/021304af-7a6f-4a9d-b654-7541e237feb5-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-pl42n\" (UID: \"021304af-7a6f-4a9d-b654-7541e237feb5\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pl42n"
Sep 17 16:41:58.496167 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:58.496078    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/021304af-7a6f-4a9d-b654-7541e237feb5-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-pl42n\" (UID: \"021304af-7a6f-4a9d-b654-7541e237feb5\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pl42n"
Sep 17 16:41:58.496208 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:58.496162    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/021304af-7a6f-4a9d-b654-7541e237feb5-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-pl42n\" (UID: \"021304af-7a6f-4a9d-b654-7541e237feb5\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pl42n"
Sep 17 16:41:58.496607 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:58.496536    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/021304af-7a6f-4a9d-b654-7541e237feb5-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-pl42n\" (UID: \"021304af-7a6f-4a9d-b654-7541e237feb5\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pl42n"
Sep 17 16:41:58.496739 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:58.496633    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/021304af-7a6f-4a9d-b654-7541e237feb5-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-pl42n\" (UID: \"021304af-7a6f-4a9d-b654-7541e237feb5\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pl42n"
Sep 17 16:41:58.496739 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:58.496711    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/021304af-7a6f-4a9d-b654-7541e237feb5-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-pl42n\" (UID: \"021304af-7a6f-4a9d-b654-7541e237feb5\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pl42n"
Sep 17 16:41:58.498573 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:58.498557    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/021304af-7a6f-4a9d-b654-7541e237feb5-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-pl42n\" (UID: \"021304af-7a6f-4a9d-b654-7541e237feb5\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pl42n"
Sep 17 16:41:58.507907 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:58.507884    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbxkc\" (UniqueName: \"kubernetes.io/projected/021304af-7a6f-4a9d-b654-7541e237feb5-kube-api-access-dbxkc\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-pl42n\" (UID: \"021304af-7a6f-4a9d-b654-7541e237feb5\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pl42n"
Sep 17 16:41:58.561327 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:58.561287    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pl42n"
Sep 17 16:41:58.851193 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:58.851112    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pl42n"]
Sep 17 16:41:58.854372 ip-10-0-135-43 kubenswrapper[2585]: W0917 16:41:58.854341    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod021304af_7a6f_4a9d_b654_7541e237feb5.slice/crio-c61f5af898155efcb0049982ef2e6260341ce1c6a87de23fa7c730002dbd79be WatchSource:0}: Error finding container c61f5af898155efcb0049982ef2e6260341ce1c6a87de23fa7c730002dbd79be: Status 404 returned error can't find the container with id c61f5af898155efcb0049982ef2e6260341ce1c6a87de23fa7c730002dbd79be
Sep 17 16:41:59.391904 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:59.391864    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pl42n" event={"ID":"021304af-7a6f-4a9d-b654-7541e237feb5","Type":"ContainerStarted","Data":"521f973f24832ac59194a19d934c4271ae63080e87a7fdf7615c840391a55916"}
Sep 17 16:41:59.392396 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:41:59.391938    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pl42n" event={"ID":"021304af-7a6f-4a9d-b654-7541e237feb5","Type":"ContainerStarted","Data":"c61f5af898155efcb0049982ef2e6260341ce1c6a87de23fa7c730002dbd79be"}
Sep 17 16:42:00.229110 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:00.229068    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd" podUID="f5abde47-a7e5-4a2c-8726-b6eba48a54b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 17 16:42:03.414503 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:03.414465    2585 generic.go:356] "Generic (PLEG): container finished" podID="021304af-7a6f-4a9d-b654-7541e237feb5" containerID="521f973f24832ac59194a19d934c4271ae63080e87a7fdf7615c840391a55916" exitCode=0
Sep 17 16:42:03.415082 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:03.414548    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pl42n" event={"ID":"021304af-7a6f-4a9d-b654-7541e237feb5","Type":"ContainerDied","Data":"521f973f24832ac59194a19d934c4271ae63080e87a7fdf7615c840391a55916"}
Sep 17 16:42:04.421591 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:04.421559    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pl42n" event={"ID":"021304af-7a6f-4a9d-b654-7541e237feb5","Type":"ContainerStarted","Data":"e51b0b4ce0f5a2656db65dfd20e829492959939c8a1a5a76ef94a8cac0bfaf45"}
Sep 17 16:42:04.422057 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:04.421617    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pl42n" event={"ID":"021304af-7a6f-4a9d-b654-7541e237feb5","Type":"ContainerStarted","Data":"c17976df0cfbbc0e486c4c497c004a63ea52a0ad54beb25a6d3eac8bd6fcf612"}
Sep 17 16:42:04.422057 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:04.421816    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pl42n"
Sep 17 16:42:04.422057 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:04.421864    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pl42n"
Sep 17 16:42:04.450361 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:04.450309    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pl42n" podStartSLOduration=2.167084022 podStartE2EDuration="6.450292076s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.283208054s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:41:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:42:04.448751472 +0000 UTC m=+2880.219083270" watchObservedRunningTime="2026-09-17 16:42:04.450292076 +0000 UTC m=+2880.220623874"
Sep 17 16:42:10.229340 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:10.229295    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd" podUID="f5abde47-a7e5-4a2c-8726-b6eba48a54b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 17 16:42:10.432254 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:10.432224    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pl42n"
Sep 17 16:42:20.229956 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:20.229918    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd" podUID="f5abde47-a7e5-4a2c-8726-b6eba48a54b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 17 16:42:30.230105 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:30.230064    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd" podUID="f5abde47-a7e5-4a2c-8726-b6eba48a54b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Sep 17 16:42:40.230408 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:40.230373    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd"
Sep 17 16:42:40.436265 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:40.436232    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pl42n"
Sep 17 16:42:48.269797 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:48.269760    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pl42n"]
Sep 17 16:42:48.270390 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:48.270043    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pl42n" podUID="021304af-7a6f-4a9d-b654-7541e237feb5" containerName="kserve-container" containerID="cri-o://c17976df0cfbbc0e486c4c497c004a63ea52a0ad54beb25a6d3eac8bd6fcf612" gracePeriod=30
Sep 17 16:42:48.270390 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:48.270067    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pl42n" podUID="021304af-7a6f-4a9d-b654-7541e237feb5" containerName="kube-rbac-proxy" containerID="cri-o://e51b0b4ce0f5a2656db65dfd20e829492959939c8a1a5a76ef94a8cac0bfaf45" gracePeriod=30
Sep 17 16:42:48.631716 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:48.631616    2585 generic.go:356] "Generic (PLEG): container finished" podID="021304af-7a6f-4a9d-b654-7541e237feb5" containerID="e51b0b4ce0f5a2656db65dfd20e829492959939c8a1a5a76ef94a8cac0bfaf45" exitCode=2
Sep 17 16:42:48.631716 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:48.631688    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pl42n" event={"ID":"021304af-7a6f-4a9d-b654-7541e237feb5","Type":"ContainerDied","Data":"e51b0b4ce0f5a2656db65dfd20e829492959939c8a1a5a76ef94a8cac0bfaf45"}
Sep 17 16:42:49.610027 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:49.610001    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pl42n"
Sep 17 16:42:49.639116 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:49.639083    2585 generic.go:356] "Generic (PLEG): container finished" podID="021304af-7a6f-4a9d-b654-7541e237feb5" containerID="c17976df0cfbbc0e486c4c497c004a63ea52a0ad54beb25a6d3eac8bd6fcf612" exitCode=0
Sep 17 16:42:49.639306 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:49.639137    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pl42n"
Sep 17 16:42:49.639306 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:49.639141    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pl42n" event={"ID":"021304af-7a6f-4a9d-b654-7541e237feb5","Type":"ContainerDied","Data":"c17976df0cfbbc0e486c4c497c004a63ea52a0ad54beb25a6d3eac8bd6fcf612"}
Sep 17 16:42:49.639306 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:49.639246    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pl42n" event={"ID":"021304af-7a6f-4a9d-b654-7541e237feb5","Type":"ContainerDied","Data":"c61f5af898155efcb0049982ef2e6260341ce1c6a87de23fa7c730002dbd79be"}
Sep 17 16:42:49.639306 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:49.639286    2585 scope.go:117] "RemoveContainer" containerID="e51b0b4ce0f5a2656db65dfd20e829492959939c8a1a5a76ef94a8cac0bfaf45"
Sep 17 16:42:49.648229 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:49.648209    2585 scope.go:117] "RemoveContainer" containerID="c17976df0cfbbc0e486c4c497c004a63ea52a0ad54beb25a6d3eac8bd6fcf612"
Sep 17 16:42:49.659226 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:49.659122    2585 scope.go:117] "RemoveContainer" containerID="521f973f24832ac59194a19d934c4271ae63080e87a7fdf7615c840391a55916"
Sep 17 16:42:49.667996 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:49.667971    2585 scope.go:117] "RemoveContainer" containerID="e51b0b4ce0f5a2656db65dfd20e829492959939c8a1a5a76ef94a8cac0bfaf45"
Sep 17 16:42:49.668340 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:42:49.668318    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e51b0b4ce0f5a2656db65dfd20e829492959939c8a1a5a76ef94a8cac0bfaf45\": container with ID starting with e51b0b4ce0f5a2656db65dfd20e829492959939c8a1a5a76ef94a8cac0bfaf45 not found: ID does not exist" containerID="e51b0b4ce0f5a2656db65dfd20e829492959939c8a1a5a76ef94a8cac0bfaf45"
Sep 17 16:42:49.668436 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:49.668349    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e51b0b4ce0f5a2656db65dfd20e829492959939c8a1a5a76ef94a8cac0bfaf45"} err="failed to get container status \"e51b0b4ce0f5a2656db65dfd20e829492959939c8a1a5a76ef94a8cac0bfaf45\": rpc error: code = NotFound desc = could not find container \"e51b0b4ce0f5a2656db65dfd20e829492959939c8a1a5a76ef94a8cac0bfaf45\": container with ID starting with e51b0b4ce0f5a2656db65dfd20e829492959939c8a1a5a76ef94a8cac0bfaf45 not found: ID does not exist"
Sep 17 16:42:49.668436 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:49.668369    2585 scope.go:117] "RemoveContainer" containerID="c17976df0cfbbc0e486c4c497c004a63ea52a0ad54beb25a6d3eac8bd6fcf612"
Sep 17 16:42:49.668650 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:42:49.668631    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c17976df0cfbbc0e486c4c497c004a63ea52a0ad54beb25a6d3eac8bd6fcf612\": container with ID starting with c17976df0cfbbc0e486c4c497c004a63ea52a0ad54beb25a6d3eac8bd6fcf612 not found: ID does not exist" containerID="c17976df0cfbbc0e486c4c497c004a63ea52a0ad54beb25a6d3eac8bd6fcf612"
Sep 17 16:42:49.668695 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:49.668656    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c17976df0cfbbc0e486c4c497c004a63ea52a0ad54beb25a6d3eac8bd6fcf612"} err="failed to get container status \"c17976df0cfbbc0e486c4c497c004a63ea52a0ad54beb25a6d3eac8bd6fcf612\": rpc error: code = NotFound desc = could not find container \"c17976df0cfbbc0e486c4c497c004a63ea52a0ad54beb25a6d3eac8bd6fcf612\": container with ID starting with c17976df0cfbbc0e486c4c497c004a63ea52a0ad54beb25a6d3eac8bd6fcf612 not found: ID does not exist"
Sep 17 16:42:49.668695 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:49.668672    2585 scope.go:117] "RemoveContainer" containerID="521f973f24832ac59194a19d934c4271ae63080e87a7fdf7615c840391a55916"
Sep 17 16:42:49.668931 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:42:49.668910    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"521f973f24832ac59194a19d934c4271ae63080e87a7fdf7615c840391a55916\": container with ID starting with 521f973f24832ac59194a19d934c4271ae63080e87a7fdf7615c840391a55916 not found: ID does not exist" containerID="521f973f24832ac59194a19d934c4271ae63080e87a7fdf7615c840391a55916"
Sep 17 16:42:49.668972 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:49.668939    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"521f973f24832ac59194a19d934c4271ae63080e87a7fdf7615c840391a55916"} err="failed to get container status \"521f973f24832ac59194a19d934c4271ae63080e87a7fdf7615c840391a55916\": rpc error: code = NotFound desc = could not find container \"521f973f24832ac59194a19d934c4271ae63080e87a7fdf7615c840391a55916\": container with ID starting with 521f973f24832ac59194a19d934c4271ae63080e87a7fdf7615c840391a55916 not found: ID does not exist"
Sep 17 16:42:49.733885 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:49.733846    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbxkc\" (UniqueName: \"kubernetes.io/projected/021304af-7a6f-4a9d-b654-7541e237feb5-kube-api-access-dbxkc\") pod \"021304af-7a6f-4a9d-b654-7541e237feb5\" (UID: \"021304af-7a6f-4a9d-b654-7541e237feb5\") "
Sep 17 16:42:49.733885 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:49.733887    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/021304af-7a6f-4a9d-b654-7541e237feb5-kube-rbac-proxy-sar-config\") pod \"021304af-7a6f-4a9d-b654-7541e237feb5\" (UID: \"021304af-7a6f-4a9d-b654-7541e237feb5\") "
Sep 17 16:42:49.734086 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:49.733909    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/021304af-7a6f-4a9d-b654-7541e237feb5-kserve-provision-location\") pod \"021304af-7a6f-4a9d-b654-7541e237feb5\" (UID: \"021304af-7a6f-4a9d-b654-7541e237feb5\") "
Sep 17 16:42:49.734140 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:49.734093    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/021304af-7a6f-4a9d-b654-7541e237feb5-proxy-tls\") pod \"021304af-7a6f-4a9d-b654-7541e237feb5\" (UID: \"021304af-7a6f-4a9d-b654-7541e237feb5\") "
Sep 17 16:42:49.734180 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:49.734156    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/021304af-7a6f-4a9d-b654-7541e237feb5-cabundle-cert\") pod \"021304af-7a6f-4a9d-b654-7541e237feb5\" (UID: \"021304af-7a6f-4a9d-b654-7541e237feb5\") "
Sep 17 16:42:49.734265 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:49.734247    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/021304af-7a6f-4a9d-b654-7541e237feb5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "021304af-7a6f-4a9d-b654-7541e237feb5" (UID: "021304af-7a6f-4a9d-b654-7541e237feb5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:42:49.734332 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:49.734270    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/021304af-7a6f-4a9d-b654-7541e237feb5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "021304af-7a6f-4a9d-b654-7541e237feb5" (UID: "021304af-7a6f-4a9d-b654-7541e237feb5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:42:49.734545 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:49.734523    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/021304af-7a6f-4a9d-b654-7541e237feb5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:42:49.734545 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:49.734545    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/021304af-7a6f-4a9d-b654-7541e237feb5-kserve-provision-location\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:42:49.734691 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:49.734553    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/021304af-7a6f-4a9d-b654-7541e237feb5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "021304af-7a6f-4a9d-b654-7541e237feb5" (UID: "021304af-7a6f-4a9d-b654-7541e237feb5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:42:49.736105 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:49.736088    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/021304af-7a6f-4a9d-b654-7541e237feb5-kube-api-access-dbxkc" (OuterVolumeSpecName: "kube-api-access-dbxkc") pod "021304af-7a6f-4a9d-b654-7541e237feb5" (UID: "021304af-7a6f-4a9d-b654-7541e237feb5"). InnerVolumeSpecName "kube-api-access-dbxkc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:42:49.736223 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:49.736200    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/021304af-7a6f-4a9d-b654-7541e237feb5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "021304af-7a6f-4a9d-b654-7541e237feb5" (UID: "021304af-7a6f-4a9d-b654-7541e237feb5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:42:49.835810 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:49.835770    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbxkc\" (UniqueName: \"kubernetes.io/projected/021304af-7a6f-4a9d-b654-7541e237feb5-kube-api-access-dbxkc\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:42:49.835810 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:49.835802    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/021304af-7a6f-4a9d-b654-7541e237feb5-proxy-tls\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:42:49.835810 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:49.835814    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/021304af-7a6f-4a9d-b654-7541e237feb5-cabundle-cert\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:42:49.964030 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:49.963994    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pl42n"]
Sep 17 16:42:49.969065 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:49.969038    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-pl42n"]
Sep 17 16:42:50.513691 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:50.513658    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="021304af-7a6f-4a9d-b654-7541e237feb5" path="/var/lib/kubelet/pods/021304af-7a6f-4a9d-b654-7541e237feb5/volumes"
Sep 17 16:42:50.617323 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:50.617288    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd"]
Sep 17 16:42:50.617828 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:50.617597    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd" podUID="f5abde47-a7e5-4a2c-8726-b6eba48a54b6" containerName="kserve-container" containerID="cri-o://a8531bbf6dbe54c0108ff1d30a68e331005a9237845ab52bb6a578e080cccd3c" gracePeriod=30
Sep 17 16:42:50.617828 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:50.617613    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd" podUID="f5abde47-a7e5-4a2c-8726-b6eba48a54b6" containerName="kube-rbac-proxy" containerID="cri-o://078884b492c64828e868152eb2d49943d5d67486f7c8714a9e23e3e8f457ca45" gracePeriod=30
Sep 17 16:42:51.654080 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:51.654045    2585 generic.go:356] "Generic (PLEG): container finished" podID="f5abde47-a7e5-4a2c-8726-b6eba48a54b6" containerID="078884b492c64828e868152eb2d49943d5d67486f7c8714a9e23e3e8f457ca45" exitCode=2
Sep 17 16:42:51.654479 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:51.654091    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd" event={"ID":"f5abde47-a7e5-4a2c-8726-b6eba48a54b6","Type":"ContainerDied","Data":"078884b492c64828e868152eb2d49943d5d67486f7c8714a9e23e3e8f457ca45"}
Sep 17 16:42:51.729016 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:51.728977    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5cb97ccc64-7zdpd"]
Sep 17 16:42:51.729338 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:51.729326    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="021304af-7a6f-4a9d-b654-7541e237feb5" containerName="storage-initializer"
Sep 17 16:42:51.729388 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:51.729340    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="021304af-7a6f-4a9d-b654-7541e237feb5" containerName="storage-initializer"
Sep 17 16:42:51.729388 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:51.729348    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="021304af-7a6f-4a9d-b654-7541e237feb5" containerName="kserve-container"
Sep 17 16:42:51.729388 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:51.729353    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="021304af-7a6f-4a9d-b654-7541e237feb5" containerName="kserve-container"
Sep 17 16:42:51.729388 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:51.729365    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="021304af-7a6f-4a9d-b654-7541e237feb5" containerName="kube-rbac-proxy"
Sep 17 16:42:51.729388 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:51.729370    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="021304af-7a6f-4a9d-b654-7541e237feb5" containerName="kube-rbac-proxy"
Sep 17 16:42:51.729567 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:51.729444    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="021304af-7a6f-4a9d-b654-7541e237feb5" containerName="kserve-container"
Sep 17 16:42:51.729567 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:51.729454    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="021304af-7a6f-4a9d-b654-7541e237feb5" containerName="kube-rbac-proxy"
Sep 17 16:42:51.732670 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:51.732643    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5cb97ccc64-7zdpd"
Sep 17 16:42:51.735634 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:51.735611    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
Sep 17 16:42:51.736043 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:51.736021    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
Sep 17 16:42:51.747560 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:51.747534    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5cb97ccc64-7zdpd"]
Sep 17 16:42:51.852259 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:51.852219    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4fdce04-2629-43a5-81b6-d2e9d1d36e01-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5cb97ccc64-7zdpd\" (UID: \"b4fdce04-2629-43a5-81b6-d2e9d1d36e01\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5cb97ccc64-7zdpd"
Sep 17 16:42:51.852464 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:51.852268    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4fdce04-2629-43a5-81b6-d2e9d1d36e01-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5cb97ccc64-7zdpd\" (UID: \"b4fdce04-2629-43a5-81b6-d2e9d1d36e01\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5cb97ccc64-7zdpd"
Sep 17 16:42:51.852464 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:51.852331    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4fdce04-2629-43a5-81b6-d2e9d1d36e01-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5cb97ccc64-7zdpd\" (UID: \"b4fdce04-2629-43a5-81b6-d2e9d1d36e01\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5cb97ccc64-7zdpd"
Sep 17 16:42:51.852464 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:51.852378    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvb8p\" (UniqueName: \"kubernetes.io/projected/b4fdce04-2629-43a5-81b6-d2e9d1d36e01-kube-api-access-xvb8p\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5cb97ccc64-7zdpd\" (UID: \"b4fdce04-2629-43a5-81b6-d2e9d1d36e01\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5cb97ccc64-7zdpd"
Sep 17 16:42:51.953526 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:51.953486    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4fdce04-2629-43a5-81b6-d2e9d1d36e01-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5cb97ccc64-7zdpd\" (UID: \"b4fdce04-2629-43a5-81b6-d2e9d1d36e01\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5cb97ccc64-7zdpd"
Sep 17 16:42:51.953705 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:51.953536    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4fdce04-2629-43a5-81b6-d2e9d1d36e01-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5cb97ccc64-7zdpd\" (UID: \"b4fdce04-2629-43a5-81b6-d2e9d1d36e01\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5cb97ccc64-7zdpd"
Sep 17 16:42:51.953705 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:51.953580    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvb8p\" (UniqueName: \"kubernetes.io/projected/b4fdce04-2629-43a5-81b6-d2e9d1d36e01-kube-api-access-xvb8p\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5cb97ccc64-7zdpd\" (UID: \"b4fdce04-2629-43a5-81b6-d2e9d1d36e01\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5cb97ccc64-7zdpd"
Sep 17 16:42:51.953705 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:51.953679    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4fdce04-2629-43a5-81b6-d2e9d1d36e01-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5cb97ccc64-7zdpd\" (UID: \"b4fdce04-2629-43a5-81b6-d2e9d1d36e01\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5cb97ccc64-7zdpd"
Sep 17 16:42:51.954056 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:51.954035    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4fdce04-2629-43a5-81b6-d2e9d1d36e01-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5cb97ccc64-7zdpd\" (UID: \"b4fdce04-2629-43a5-81b6-d2e9d1d36e01\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5cb97ccc64-7zdpd"
Sep 17 16:42:51.954291 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:51.954271    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4fdce04-2629-43a5-81b6-d2e9d1d36e01-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5cb97ccc64-7zdpd\" (UID: \"b4fdce04-2629-43a5-81b6-d2e9d1d36e01\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5cb97ccc64-7zdpd"
Sep 17 16:42:51.956220 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:51.956201    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4fdce04-2629-43a5-81b6-d2e9d1d36e01-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5cb97ccc64-7zdpd\" (UID: \"b4fdce04-2629-43a5-81b6-d2e9d1d36e01\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5cb97ccc64-7zdpd"
Sep 17 16:42:51.962630 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:51.962607    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvb8p\" (UniqueName: \"kubernetes.io/projected/b4fdce04-2629-43a5-81b6-d2e9d1d36e01-kube-api-access-xvb8p\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5cb97ccc64-7zdpd\" (UID: \"b4fdce04-2629-43a5-81b6-d2e9d1d36e01\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5cb97ccc64-7zdpd"
Sep 17 16:42:52.044186 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:52.044129    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5cb97ccc64-7zdpd"
Sep 17 16:42:52.325408 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:52.325318    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5cb97ccc64-7zdpd"]
Sep 17 16:42:52.328403 ip-10-0-135-43 kubenswrapper[2585]: W0917 16:42:52.328376    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4fdce04_2629_43a5_81b6_d2e9d1d36e01.slice/crio-804b59a7279bb496c45cf87f4ee155a1c95d4d9cf09110535ed60ecf7da488d5 WatchSource:0}: Error finding container 804b59a7279bb496c45cf87f4ee155a1c95d4d9cf09110535ed60ecf7da488d5: Status 404 returned error can't find the container with id 804b59a7279bb496c45cf87f4ee155a1c95d4d9cf09110535ed60ecf7da488d5
Sep 17 16:42:52.661489 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:52.661387    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5cb97ccc64-7zdpd" event={"ID":"b4fdce04-2629-43a5-81b6-d2e9d1d36e01","Type":"ContainerStarted","Data":"aaf0ab7b445dffd8b4c0f1ca1bc5b116e0f11f1643ee8593e5ce9fc7aac475c3"}
Sep 17 16:42:52.661489 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:52.661480    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5cb97ccc64-7zdpd" event={"ID":"b4fdce04-2629-43a5-81b6-d2e9d1d36e01","Type":"ContainerStarted","Data":"804b59a7279bb496c45cf87f4ee155a1c95d4d9cf09110535ed60ecf7da488d5"}
Sep 17 16:42:55.224572 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:55.224525    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd" podUID="f5abde47-a7e5-4a2c-8726-b6eba48a54b6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.59:8643/healthz\": dial tcp 10.134.0.59:8643: connect: connection refused"
Sep 17 16:42:55.566467 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:55.566412    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd"
Sep 17 16:42:55.586532 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:55.586497    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5abde47-a7e5-4a2c-8726-b6eba48a54b6-proxy-tls\") pod \"f5abde47-a7e5-4a2c-8726-b6eba48a54b6\" (UID: \"f5abde47-a7e5-4a2c-8726-b6eba48a54b6\") "
Sep 17 16:42:55.587817 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:55.586553    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5abde47-a7e5-4a2c-8726-b6eba48a54b6-kube-rbac-proxy-sar-config\") pod \"f5abde47-a7e5-4a2c-8726-b6eba48a54b6\" (UID: \"f5abde47-a7e5-4a2c-8726-b6eba48a54b6\") "
Sep 17 16:42:55.587817 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:55.586577    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5abde47-a7e5-4a2c-8726-b6eba48a54b6-cabundle-cert\") pod \"f5abde47-a7e5-4a2c-8726-b6eba48a54b6\" (UID: \"f5abde47-a7e5-4a2c-8726-b6eba48a54b6\") "
Sep 17 16:42:55.587817 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:55.586602    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nhqmx\" (UniqueName: \"kubernetes.io/projected/f5abde47-a7e5-4a2c-8726-b6eba48a54b6-kube-api-access-nhqmx\") pod \"f5abde47-a7e5-4a2c-8726-b6eba48a54b6\" (UID: \"f5abde47-a7e5-4a2c-8726-b6eba48a54b6\") "
Sep 17 16:42:55.587817 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:55.587224    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5abde47-a7e5-4a2c-8726-b6eba48a54b6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f5abde47-a7e5-4a2c-8726-b6eba48a54b6" (UID: "f5abde47-a7e5-4a2c-8726-b6eba48a54b6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:42:55.587817 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:55.587480    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5abde47-a7e5-4a2c-8726-b6eba48a54b6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f5abde47-a7e5-4a2c-8726-b6eba48a54b6" (UID: "f5abde47-a7e5-4a2c-8726-b6eba48a54b6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:42:55.587817 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:55.587747    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5abde47-a7e5-4a2c-8726-b6eba48a54b6-kserve-provision-location\") pod \"f5abde47-a7e5-4a2c-8726-b6eba48a54b6\" (UID: \"f5abde47-a7e5-4a2c-8726-b6eba48a54b6\") "
Sep 17 16:42:55.588433 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:55.588396    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5abde47-a7e5-4a2c-8726-b6eba48a54b6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:42:55.588493 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:55.588443    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5abde47-a7e5-4a2c-8726-b6eba48a54b6-cabundle-cert\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:42:55.588493 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:55.588441    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5abde47-a7e5-4a2c-8726-b6eba48a54b6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f5abde47-a7e5-4a2c-8726-b6eba48a54b6" (UID: "f5abde47-a7e5-4a2c-8726-b6eba48a54b6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:42:55.589234 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:55.589207    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5abde47-a7e5-4a2c-8726-b6eba48a54b6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f5abde47-a7e5-4a2c-8726-b6eba48a54b6" (UID: "f5abde47-a7e5-4a2c-8726-b6eba48a54b6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:42:55.589480 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:55.589457    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5abde47-a7e5-4a2c-8726-b6eba48a54b6-kube-api-access-nhqmx" (OuterVolumeSpecName: "kube-api-access-nhqmx") pod "f5abde47-a7e5-4a2c-8726-b6eba48a54b6" (UID: "f5abde47-a7e5-4a2c-8726-b6eba48a54b6"). InnerVolumeSpecName "kube-api-access-nhqmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:42:55.679124 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:55.679090    2585 generic.go:356] "Generic (PLEG): container finished" podID="f5abde47-a7e5-4a2c-8726-b6eba48a54b6" containerID="a8531bbf6dbe54c0108ff1d30a68e331005a9237845ab52bb6a578e080cccd3c" exitCode=0
Sep 17 16:42:55.679296 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:55.679141    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd"
Sep 17 16:42:55.679296 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:55.679171    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd" event={"ID":"f5abde47-a7e5-4a2c-8726-b6eba48a54b6","Type":"ContainerDied","Data":"a8531bbf6dbe54c0108ff1d30a68e331005a9237845ab52bb6a578e080cccd3c"}
Sep 17 16:42:55.679296 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:55.679230    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd" event={"ID":"f5abde47-a7e5-4a2c-8726-b6eba48a54b6","Type":"ContainerDied","Data":"add57e155c853057b837f591976e4360d396c4bfdccc0212102353040baa2f1a"}
Sep 17 16:42:55.679296 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:55.679257    2585 scope.go:117] "RemoveContainer" containerID="078884b492c64828e868152eb2d49943d5d67486f7c8714a9e23e3e8f457ca45"
Sep 17 16:42:55.688810 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:55.688789    2585 scope.go:117] "RemoveContainer" containerID="a8531bbf6dbe54c0108ff1d30a68e331005a9237845ab52bb6a578e080cccd3c"
Sep 17 16:42:55.689057 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:55.689043    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5abde47-a7e5-4a2c-8726-b6eba48a54b6-proxy-tls\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:42:55.689109 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:55.689061    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhqmx\" (UniqueName: \"kubernetes.io/projected/f5abde47-a7e5-4a2c-8726-b6eba48a54b6-kube-api-access-nhqmx\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:42:55.689109 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:55.689071    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5abde47-a7e5-4a2c-8726-b6eba48a54b6-kserve-provision-location\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:42:55.697042 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:55.697025    2585 scope.go:117] "RemoveContainer" containerID="4577e0d61b961ab8405baaaba4bdbab6b989d5912fdcc0cd61458712a8119923"
Sep 17 16:42:55.702551 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:55.702527    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd"]
Sep 17 16:42:55.705311 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:55.705292    2585 scope.go:117] "RemoveContainer" containerID="078884b492c64828e868152eb2d49943d5d67486f7c8714a9e23e3e8f457ca45"
Sep 17 16:42:55.705663 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:42:55.705641    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"078884b492c64828e868152eb2d49943d5d67486f7c8714a9e23e3e8f457ca45\": container with ID starting with 078884b492c64828e868152eb2d49943d5d67486f7c8714a9e23e3e8f457ca45 not found: ID does not exist" containerID="078884b492c64828e868152eb2d49943d5d67486f7c8714a9e23e3e8f457ca45"
Sep 17 16:42:55.705769 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:55.705671    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"078884b492c64828e868152eb2d49943d5d67486f7c8714a9e23e3e8f457ca45"} err="failed to get container status \"078884b492c64828e868152eb2d49943d5d67486f7c8714a9e23e3e8f457ca45\": rpc error: code = NotFound desc = could not find container \"078884b492c64828e868152eb2d49943d5d67486f7c8714a9e23e3e8f457ca45\": container with ID starting with 078884b492c64828e868152eb2d49943d5d67486f7c8714a9e23e3e8f457ca45 not found: ID does not exist"
Sep 17 16:42:55.705769 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:55.705693    2585 scope.go:117] "RemoveContainer" containerID="a8531bbf6dbe54c0108ff1d30a68e331005a9237845ab52bb6a578e080cccd3c"
Sep 17 16:42:55.705967 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:42:55.705950    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8531bbf6dbe54c0108ff1d30a68e331005a9237845ab52bb6a578e080cccd3c\": container with ID starting with a8531bbf6dbe54c0108ff1d30a68e331005a9237845ab52bb6a578e080cccd3c not found: ID does not exist" containerID="a8531bbf6dbe54c0108ff1d30a68e331005a9237845ab52bb6a578e080cccd3c"
Sep 17 16:42:55.706031 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:55.705971    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8531bbf6dbe54c0108ff1d30a68e331005a9237845ab52bb6a578e080cccd3c"} err="failed to get container status \"a8531bbf6dbe54c0108ff1d30a68e331005a9237845ab52bb6a578e080cccd3c\": rpc error: code = NotFound desc = could not find container \"a8531bbf6dbe54c0108ff1d30a68e331005a9237845ab52bb6a578e080cccd3c\": container with ID starting with a8531bbf6dbe54c0108ff1d30a68e331005a9237845ab52bb6a578e080cccd3c not found: ID does not exist"
Sep 17 16:42:55.706031 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:55.705986    2585 scope.go:117] "RemoveContainer" containerID="4577e0d61b961ab8405baaaba4bdbab6b989d5912fdcc0cd61458712a8119923"
Sep 17 16:42:55.706257 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:42:55.706242    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4577e0d61b961ab8405baaaba4bdbab6b989d5912fdcc0cd61458712a8119923\": container with ID starting with 4577e0d61b961ab8405baaaba4bdbab6b989d5912fdcc0cd61458712a8119923 not found: ID does not exist" containerID="4577e0d61b961ab8405baaaba4bdbab6b989d5912fdcc0cd61458712a8119923"
Sep 17 16:42:55.706321 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:55.706260    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4577e0d61b961ab8405baaaba4bdbab6b989d5912fdcc0cd61458712a8119923"} err="failed to get container status \"4577e0d61b961ab8405baaaba4bdbab6b989d5912fdcc0cd61458712a8119923\": rpc error: code = NotFound desc = could not find container \"4577e0d61b961ab8405baaaba4bdbab6b989d5912fdcc0cd61458712a8119923\": container with ID starting with 4577e0d61b961ab8405baaaba4bdbab6b989d5912fdcc0cd61458712a8119923 not found: ID does not exist"
Sep 17 16:42:55.706646 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:55.706625    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-66b66dd5d4-t78vd"]
Sep 17 16:42:56.512887 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:56.512845    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5abde47-a7e5-4a2c-8726-b6eba48a54b6" path="/var/lib/kubelet/pods/f5abde47-a7e5-4a2c-8726-b6eba48a54b6/volumes"
Sep 17 16:42:58.695534 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:58.695508    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-5cb97ccc64-7zdpd_b4fdce04-2629-43a5-81b6-d2e9d1d36e01/storage-initializer/0.log"
Sep 17 16:42:58.695926 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:58.695546    2585 generic.go:356] "Generic (PLEG): container finished" podID="b4fdce04-2629-43a5-81b6-d2e9d1d36e01" containerID="aaf0ab7b445dffd8b4c0f1ca1bc5b116e0f11f1643ee8593e5ce9fc7aac475c3" exitCode=1
Sep 17 16:42:58.695926 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:58.695579    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5cb97ccc64-7zdpd" event={"ID":"b4fdce04-2629-43a5-81b6-d2e9d1d36e01","Type":"ContainerDied","Data":"aaf0ab7b445dffd8b4c0f1ca1bc5b116e0f11f1643ee8593e5ce9fc7aac475c3"}
Sep 17 16:42:59.702138 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:59.702110    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-5cb97ccc64-7zdpd_b4fdce04-2629-43a5-81b6-d2e9d1d36e01/storage-initializer/0.log"
Sep 17 16:42:59.702533 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:42:59.702191    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5cb97ccc64-7zdpd" event={"ID":"b4fdce04-2629-43a5-81b6-d2e9d1d36e01","Type":"ContainerStarted","Data":"b19d32db12fadfcf7e166f255fade6172159bff1fa8d3912d3277ec0db6f8331"}
Sep 17 16:43:01.688868 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:01.688827    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5cb97ccc64-7zdpd"]
Sep 17 16:43:01.689320 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:01.689090    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5cb97ccc64-7zdpd" podUID="b4fdce04-2629-43a5-81b6-d2e9d1d36e01" containerName="storage-initializer" containerID="cri-o://b19d32db12fadfcf7e166f255fade6172159bff1fa8d3912d3277ec0db6f8331" gracePeriod=30
Sep 17 16:43:02.850926 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:02.850887    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq"]
Sep 17 16:43:02.851315 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:02.851231    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5abde47-a7e5-4a2c-8726-b6eba48a54b6" containerName="kube-rbac-proxy"
Sep 17 16:43:02.851315 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:02.851242    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5abde47-a7e5-4a2c-8726-b6eba48a54b6" containerName="kube-rbac-proxy"
Sep 17 16:43:02.851315 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:02.851251    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5abde47-a7e5-4a2c-8726-b6eba48a54b6" containerName="kserve-container"
Sep 17 16:43:02.851315 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:02.851257    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5abde47-a7e5-4a2c-8726-b6eba48a54b6" containerName="kserve-container"
Sep 17 16:43:02.851315 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:02.851269    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5abde47-a7e5-4a2c-8726-b6eba48a54b6" containerName="storage-initializer"
Sep 17 16:43:02.851315 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:02.851275    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5abde47-a7e5-4a2c-8726-b6eba48a54b6" containerName="storage-initializer"
Sep 17 16:43:02.851534 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:02.851334    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5abde47-a7e5-4a2c-8726-b6eba48a54b6" containerName="kserve-container"
Sep 17 16:43:02.851534 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:02.851343    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5abde47-a7e5-4a2c-8726-b6eba48a54b6" containerName="kube-rbac-proxy"
Sep 17 16:43:02.854576 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:02.854554    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq"
Sep 17 16:43:02.857359 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:02.857330    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
Sep 17 16:43:02.857519 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:02.857503    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:43:02.857762 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:02.857735    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
Sep 17 16:43:02.866743 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:02.866714    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq"]
Sep 17 16:43:02.950063 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:02.950020    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1d93ba27-19fe-4a3e-97e0-9c678991e614-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq\" (UID: \"1d93ba27-19fe-4a3e-97e0-9c678991e614\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq"
Sep 17 16:43:02.950063 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:02.950064    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5c8h\" (UniqueName: \"kubernetes.io/projected/1d93ba27-19fe-4a3e-97e0-9c678991e614-kube-api-access-m5c8h\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq\" (UID: \"1d93ba27-19fe-4a3e-97e0-9c678991e614\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq"
Sep 17 16:43:02.950307 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:02.950151    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d93ba27-19fe-4a3e-97e0-9c678991e614-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq\" (UID: \"1d93ba27-19fe-4a3e-97e0-9c678991e614\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq"
Sep 17 16:43:02.950307 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:02.950228    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d93ba27-19fe-4a3e-97e0-9c678991e614-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq\" (UID: \"1d93ba27-19fe-4a3e-97e0-9c678991e614\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq"
Sep 17 16:43:02.950307 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:02.950261    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1d93ba27-19fe-4a3e-97e0-9c678991e614-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq\" (UID: \"1d93ba27-19fe-4a3e-97e0-9c678991e614\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq"
Sep 17 16:43:03.051037 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:03.050993    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5c8h\" (UniqueName: \"kubernetes.io/projected/1d93ba27-19fe-4a3e-97e0-9c678991e614-kube-api-access-m5c8h\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq\" (UID: \"1d93ba27-19fe-4a3e-97e0-9c678991e614\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq"
Sep 17 16:43:03.051236 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:03.051057    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d93ba27-19fe-4a3e-97e0-9c678991e614-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq\" (UID: \"1d93ba27-19fe-4a3e-97e0-9c678991e614\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq"
Sep 17 16:43:03.051236 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:03.051096    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d93ba27-19fe-4a3e-97e0-9c678991e614-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq\" (UID: \"1d93ba27-19fe-4a3e-97e0-9c678991e614\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq"
Sep 17 16:43:03.051236 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:03.051114    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1d93ba27-19fe-4a3e-97e0-9c678991e614-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq\" (UID: \"1d93ba27-19fe-4a3e-97e0-9c678991e614\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq"
Sep 17 16:43:03.051236 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:03.051149    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1d93ba27-19fe-4a3e-97e0-9c678991e614-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq\" (UID: \"1d93ba27-19fe-4a3e-97e0-9c678991e614\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq"
Sep 17 16:43:03.051236 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:43:03.051226    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert" not found
Sep 17 16:43:03.051486 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:43:03.051298    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d93ba27-19fe-4a3e-97e0-9c678991e614-proxy-tls podName:1d93ba27-19fe-4a3e-97e0-9c678991e614 nodeName:}" failed. No retries permitted until 2026-09-17 16:43:03.55127831 +0000 UTC m=+2939.321610104 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1d93ba27-19fe-4a3e-97e0-9c678991e614-proxy-tls") pod "isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq" (UID: "1d93ba27-19fe-4a3e-97e0-9c678991e614") : secret "isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert" not found
Sep 17 16:43:03.051656 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:03.051635    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1d93ba27-19fe-4a3e-97e0-9c678991e614-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq\" (UID: \"1d93ba27-19fe-4a3e-97e0-9c678991e614\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq"
Sep 17 16:43:03.051849 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:03.051830    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1d93ba27-19fe-4a3e-97e0-9c678991e614-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq\" (UID: \"1d93ba27-19fe-4a3e-97e0-9c678991e614\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq"
Sep 17 16:43:03.051890 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:03.051861    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d93ba27-19fe-4a3e-97e0-9c678991e614-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq\" (UID: \"1d93ba27-19fe-4a3e-97e0-9c678991e614\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq"
Sep 17 16:43:03.068073 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:03.068026    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5c8h\" (UniqueName: \"kubernetes.io/projected/1d93ba27-19fe-4a3e-97e0-9c678991e614-kube-api-access-m5c8h\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq\" (UID: \"1d93ba27-19fe-4a3e-97e0-9c678991e614\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq"
Sep 17 16:43:03.555815 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:03.555771    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d93ba27-19fe-4a3e-97e0-9c678991e614-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq\" (UID: \"1d93ba27-19fe-4a3e-97e0-9c678991e614\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq"
Sep 17 16:43:03.558279 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:03.558250    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d93ba27-19fe-4a3e-97e0-9c678991e614-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq\" (UID: \"1d93ba27-19fe-4a3e-97e0-9c678991e614\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq"
Sep 17 16:43:03.766152 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:03.766123    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq"
Sep 17 16:43:03.941140 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:03.941118    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-5cb97ccc64-7zdpd_b4fdce04-2629-43a5-81b6-d2e9d1d36e01/storage-initializer/1.log"
Sep 17 16:43:03.941532 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:03.941517    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-5cb97ccc64-7zdpd_b4fdce04-2629-43a5-81b6-d2e9d1d36e01/storage-initializer/0.log"
Sep 17 16:43:03.943883 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:03.943868    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5cb97ccc64-7zdpd"
Sep 17 16:43:03.958962 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:03.958942    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4fdce04-2629-43a5-81b6-d2e9d1d36e01-proxy-tls\") pod \"b4fdce04-2629-43a5-81b6-d2e9d1d36e01\" (UID: \"b4fdce04-2629-43a5-81b6-d2e9d1d36e01\") "
Sep 17 16:43:03.959072 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:03.958996    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4fdce04-2629-43a5-81b6-d2e9d1d36e01-kserve-provision-location\") pod \"b4fdce04-2629-43a5-81b6-d2e9d1d36e01\" (UID: \"b4fdce04-2629-43a5-81b6-d2e9d1d36e01\") "
Sep 17 16:43:03.959072 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:03.959021    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvb8p\" (UniqueName: \"kubernetes.io/projected/b4fdce04-2629-43a5-81b6-d2e9d1d36e01-kube-api-access-xvb8p\") pod \"b4fdce04-2629-43a5-81b6-d2e9d1d36e01\" (UID: \"b4fdce04-2629-43a5-81b6-d2e9d1d36e01\") "
Sep 17 16:43:03.959072 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:03.959060    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4fdce04-2629-43a5-81b6-d2e9d1d36e01-kube-rbac-proxy-sar-config\") pod \"b4fdce04-2629-43a5-81b6-d2e9d1d36e01\" (UID: \"b4fdce04-2629-43a5-81b6-d2e9d1d36e01\") "
Sep 17 16:43:03.959207 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:03.959186    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4fdce04-2629-43a5-81b6-d2e9d1d36e01-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b4fdce04-2629-43a5-81b6-d2e9d1d36e01" (UID: "b4fdce04-2629-43a5-81b6-d2e9d1d36e01"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:43:03.959507 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:03.959484    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4fdce04-2629-43a5-81b6-d2e9d1d36e01-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b4fdce04-2629-43a5-81b6-d2e9d1d36e01" (UID: "b4fdce04-2629-43a5-81b6-d2e9d1d36e01"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:43:03.961103 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:03.961077    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4fdce04-2629-43a5-81b6-d2e9d1d36e01-kube-api-access-xvb8p" (OuterVolumeSpecName: "kube-api-access-xvb8p") pod "b4fdce04-2629-43a5-81b6-d2e9d1d36e01" (UID: "b4fdce04-2629-43a5-81b6-d2e9d1d36e01"). InnerVolumeSpecName "kube-api-access-xvb8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:43:03.961175 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:03.961132    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4fdce04-2629-43a5-81b6-d2e9d1d36e01-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b4fdce04-2629-43a5-81b6-d2e9d1d36e01" (UID: "b4fdce04-2629-43a5-81b6-d2e9d1d36e01"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:43:04.048670 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:04.048646    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq"]
Sep 17 16:43:04.050855 ip-10-0-135-43 kubenswrapper[2585]: W0917 16:43:04.050814    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d93ba27_19fe_4a3e_97e0_9c678991e614.slice/crio-4e5ceeee6de864a92f5ada9907abc54ea8cbd7c7d3c6b2dcc478a684a3b9d098 WatchSource:0}: Error finding container 4e5ceeee6de864a92f5ada9907abc54ea8cbd7c7d3c6b2dcc478a684a3b9d098: Status 404 returned error can't find the container with id 4e5ceeee6de864a92f5ada9907abc54ea8cbd7c7d3c6b2dcc478a684a3b9d098
Sep 17 16:43:04.059964 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:04.059940    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4fdce04-2629-43a5-81b6-d2e9d1d36e01-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:43:04.060045 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:04.059969    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4fdce04-2629-43a5-81b6-d2e9d1d36e01-proxy-tls\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:43:04.060045 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:04.059983    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4fdce04-2629-43a5-81b6-d2e9d1d36e01-kserve-provision-location\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:43:04.060045 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:04.059991    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvb8p\" (UniqueName: \"kubernetes.io/projected/b4fdce04-2629-43a5-81b6-d2e9d1d36e01-kube-api-access-xvb8p\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:43:04.728662 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:04.728630    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-5cb97ccc64-7zdpd_b4fdce04-2629-43a5-81b6-d2e9d1d36e01/storage-initializer/1.log"
Sep 17 16:43:04.729107 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:04.729089    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-5cb97ccc64-7zdpd_b4fdce04-2629-43a5-81b6-d2e9d1d36e01/storage-initializer/0.log"
Sep 17 16:43:04.729176 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:04.729128    2585 generic.go:356] "Generic (PLEG): container finished" podID="b4fdce04-2629-43a5-81b6-d2e9d1d36e01" containerID="b19d32db12fadfcf7e166f255fade6172159bff1fa8d3912d3277ec0db6f8331" exitCode=1
Sep 17 16:43:04.729176 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:04.729156    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5cb97ccc64-7zdpd" event={"ID":"b4fdce04-2629-43a5-81b6-d2e9d1d36e01","Type":"ContainerDied","Data":"b19d32db12fadfcf7e166f255fade6172159bff1fa8d3912d3277ec0db6f8331"}
Sep 17 16:43:04.729282 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:04.729230    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5cb97ccc64-7zdpd"
Sep 17 16:43:04.729282 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:04.729247    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5cb97ccc64-7zdpd" event={"ID":"b4fdce04-2629-43a5-81b6-d2e9d1d36e01","Type":"ContainerDied","Data":"804b59a7279bb496c45cf87f4ee155a1c95d4d9cf09110535ed60ecf7da488d5"}
Sep 17 16:43:04.729282 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:04.729280    2585 scope.go:117] "RemoveContainer" containerID="b19d32db12fadfcf7e166f255fade6172159bff1fa8d3912d3277ec0db6f8331"
Sep 17 16:43:04.731103 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:04.731077    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq" event={"ID":"1d93ba27-19fe-4a3e-97e0-9c678991e614","Type":"ContainerStarted","Data":"5a2ff47cef043baa940259284dfcecb6477a8c386283fdbee864c5b15852f4c6"}
Sep 17 16:43:04.731225 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:04.731129    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq" event={"ID":"1d93ba27-19fe-4a3e-97e0-9c678991e614","Type":"ContainerStarted","Data":"4e5ceeee6de864a92f5ada9907abc54ea8cbd7c7d3c6b2dcc478a684a3b9d098"}
Sep 17 16:43:04.738188 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:04.738163    2585 scope.go:117] "RemoveContainer" containerID="aaf0ab7b445dffd8b4c0f1ca1bc5b116e0f11f1643ee8593e5ce9fc7aac475c3"
Sep 17 16:43:04.745980 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:04.745959    2585 scope.go:117] "RemoveContainer" containerID="b19d32db12fadfcf7e166f255fade6172159bff1fa8d3912d3277ec0db6f8331"
Sep 17 16:43:04.746304 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:43:04.746283    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b19d32db12fadfcf7e166f255fade6172159bff1fa8d3912d3277ec0db6f8331\": container with ID starting with b19d32db12fadfcf7e166f255fade6172159bff1fa8d3912d3277ec0db6f8331 not found: ID does not exist" containerID="b19d32db12fadfcf7e166f255fade6172159bff1fa8d3912d3277ec0db6f8331"
Sep 17 16:43:04.746383 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:04.746313    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b19d32db12fadfcf7e166f255fade6172159bff1fa8d3912d3277ec0db6f8331"} err="failed to get container status \"b19d32db12fadfcf7e166f255fade6172159bff1fa8d3912d3277ec0db6f8331\": rpc error: code = NotFound desc = could not find container \"b19d32db12fadfcf7e166f255fade6172159bff1fa8d3912d3277ec0db6f8331\": container with ID starting with b19d32db12fadfcf7e166f255fade6172159bff1fa8d3912d3277ec0db6f8331 not found: ID does not exist"
Sep 17 16:43:04.746383 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:04.746331    2585 scope.go:117] "RemoveContainer" containerID="aaf0ab7b445dffd8b4c0f1ca1bc5b116e0f11f1643ee8593e5ce9fc7aac475c3"
Sep 17 16:43:04.746619 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:43:04.746601    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaf0ab7b445dffd8b4c0f1ca1bc5b116e0f11f1643ee8593e5ce9fc7aac475c3\": container with ID starting with aaf0ab7b445dffd8b4c0f1ca1bc5b116e0f11f1643ee8593e5ce9fc7aac475c3 not found: ID does not exist" containerID="aaf0ab7b445dffd8b4c0f1ca1bc5b116e0f11f1643ee8593e5ce9fc7aac475c3"
Sep 17 16:43:04.746673 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:04.746625    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaf0ab7b445dffd8b4c0f1ca1bc5b116e0f11f1643ee8593e5ce9fc7aac475c3"} err="failed to get container status \"aaf0ab7b445dffd8b4c0f1ca1bc5b116e0f11f1643ee8593e5ce9fc7aac475c3\": rpc error: code = NotFound desc = could not find container \"aaf0ab7b445dffd8b4c0f1ca1bc5b116e0f11f1643ee8593e5ce9fc7aac475c3\": container with ID starting with aaf0ab7b445dffd8b4c0f1ca1bc5b116e0f11f1643ee8593e5ce9fc7aac475c3 not found: ID does not exist"
Sep 17 16:43:04.768545 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:04.768517    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5cb97ccc64-7zdpd"]
Sep 17 16:43:04.777859 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:04.777777    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5cb97ccc64-7zdpd"]
Sep 17 16:43:05.737942 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:05.737903    2585 generic.go:356] "Generic (PLEG): container finished" podID="1d93ba27-19fe-4a3e-97e0-9c678991e614" containerID="5a2ff47cef043baa940259284dfcecb6477a8c386283fdbee864c5b15852f4c6" exitCode=0
Sep 17 16:43:05.738334 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:05.737990    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq" event={"ID":"1d93ba27-19fe-4a3e-97e0-9c678991e614","Type":"ContainerDied","Data":"5a2ff47cef043baa940259284dfcecb6477a8c386283fdbee864c5b15852f4c6"}
Sep 17 16:43:06.514046 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:06.514009    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4fdce04-2629-43a5-81b6-d2e9d1d36e01" path="/var/lib/kubelet/pods/b4fdce04-2629-43a5-81b6-d2e9d1d36e01/volumes"
Sep 17 16:43:06.749057 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:06.749019    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq" event={"ID":"1d93ba27-19fe-4a3e-97e0-9c678991e614","Type":"ContainerStarted","Data":"10d4be5037d5de276e1ef86c88fee787c2dd4ec2d1b82f21d4a4b8d10b76e54c"}
Sep 17 16:43:06.749534 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:06.749085    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq"
Sep 17 16:43:06.749534 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:06.749105    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq" event={"ID":"1d93ba27-19fe-4a3e-97e0-9c678991e614","Type":"ContainerStarted","Data":"e8ff4cd2be5548aca717d5be4e0be5a0f252cd1954f810bece16f37f13701f6f"}
Sep 17 16:43:06.749534 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:06.749237    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq"
Sep 17 16:43:06.750667 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:06.750641    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq" podUID="1d93ba27-19fe-4a3e-97e0-9c678991e614" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 17 16:43:07.753759 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:07.753714    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq" podUID="1d93ba27-19fe-4a3e-97e0-9c678991e614" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 17 16:43:12.758370 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:12.758341    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq"
Sep 17 16:43:12.758864 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:12.758842    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq" podUID="1d93ba27-19fe-4a3e-97e0-9c678991e614" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 17 16:43:12.782108 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:12.782050    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq" podStartSLOduration=9.583892293 podStartE2EDuration="10.782029293s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.198137s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:43:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:43:06.784370015 +0000 UTC m=+2942.554701815" watchObservedRunningTime="2026-09-17 16:43:12.782029293 +0000 UTC m=+2948.552361094"
Sep 17 16:43:22.759570 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:22.759488    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq" podUID="1d93ba27-19fe-4a3e-97e0-9c678991e614" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 17 16:43:32.759557 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:32.759517    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq" podUID="1d93ba27-19fe-4a3e-97e0-9c678991e614" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 17 16:43:42.759627 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:42.759588    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq" podUID="1d93ba27-19fe-4a3e-97e0-9c678991e614" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 17 16:43:52.759311 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:43:52.759265    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq" podUID="1d93ba27-19fe-4a3e-97e0-9c678991e614" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 17 16:44:02.759437 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:02.759389    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq" podUID="1d93ba27-19fe-4a3e-97e0-9c678991e614" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 17 16:44:12.759187 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:12.759142    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq" podUID="1d93ba27-19fe-4a3e-97e0-9c678991e614" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 17 16:44:18.507005 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:18.506959    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq" podUID="1d93ba27-19fe-4a3e-97e0-9c678991e614" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 17 16:44:28.512063 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:28.512031    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq"
Sep 17 16:44:32.893888 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:32.893852    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq"]
Sep 17 16:44:32.894321 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:32.894147    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq" podUID="1d93ba27-19fe-4a3e-97e0-9c678991e614" containerName="kserve-container" containerID="cri-o://e8ff4cd2be5548aca717d5be4e0be5a0f252cd1954f810bece16f37f13701f6f" gracePeriod=30
Sep 17 16:44:32.894321 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:32.894166    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq" podUID="1d93ba27-19fe-4a3e-97e0-9c678991e614" containerName="kube-rbac-proxy" containerID="cri-o://10d4be5037d5de276e1ef86c88fee787c2dd4ec2d1b82f21d4a4b8d10b76e54c" gracePeriod=30
Sep 17 16:44:33.152935 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:33.152843    2585 generic.go:356] "Generic (PLEG): container finished" podID="1d93ba27-19fe-4a3e-97e0-9c678991e614" containerID="10d4be5037d5de276e1ef86c88fee787c2dd4ec2d1b82f21d4a4b8d10b76e54c" exitCode=2
Sep 17 16:44:33.152935 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:33.152916    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq" event={"ID":"1d93ba27-19fe-4a3e-97e0-9c678991e614","Type":"ContainerDied","Data":"10d4be5037d5de276e1ef86c88fee787c2dd4ec2d1b82f21d4a4b8d10b76e54c"}
Sep 17 16:44:34.255912 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:34.255879    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b64946bf6-nz6tq"]
Sep 17 16:44:34.256326 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:34.256243    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4fdce04-2629-43a5-81b6-d2e9d1d36e01" containerName="storage-initializer"
Sep 17 16:44:34.256326 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:34.256254    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4fdce04-2629-43a5-81b6-d2e9d1d36e01" containerName="storage-initializer"
Sep 17 16:44:34.256326 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:34.256322    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4fdce04-2629-43a5-81b6-d2e9d1d36e01" containerName="storage-initializer"
Sep 17 16:44:34.256458 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:34.256387    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4fdce04-2629-43a5-81b6-d2e9d1d36e01" containerName="storage-initializer"
Sep 17 16:44:34.256458 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:34.256393    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4fdce04-2629-43a5-81b6-d2e9d1d36e01" containerName="storage-initializer"
Sep 17 16:44:34.256556 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:34.256466    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4fdce04-2629-43a5-81b6-d2e9d1d36e01" containerName="storage-initializer"
Sep 17 16:44:34.259540 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:34.259517    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b64946bf6-nz6tq"
Sep 17 16:44:34.264283 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:34.264259    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
Sep 17 16:44:34.264497 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:34.264478    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
Sep 17 16:44:34.276074 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:34.276040    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b64946bf6-nz6tq"]
Sep 17 16:44:34.425493 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:34.425453    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e537f5ca-8be5-41dc-8a42-5cb565779a37-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7b64946bf6-nz6tq\" (UID: \"e537f5ca-8be5-41dc-8a42-5cb565779a37\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b64946bf6-nz6tq"
Sep 17 16:44:34.425713 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:34.425523    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r27zp\" (UniqueName: \"kubernetes.io/projected/e537f5ca-8be5-41dc-8a42-5cb565779a37-kube-api-access-r27zp\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7b64946bf6-nz6tq\" (UID: \"e537f5ca-8be5-41dc-8a42-5cb565779a37\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b64946bf6-nz6tq"
Sep 17 16:44:34.425713 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:34.425569    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e537f5ca-8be5-41dc-8a42-5cb565779a37-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7b64946bf6-nz6tq\" (UID: \"e537f5ca-8be5-41dc-8a42-5cb565779a37\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b64946bf6-nz6tq"
Sep 17 16:44:34.425713 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:34.425598    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e537f5ca-8be5-41dc-8a42-5cb565779a37-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7b64946bf6-nz6tq\" (UID: \"e537f5ca-8be5-41dc-8a42-5cb565779a37\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b64946bf6-nz6tq"
Sep 17 16:44:34.526396 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:34.526301    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r27zp\" (UniqueName: \"kubernetes.io/projected/e537f5ca-8be5-41dc-8a42-5cb565779a37-kube-api-access-r27zp\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7b64946bf6-nz6tq\" (UID: \"e537f5ca-8be5-41dc-8a42-5cb565779a37\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b64946bf6-nz6tq"
Sep 17 16:44:34.526396 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:34.526387    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e537f5ca-8be5-41dc-8a42-5cb565779a37-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7b64946bf6-nz6tq\" (UID: \"e537f5ca-8be5-41dc-8a42-5cb565779a37\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b64946bf6-nz6tq"
Sep 17 16:44:34.526651 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:34.526449    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e537f5ca-8be5-41dc-8a42-5cb565779a37-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7b64946bf6-nz6tq\" (UID: \"e537f5ca-8be5-41dc-8a42-5cb565779a37\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b64946bf6-nz6tq"
Sep 17 16:44:34.526651 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:34.526520    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e537f5ca-8be5-41dc-8a42-5cb565779a37-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7b64946bf6-nz6tq\" (UID: \"e537f5ca-8be5-41dc-8a42-5cb565779a37\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b64946bf6-nz6tq"
Sep 17 16:44:34.526899 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:34.526877    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e537f5ca-8be5-41dc-8a42-5cb565779a37-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7b64946bf6-nz6tq\" (UID: \"e537f5ca-8be5-41dc-8a42-5cb565779a37\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b64946bf6-nz6tq"
Sep 17 16:44:34.527110 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:34.527089    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e537f5ca-8be5-41dc-8a42-5cb565779a37-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7b64946bf6-nz6tq\" (UID: \"e537f5ca-8be5-41dc-8a42-5cb565779a37\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b64946bf6-nz6tq"
Sep 17 16:44:34.529126 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:34.529105    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e537f5ca-8be5-41dc-8a42-5cb565779a37-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7b64946bf6-nz6tq\" (UID: \"e537f5ca-8be5-41dc-8a42-5cb565779a37\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b64946bf6-nz6tq"
Sep 17 16:44:34.537019 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:34.536993    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r27zp\" (UniqueName: \"kubernetes.io/projected/e537f5ca-8be5-41dc-8a42-5cb565779a37-kube-api-access-r27zp\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7b64946bf6-nz6tq\" (UID: \"e537f5ca-8be5-41dc-8a42-5cb565779a37\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b64946bf6-nz6tq"
Sep 17 16:44:34.570260 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:34.570219    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b64946bf6-nz6tq"
Sep 17 16:44:34.854833 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:34.854737    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b64946bf6-nz6tq"]
Sep 17 16:44:34.857936 ip-10-0-135-43 kubenswrapper[2585]: W0917 16:44:34.857902    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode537f5ca_8be5_41dc_8a42_5cb565779a37.slice/crio-cc76c76056a3e4528eb31338bc6fad2933d62c609806b4fdde20bbfb0e7911c2 WatchSource:0}: Error finding container cc76c76056a3e4528eb31338bc6fad2933d62c609806b4fdde20bbfb0e7911c2: Status 404 returned error can't find the container with id cc76c76056a3e4528eb31338bc6fad2933d62c609806b4fdde20bbfb0e7911c2
Sep 17 16:44:35.166949 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:35.166860    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b64946bf6-nz6tq" event={"ID":"e537f5ca-8be5-41dc-8a42-5cb565779a37","Type":"ContainerStarted","Data":"99555302da10ff8e2075e847e88d373306b7477deb456e886389d5228a972d49"}
Sep 17 16:44:35.166949 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:35.166920    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b64946bf6-nz6tq" event={"ID":"e537f5ca-8be5-41dc-8a42-5cb565779a37","Type":"ContainerStarted","Data":"cc76c76056a3e4528eb31338bc6fad2933d62c609806b4fdde20bbfb0e7911c2"}
Sep 17 16:44:37.754262 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:37.754229    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq" podUID="1d93ba27-19fe-4a3e-97e0-9c678991e614" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.62:8643/healthz\": dial tcp 10.134.0.62:8643: connect: connection refused"
Sep 17 16:44:38.147251 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:38.147224    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq"
Sep 17 16:44:38.182929 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:38.182894    2585 generic.go:356] "Generic (PLEG): container finished" podID="1d93ba27-19fe-4a3e-97e0-9c678991e614" containerID="e8ff4cd2be5548aca717d5be4e0be5a0f252cd1954f810bece16f37f13701f6f" exitCode=0
Sep 17 16:44:38.183142 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:38.182957    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq"
Sep 17 16:44:38.183142 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:38.182984    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq" event={"ID":"1d93ba27-19fe-4a3e-97e0-9c678991e614","Type":"ContainerDied","Data":"e8ff4cd2be5548aca717d5be4e0be5a0f252cd1954f810bece16f37f13701f6f"}
Sep 17 16:44:38.183142 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:38.183058    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq" event={"ID":"1d93ba27-19fe-4a3e-97e0-9c678991e614","Type":"ContainerDied","Data":"4e5ceeee6de864a92f5ada9907abc54ea8cbd7c7d3c6b2dcc478a684a3b9d098"}
Sep 17 16:44:38.183142 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:38.183097    2585 scope.go:117] "RemoveContainer" containerID="10d4be5037d5de276e1ef86c88fee787c2dd4ec2d1b82f21d4a4b8d10b76e54c"
Sep 17 16:44:38.184837 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:38.184818    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-7b64946bf6-nz6tq_e537f5ca-8be5-41dc-8a42-5cb565779a37/storage-initializer/0.log"
Sep 17 16:44:38.184937 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:38.184860    2585 generic.go:356] "Generic (PLEG): container finished" podID="e537f5ca-8be5-41dc-8a42-5cb565779a37" containerID="99555302da10ff8e2075e847e88d373306b7477deb456e886389d5228a972d49" exitCode=1
Sep 17 16:44:38.184937 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:38.184922    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b64946bf6-nz6tq" event={"ID":"e537f5ca-8be5-41dc-8a42-5cb565779a37","Type":"ContainerDied","Data":"99555302da10ff8e2075e847e88d373306b7477deb456e886389d5228a972d49"}
Sep 17 16:44:38.193390 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:38.193273    2585 scope.go:117] "RemoveContainer" containerID="e8ff4cd2be5548aca717d5be4e0be5a0f252cd1954f810bece16f37f13701f6f"
Sep 17 16:44:38.202868 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:38.202841    2585 scope.go:117] "RemoveContainer" containerID="5a2ff47cef043baa940259284dfcecb6477a8c386283fdbee864c5b15852f4c6"
Sep 17 16:44:38.216569 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:38.216544    2585 scope.go:117] "RemoveContainer" containerID="10d4be5037d5de276e1ef86c88fee787c2dd4ec2d1b82f21d4a4b8d10b76e54c"
Sep 17 16:44:38.216967 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:44:38.216950    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10d4be5037d5de276e1ef86c88fee787c2dd4ec2d1b82f21d4a4b8d10b76e54c\": container with ID starting with 10d4be5037d5de276e1ef86c88fee787c2dd4ec2d1b82f21d4a4b8d10b76e54c not found: ID does not exist" containerID="10d4be5037d5de276e1ef86c88fee787c2dd4ec2d1b82f21d4a4b8d10b76e54c"
Sep 17 16:44:38.217030 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:38.216976    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10d4be5037d5de276e1ef86c88fee787c2dd4ec2d1b82f21d4a4b8d10b76e54c"} err="failed to get container status \"10d4be5037d5de276e1ef86c88fee787c2dd4ec2d1b82f21d4a4b8d10b76e54c\": rpc error: code = NotFound desc = could not find container \"10d4be5037d5de276e1ef86c88fee787c2dd4ec2d1b82f21d4a4b8d10b76e54c\": container with ID starting with 10d4be5037d5de276e1ef86c88fee787c2dd4ec2d1b82f21d4a4b8d10b76e54c not found: ID does not exist"
Sep 17 16:44:38.217030 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:38.216995    2585 scope.go:117] "RemoveContainer" containerID="e8ff4cd2be5548aca717d5be4e0be5a0f252cd1954f810bece16f37f13701f6f"
Sep 17 16:44:38.217296 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:44:38.217280    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8ff4cd2be5548aca717d5be4e0be5a0f252cd1954f810bece16f37f13701f6f\": container with ID starting with e8ff4cd2be5548aca717d5be4e0be5a0f252cd1954f810bece16f37f13701f6f not found: ID does not exist" containerID="e8ff4cd2be5548aca717d5be4e0be5a0f252cd1954f810bece16f37f13701f6f"
Sep 17 16:44:38.217358 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:38.217303    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8ff4cd2be5548aca717d5be4e0be5a0f252cd1954f810bece16f37f13701f6f"} err="failed to get container status \"e8ff4cd2be5548aca717d5be4e0be5a0f252cd1954f810bece16f37f13701f6f\": rpc error: code = NotFound desc = could not find container \"e8ff4cd2be5548aca717d5be4e0be5a0f252cd1954f810bece16f37f13701f6f\": container with ID starting with e8ff4cd2be5548aca717d5be4e0be5a0f252cd1954f810bece16f37f13701f6f not found: ID does not exist"
Sep 17 16:44:38.217358 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:38.217320    2585 scope.go:117] "RemoveContainer" containerID="5a2ff47cef043baa940259284dfcecb6477a8c386283fdbee864c5b15852f4c6"
Sep 17 16:44:38.217686 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:44:38.217660    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a2ff47cef043baa940259284dfcecb6477a8c386283fdbee864c5b15852f4c6\": container with ID starting with 5a2ff47cef043baa940259284dfcecb6477a8c386283fdbee864c5b15852f4c6 not found: ID does not exist" containerID="5a2ff47cef043baa940259284dfcecb6477a8c386283fdbee864c5b15852f4c6"
Sep 17 16:44:38.217799 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:38.217697    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a2ff47cef043baa940259284dfcecb6477a8c386283fdbee864c5b15852f4c6"} err="failed to get container status \"5a2ff47cef043baa940259284dfcecb6477a8c386283fdbee864c5b15852f4c6\": rpc error: code = NotFound desc = could not find container \"5a2ff47cef043baa940259284dfcecb6477a8c386283fdbee864c5b15852f4c6\": container with ID starting with 5a2ff47cef043baa940259284dfcecb6477a8c386283fdbee864c5b15852f4c6 not found: ID does not exist"
Sep 17 16:44:38.264264 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:38.264180    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1d93ba27-19fe-4a3e-97e0-9c678991e614-kserve-provision-location\") pod \"1d93ba27-19fe-4a3e-97e0-9c678991e614\" (UID: \"1d93ba27-19fe-4a3e-97e0-9c678991e614\") "
Sep 17 16:44:38.264264 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:38.264229    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5c8h\" (UniqueName: \"kubernetes.io/projected/1d93ba27-19fe-4a3e-97e0-9c678991e614-kube-api-access-m5c8h\") pod \"1d93ba27-19fe-4a3e-97e0-9c678991e614\" (UID: \"1d93ba27-19fe-4a3e-97e0-9c678991e614\") "
Sep 17 16:44:38.264264 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:38.264258    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d93ba27-19fe-4a3e-97e0-9c678991e614-kube-rbac-proxy-sar-config\") pod \"1d93ba27-19fe-4a3e-97e0-9c678991e614\" (UID: \"1d93ba27-19fe-4a3e-97e0-9c678991e614\") "
Sep 17 16:44:38.264588 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:38.264280    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1d93ba27-19fe-4a3e-97e0-9c678991e614-cabundle-cert\") pod \"1d93ba27-19fe-4a3e-97e0-9c678991e614\" (UID: \"1d93ba27-19fe-4a3e-97e0-9c678991e614\") "
Sep 17 16:44:38.264588 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:38.264349    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d93ba27-19fe-4a3e-97e0-9c678991e614-proxy-tls\") pod \"1d93ba27-19fe-4a3e-97e0-9c678991e614\" (UID: \"1d93ba27-19fe-4a3e-97e0-9c678991e614\") "
Sep 17 16:44:38.264700 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:38.264581    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d93ba27-19fe-4a3e-97e0-9c678991e614-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1d93ba27-19fe-4a3e-97e0-9c678991e614" (UID: "1d93ba27-19fe-4a3e-97e0-9c678991e614"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:44:38.264700 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:38.264665    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1d93ba27-19fe-4a3e-97e0-9c678991e614-kserve-provision-location\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:44:38.264806 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:38.264698    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d93ba27-19fe-4a3e-97e0-9c678991e614-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1d93ba27-19fe-4a3e-97e0-9c678991e614" (UID: "1d93ba27-19fe-4a3e-97e0-9c678991e614"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:44:38.264806 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:38.264696    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d93ba27-19fe-4a3e-97e0-9c678991e614-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1d93ba27-19fe-4a3e-97e0-9c678991e614" (UID: "1d93ba27-19fe-4a3e-97e0-9c678991e614"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:44:38.266484 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:38.266462    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d93ba27-19fe-4a3e-97e0-9c678991e614-kube-api-access-m5c8h" (OuterVolumeSpecName: "kube-api-access-m5c8h") pod "1d93ba27-19fe-4a3e-97e0-9c678991e614" (UID: "1d93ba27-19fe-4a3e-97e0-9c678991e614"). InnerVolumeSpecName "kube-api-access-m5c8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:44:38.266573 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:38.266481    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d93ba27-19fe-4a3e-97e0-9c678991e614-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1d93ba27-19fe-4a3e-97e0-9c678991e614" (UID: "1d93ba27-19fe-4a3e-97e0-9c678991e614"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:44:38.366092 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:38.366055    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d93ba27-19fe-4a3e-97e0-9c678991e614-proxy-tls\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:44:38.366092 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:38.366090    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5c8h\" (UniqueName: \"kubernetes.io/projected/1d93ba27-19fe-4a3e-97e0-9c678991e614-kube-api-access-m5c8h\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:44:38.366092 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:38.366100    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d93ba27-19fe-4a3e-97e0-9c678991e614-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:44:38.366457 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:38.366118    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1d93ba27-19fe-4a3e-97e0-9c678991e614-cabundle-cert\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:44:38.515217 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:38.515116    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq"]
Sep 17 16:44:38.516458 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:38.516406    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-698dfcd8b4-bcszq"]
Sep 17 16:44:39.193492 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:39.193461    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-7b64946bf6-nz6tq_e537f5ca-8be5-41dc-8a42-5cb565779a37/storage-initializer/0.log"
Sep 17 16:44:39.193894 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:39.193546    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b64946bf6-nz6tq" event={"ID":"e537f5ca-8be5-41dc-8a42-5cb565779a37","Type":"ContainerStarted","Data":"8e3ae83f478dfa338884a283579bb831927360bf68ccb3e79de1a81b6720b63d"}
Sep 17 16:44:40.512927 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:40.512894    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d93ba27-19fe-4a3e-97e0-9c678991e614" path="/var/lib/kubelet/pods/1d93ba27-19fe-4a3e-97e0-9c678991e614/volumes"
Sep 17 16:44:44.038880 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:44.038791    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b64946bf6-nz6tq"]
Sep 17 16:44:44.039412 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:44.039017    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b64946bf6-nz6tq" podUID="e537f5ca-8be5-41dc-8a42-5cb565779a37" containerName="storage-initializer" containerID="cri-o://8e3ae83f478dfa338884a283579bb831927360bf68ccb3e79de1a81b6720b63d" gracePeriod=30
Sep 17 16:44:44.179128 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:44.179102    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-7b64946bf6-nz6tq_e537f5ca-8be5-41dc-8a42-5cb565779a37/storage-initializer/1.log"
Sep 17 16:44:44.179544 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:44.179525    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-7b64946bf6-nz6tq_e537f5ca-8be5-41dc-8a42-5cb565779a37/storage-initializer/0.log"
Sep 17 16:44:44.181926 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:44.181902    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b64946bf6-nz6tq"
Sep 17 16:44:44.218868 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:44.218836    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-7b64946bf6-nz6tq_e537f5ca-8be5-41dc-8a42-5cb565779a37/storage-initializer/1.log"
Sep 17 16:44:44.219299 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:44.219279    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-7b64946bf6-nz6tq_e537f5ca-8be5-41dc-8a42-5cb565779a37/storage-initializer/0.log"
Sep 17 16:44:44.219447 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:44.219324    2585 generic.go:356] "Generic (PLEG): container finished" podID="e537f5ca-8be5-41dc-8a42-5cb565779a37" containerID="8e3ae83f478dfa338884a283579bb831927360bf68ccb3e79de1a81b6720b63d" exitCode=1
Sep 17 16:44:44.219447 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:44.219378    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b64946bf6-nz6tq"
Sep 17 16:44:44.219447 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:44.219382    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b64946bf6-nz6tq" event={"ID":"e537f5ca-8be5-41dc-8a42-5cb565779a37","Type":"ContainerDied","Data":"8e3ae83f478dfa338884a283579bb831927360bf68ccb3e79de1a81b6720b63d"}
Sep 17 16:44:44.219630 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:44.219464    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b64946bf6-nz6tq" event={"ID":"e537f5ca-8be5-41dc-8a42-5cb565779a37","Type":"ContainerDied","Data":"cc76c76056a3e4528eb31338bc6fad2933d62c609806b4fdde20bbfb0e7911c2"}
Sep 17 16:44:44.219630 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:44.219508    2585 scope.go:117] "RemoveContainer" containerID="8e3ae83f478dfa338884a283579bb831927360bf68ccb3e79de1a81b6720b63d"
Sep 17 16:44:44.228222 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:44.228202    2585 scope.go:117] "RemoveContainer" containerID="99555302da10ff8e2075e847e88d373306b7477deb456e886389d5228a972d49"
Sep 17 16:44:44.236391 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:44.236368    2585 scope.go:117] "RemoveContainer" containerID="8e3ae83f478dfa338884a283579bb831927360bf68ccb3e79de1a81b6720b63d"
Sep 17 16:44:44.236730 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:44:44.236709    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e3ae83f478dfa338884a283579bb831927360bf68ccb3e79de1a81b6720b63d\": container with ID starting with 8e3ae83f478dfa338884a283579bb831927360bf68ccb3e79de1a81b6720b63d not found: ID does not exist" containerID="8e3ae83f478dfa338884a283579bb831927360bf68ccb3e79de1a81b6720b63d"
Sep 17 16:44:44.236807 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:44.236737    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e3ae83f478dfa338884a283579bb831927360bf68ccb3e79de1a81b6720b63d"} err="failed to get container status \"8e3ae83f478dfa338884a283579bb831927360bf68ccb3e79de1a81b6720b63d\": rpc error: code = NotFound desc = could not find container \"8e3ae83f478dfa338884a283579bb831927360bf68ccb3e79de1a81b6720b63d\": container with ID starting with 8e3ae83f478dfa338884a283579bb831927360bf68ccb3e79de1a81b6720b63d not found: ID does not exist"
Sep 17 16:44:44.236807 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:44.236756    2585 scope.go:117] "RemoveContainer" containerID="99555302da10ff8e2075e847e88d373306b7477deb456e886389d5228a972d49"
Sep 17 16:44:44.237048 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:44:44.237028    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99555302da10ff8e2075e847e88d373306b7477deb456e886389d5228a972d49\": container with ID starting with 99555302da10ff8e2075e847e88d373306b7477deb456e886389d5228a972d49 not found: ID does not exist" containerID="99555302da10ff8e2075e847e88d373306b7477deb456e886389d5228a972d49"
Sep 17 16:44:44.237105 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:44.237063    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99555302da10ff8e2075e847e88d373306b7477deb456e886389d5228a972d49"} err="failed to get container status \"99555302da10ff8e2075e847e88d373306b7477deb456e886389d5228a972d49\": rpc error: code = NotFound desc = could not find container \"99555302da10ff8e2075e847e88d373306b7477deb456e886389d5228a972d49\": container with ID starting with 99555302da10ff8e2075e847e88d373306b7477deb456e886389d5228a972d49 not found: ID does not exist"
Sep 17 16:44:44.320017 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:44.319921    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e537f5ca-8be5-41dc-8a42-5cb565779a37-kube-rbac-proxy-sar-config\") pod \"e537f5ca-8be5-41dc-8a42-5cb565779a37\" (UID: \"e537f5ca-8be5-41dc-8a42-5cb565779a37\") "
Sep 17 16:44:44.320017 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:44.319987    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r27zp\" (UniqueName: \"kubernetes.io/projected/e537f5ca-8be5-41dc-8a42-5cb565779a37-kube-api-access-r27zp\") pod \"e537f5ca-8be5-41dc-8a42-5cb565779a37\" (UID: \"e537f5ca-8be5-41dc-8a42-5cb565779a37\") "
Sep 17 16:44:44.320017 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:44.320010    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e537f5ca-8be5-41dc-8a42-5cb565779a37-kserve-provision-location\") pod \"e537f5ca-8be5-41dc-8a42-5cb565779a37\" (UID: \"e537f5ca-8be5-41dc-8a42-5cb565779a37\") "
Sep 17 16:44:44.320257 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:44.320051    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e537f5ca-8be5-41dc-8a42-5cb565779a37-proxy-tls\") pod \"e537f5ca-8be5-41dc-8a42-5cb565779a37\" (UID: \"e537f5ca-8be5-41dc-8a42-5cb565779a37\") "
Sep 17 16:44:44.320323 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:44.320296    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e537f5ca-8be5-41dc-8a42-5cb565779a37-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e537f5ca-8be5-41dc-8a42-5cb565779a37" (UID: "e537f5ca-8be5-41dc-8a42-5cb565779a37"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:44:44.320397 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:44.320373    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e537f5ca-8be5-41dc-8a42-5cb565779a37-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e537f5ca-8be5-41dc-8a42-5cb565779a37" (UID: "e537f5ca-8be5-41dc-8a42-5cb565779a37"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:44:44.322165 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:44.322139    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e537f5ca-8be5-41dc-8a42-5cb565779a37-kube-api-access-r27zp" (OuterVolumeSpecName: "kube-api-access-r27zp") pod "e537f5ca-8be5-41dc-8a42-5cb565779a37" (UID: "e537f5ca-8be5-41dc-8a42-5cb565779a37"). InnerVolumeSpecName "kube-api-access-r27zp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:44:44.322165 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:44.322152    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e537f5ca-8be5-41dc-8a42-5cb565779a37-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e537f5ca-8be5-41dc-8a42-5cb565779a37" (UID: "e537f5ca-8be5-41dc-8a42-5cb565779a37"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:44:44.421259 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:44.421213    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e537f5ca-8be5-41dc-8a42-5cb565779a37-proxy-tls\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:44:44.421259 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:44.421250    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e537f5ca-8be5-41dc-8a42-5cb565779a37-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:44:44.421259 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:44.421261    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r27zp\" (UniqueName: \"kubernetes.io/projected/e537f5ca-8be5-41dc-8a42-5cb565779a37-kube-api-access-r27zp\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:44:44.421259 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:44.421269    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e537f5ca-8be5-41dc-8a42-5cb565779a37-kserve-provision-location\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:44:44.553998 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:44.553953    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b64946bf6-nz6tq"]
Sep 17 16:44:44.558527 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:44.558488    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7b64946bf6-nz6tq"]
Sep 17 16:44:45.158115 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:45.158075    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p"]
Sep 17 16:44:45.158527 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:45.158479    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e537f5ca-8be5-41dc-8a42-5cb565779a37" containerName="storage-initializer"
Sep 17 16:44:45.158527 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:45.158493    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e537f5ca-8be5-41dc-8a42-5cb565779a37" containerName="storage-initializer"
Sep 17 16:44:45.158527 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:45.158505    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d93ba27-19fe-4a3e-97e0-9c678991e614" containerName="kserve-container"
Sep 17 16:44:45.158527 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:45.158514    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d93ba27-19fe-4a3e-97e0-9c678991e614" containerName="kserve-container"
Sep 17 16:44:45.158527 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:45.158527    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d93ba27-19fe-4a3e-97e0-9c678991e614" containerName="storage-initializer"
Sep 17 16:44:45.158699 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:45.158533    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d93ba27-19fe-4a3e-97e0-9c678991e614" containerName="storage-initializer"
Sep 17 16:44:45.158699 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:45.158549    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d93ba27-19fe-4a3e-97e0-9c678991e614" containerName="kube-rbac-proxy"
Sep 17 16:44:45.158699 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:45.158554    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d93ba27-19fe-4a3e-97e0-9c678991e614" containerName="kube-rbac-proxy"
Sep 17 16:44:45.158699 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:45.158605    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="e537f5ca-8be5-41dc-8a42-5cb565779a37" containerName="storage-initializer"
Sep 17 16:44:45.158699 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:45.158613    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d93ba27-19fe-4a3e-97e0-9c678991e614" containerName="kube-rbac-proxy"
Sep 17 16:44:45.158699 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:45.158623    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d93ba27-19fe-4a3e-97e0-9c678991e614" containerName="kserve-container"
Sep 17 16:44:45.158699 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:45.158629    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="e537f5ca-8be5-41dc-8a42-5cb565779a37" containerName="storage-initializer"
Sep 17 16:44:45.158699 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:45.158679    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e537f5ca-8be5-41dc-8a42-5cb565779a37" containerName="storage-initializer"
Sep 17 16:44:45.158699 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:45.158686    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e537f5ca-8be5-41dc-8a42-5cb565779a37" containerName="storage-initializer"
Sep 17 16:44:45.163678 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:45.163656    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p"
Sep 17 16:44:45.166954 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:45.166932    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xkc4g\""
Sep 17 16:44:45.166954 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:45.166943    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 17 16:44:45.167147 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:45.166935    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
Sep 17 16:44:45.168009 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:45.167987    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:44:45.168161 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:45.168059    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
Sep 17 16:44:45.168161 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:45.168027    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 16:44:45.168310 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:45.167990    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 16:44:45.177285 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:45.177257    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p"]
Sep 17 16:44:45.329601 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:45.329560    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7b59977-2e71-4ea9-85b5-3b2ed101b446-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p\" (UID: \"f7b59977-2e71-4ea9-85b5-3b2ed101b446\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p"
Sep 17 16:44:45.329789 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:45.329618    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7b59977-2e71-4ea9-85b5-3b2ed101b446-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p\" (UID: \"f7b59977-2e71-4ea9-85b5-3b2ed101b446\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p"
Sep 17 16:44:45.329789 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:45.329758    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7b59977-2e71-4ea9-85b5-3b2ed101b446-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p\" (UID: \"f7b59977-2e71-4ea9-85b5-3b2ed101b446\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p"
Sep 17 16:44:45.329867 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:45.329793    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7b59977-2e71-4ea9-85b5-3b2ed101b446-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p\" (UID: \"f7b59977-2e71-4ea9-85b5-3b2ed101b446\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p"
Sep 17 16:44:45.329867 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:45.329819    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fmjj\" (UniqueName: \"kubernetes.io/projected/f7b59977-2e71-4ea9-85b5-3b2ed101b446-kube-api-access-8fmjj\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p\" (UID: \"f7b59977-2e71-4ea9-85b5-3b2ed101b446\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p"
Sep 17 16:44:45.431098 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:45.430991    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7b59977-2e71-4ea9-85b5-3b2ed101b446-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p\" (UID: \"f7b59977-2e71-4ea9-85b5-3b2ed101b446\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p"
Sep 17 16:44:45.431098 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:45.431040    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7b59977-2e71-4ea9-85b5-3b2ed101b446-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p\" (UID: \"f7b59977-2e71-4ea9-85b5-3b2ed101b446\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p"
Sep 17 16:44:45.431098 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:45.431060    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fmjj\" (UniqueName: \"kubernetes.io/projected/f7b59977-2e71-4ea9-85b5-3b2ed101b446-kube-api-access-8fmjj\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p\" (UID: \"f7b59977-2e71-4ea9-85b5-3b2ed101b446\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p"
Sep 17 16:44:45.431098 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:45.431084    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7b59977-2e71-4ea9-85b5-3b2ed101b446-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p\" (UID: \"f7b59977-2e71-4ea9-85b5-3b2ed101b446\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p"
Sep 17 16:44:45.431503 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:45.431110    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7b59977-2e71-4ea9-85b5-3b2ed101b446-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p\" (UID: \"f7b59977-2e71-4ea9-85b5-3b2ed101b446\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p"
Sep 17 16:44:45.431559 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:45.431530    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7b59977-2e71-4ea9-85b5-3b2ed101b446-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p\" (UID: \"f7b59977-2e71-4ea9-85b5-3b2ed101b446\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p"
Sep 17 16:44:45.431837 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:45.431815    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7b59977-2e71-4ea9-85b5-3b2ed101b446-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p\" (UID: \"f7b59977-2e71-4ea9-85b5-3b2ed101b446\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p"
Sep 17 16:44:45.431878 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:45.431822    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7b59977-2e71-4ea9-85b5-3b2ed101b446-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p\" (UID: \"f7b59977-2e71-4ea9-85b5-3b2ed101b446\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p"
Sep 17 16:44:45.433647 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:45.433626    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7b59977-2e71-4ea9-85b5-3b2ed101b446-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p\" (UID: \"f7b59977-2e71-4ea9-85b5-3b2ed101b446\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p"
Sep 17 16:44:45.440291 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:45.440264    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fmjj\" (UniqueName: \"kubernetes.io/projected/f7b59977-2e71-4ea9-85b5-3b2ed101b446-kube-api-access-8fmjj\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p\" (UID: \"f7b59977-2e71-4ea9-85b5-3b2ed101b446\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p"
Sep 17 16:44:45.475090 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:45.475045    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p"
Sep 17 16:44:45.754779 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:45.754702    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p"]
Sep 17 16:44:45.757254 ip-10-0-135-43 kubenswrapper[2585]: W0917 16:44:45.757210    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7b59977_2e71_4ea9_85b5_3b2ed101b446.slice/crio-48491d08502867f635c67fa171f51ee14df3c13996a72ce91092a0af2cd8386e WatchSource:0}: Error finding container 48491d08502867f635c67fa171f51ee14df3c13996a72ce91092a0af2cd8386e: Status 404 returned error can't find the container with id 48491d08502867f635c67fa171f51ee14df3c13996a72ce91092a0af2cd8386e
Sep 17 16:44:46.232094 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:46.232056    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p" event={"ID":"f7b59977-2e71-4ea9-85b5-3b2ed101b446","Type":"ContainerStarted","Data":"63224633e90c64e6456bc23888145affa5ad20c62809133fc4edb2322e68a36e"}
Sep 17 16:44:46.232516 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:46.232116    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p" event={"ID":"f7b59977-2e71-4ea9-85b5-3b2ed101b446","Type":"ContainerStarted","Data":"48491d08502867f635c67fa171f51ee14df3c13996a72ce91092a0af2cd8386e"}
Sep 17 16:44:46.512638 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:46.512545    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e537f5ca-8be5-41dc-8a42-5cb565779a37" path="/var/lib/kubelet/pods/e537f5ca-8be5-41dc-8a42-5cb565779a37/volumes"
Sep 17 16:44:47.237914 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:47.237877    2585 generic.go:356] "Generic (PLEG): container finished" podID="f7b59977-2e71-4ea9-85b5-3b2ed101b446" containerID="63224633e90c64e6456bc23888145affa5ad20c62809133fc4edb2322e68a36e" exitCode=0
Sep 17 16:44:47.238314 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:47.237925    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p" event={"ID":"f7b59977-2e71-4ea9-85b5-3b2ed101b446","Type":"ContainerDied","Data":"63224633e90c64e6456bc23888145affa5ad20c62809133fc4edb2322e68a36e"}
Sep 17 16:44:47.828594 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:47.828566    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2b47p_a1850a4d-4d01-4a71-8636-b5918f1426c2/ovn-acl-logging/0.log"
Sep 17 16:44:47.835695 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:47.835669    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2b47p_a1850a4d-4d01-4a71-8636-b5918f1426c2/ovn-acl-logging/0.log"
Sep 17 16:44:48.244940 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:48.244902    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p" event={"ID":"f7b59977-2e71-4ea9-85b5-3b2ed101b446","Type":"ContainerStarted","Data":"188cd640d7f515ff7ef4a6dc2e88db622c51fef613513bcbd6fc705f448608ed"}
Sep 17 16:44:48.245356 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:48.244965    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p" event={"ID":"f7b59977-2e71-4ea9-85b5-3b2ed101b446","Type":"ContainerStarted","Data":"bad62fc651aeb7fe2fbf3a595ab895e29568479203a03ea081b1ca38ee9700ed"}
Sep 17 16:44:48.245356 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:48.245072    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p"
Sep 17 16:44:48.266971 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:48.266917    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p" podStartSLOduration=2.054153694 podStartE2EDuration="3.26689812s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.212744426s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:44:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:44:48.265339096 +0000 UTC m=+3044.035670897" watchObservedRunningTime="2026-09-17 16:44:48.26689812 +0000 UTC m=+3044.037229915"
Sep 17 16:44:49.249642 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:49.249606    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p"
Sep 17 16:44:49.250985 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:49.250959    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p" podUID="f7b59977-2e71-4ea9-85b5-3b2ed101b446" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 17 16:44:50.254412 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:50.254365    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p" podUID="f7b59977-2e71-4ea9-85b5-3b2ed101b446" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 17 16:44:55.259096 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:55.259065    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p"
Sep 17 16:44:55.259613 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:44:55.259585    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p" podUID="f7b59977-2e71-4ea9-85b5-3b2ed101b446" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 17 16:45:05.260356 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:45:05.260313    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p" podUID="f7b59977-2e71-4ea9-85b5-3b2ed101b446" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 17 16:45:15.259682 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:45:15.259639    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p" podUID="f7b59977-2e71-4ea9-85b5-3b2ed101b446" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 17 16:45:25.259929 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:45:25.259891    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p" podUID="f7b59977-2e71-4ea9-85b5-3b2ed101b446" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 17 16:45:35.259993 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:45:35.259949    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p" podUID="f7b59977-2e71-4ea9-85b5-3b2ed101b446" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 17 16:45:35.279027 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:45:35.278988    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f5zd5"]
Sep 17 16:45:35.284076 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:45:35.284053    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f5zd5"
Sep 17 16:45:35.286785 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:45:35.286758    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-predictor-serving-cert\""
Sep 17 16:45:35.286940 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:45:35.286918    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
Sep 17 16:45:35.287160 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:45:35.287145    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 16:45:35.297143 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:45:35.297107    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f5zd5"]
Sep 17 16:45:35.367922 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:45:35.367883    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc3cce66-9206-4ea3-b7e1-7f85aded460b-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-f5zd5\" (UID: \"fc3cce66-9206-4ea3-b7e1-7f85aded460b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f5zd5"
Sep 17 16:45:35.368106 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:45:35.367941    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc3cce66-9206-4ea3-b7e1-7f85aded460b-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-f5zd5\" (UID: \"fc3cce66-9206-4ea3-b7e1-7f85aded460b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f5zd5"
Sep 17 16:45:35.368106 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:45:35.367980    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p267x\" (UniqueName: \"kubernetes.io/projected/fc3cce66-9206-4ea3-b7e1-7f85aded460b-kube-api-access-p267x\") pod \"isvc-paddle-runtime-predictor-58b99fff76-f5zd5\" (UID: \"fc3cce66-9206-4ea3-b7e1-7f85aded460b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f5zd5"
Sep 17 16:45:35.368106 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:45:35.368005    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc3cce66-9206-4ea3-b7e1-7f85aded460b-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-f5zd5\" (UID: \"fc3cce66-9206-4ea3-b7e1-7f85aded460b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f5zd5"
Sep 17 16:45:35.368106 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:45:35.368093    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc3cce66-9206-4ea3-b7e1-7f85aded460b-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-f5zd5\" (UID: \"fc3cce66-9206-4ea3-b7e1-7f85aded460b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f5zd5"
Sep 17 16:45:35.469577 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:45:35.469543    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc3cce66-9206-4ea3-b7e1-7f85aded460b-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-f5zd5\" (UID: \"fc3cce66-9206-4ea3-b7e1-7f85aded460b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f5zd5"
Sep 17 16:45:35.469752 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:45:35.469589    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc3cce66-9206-4ea3-b7e1-7f85aded460b-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-f5zd5\" (UID: \"fc3cce66-9206-4ea3-b7e1-7f85aded460b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f5zd5"
Sep 17 16:45:35.469752 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:45:35.469685    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p267x\" (UniqueName: \"kubernetes.io/projected/fc3cce66-9206-4ea3-b7e1-7f85aded460b-kube-api-access-p267x\") pod \"isvc-paddle-runtime-predictor-58b99fff76-f5zd5\" (UID: \"fc3cce66-9206-4ea3-b7e1-7f85aded460b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f5zd5"
Sep 17 16:45:35.469752 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:45:35.469705    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc3cce66-9206-4ea3-b7e1-7f85aded460b-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-f5zd5\" (UID: \"fc3cce66-9206-4ea3-b7e1-7f85aded460b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f5zd5"
Sep 17 16:45:35.469752 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:45:35.469734    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc3cce66-9206-4ea3-b7e1-7f85aded460b-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-f5zd5\" (UID: \"fc3cce66-9206-4ea3-b7e1-7f85aded460b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f5zd5"
Sep 17 16:45:35.470009 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:45:35.469985    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc3cce66-9206-4ea3-b7e1-7f85aded460b-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-f5zd5\" (UID: \"fc3cce66-9206-4ea3-b7e1-7f85aded460b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f5zd5"
Sep 17 16:45:35.470381 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:45:35.470354    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc3cce66-9206-4ea3-b7e1-7f85aded460b-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-f5zd5\" (UID: \"fc3cce66-9206-4ea3-b7e1-7f85aded460b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f5zd5"
Sep 17 16:45:35.470491 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:45:35.470397    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc3cce66-9206-4ea3-b7e1-7f85aded460b-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-f5zd5\" (UID: \"fc3cce66-9206-4ea3-b7e1-7f85aded460b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f5zd5"
Sep 17 16:45:35.472529 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:45:35.472500    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc3cce66-9206-4ea3-b7e1-7f85aded460b-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-f5zd5\" (UID: \"fc3cce66-9206-4ea3-b7e1-7f85aded460b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f5zd5"
Sep 17 16:45:35.478711 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:45:35.478683    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p267x\" (UniqueName: \"kubernetes.io/projected/fc3cce66-9206-4ea3-b7e1-7f85aded460b-kube-api-access-p267x\") pod \"isvc-paddle-runtime-predictor-58b99fff76-f5zd5\" (UID: \"fc3cce66-9206-4ea3-b7e1-7f85aded460b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f5zd5"
Sep 17 16:45:35.594327 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:45:35.594237    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f5zd5"
Sep 17 16:45:35.869076 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:45:35.869046    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f5zd5"]
Sep 17 16:45:35.871573 ip-10-0-135-43 kubenswrapper[2585]: W0917 16:45:35.871542    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc3cce66_9206_4ea3_b7e1_7f85aded460b.slice/crio-6a3e9d5c82194826e31a640464f7f78a6b4df754c62be57419a195cab5f28baa WatchSource:0}: Error finding container 6a3e9d5c82194826e31a640464f7f78a6b4df754c62be57419a195cab5f28baa: Status 404 returned error can't find the container with id 6a3e9d5c82194826e31a640464f7f78a6b4df754c62be57419a195cab5f28baa
Sep 17 16:45:36.471329 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:45:36.471283    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f5zd5" event={"ID":"fc3cce66-9206-4ea3-b7e1-7f85aded460b","Type":"ContainerStarted","Data":"3962a930c7c8f86653a650a867657c00de24dfd8a1c54f0cdb7a55352bd7f986"}
Sep 17 16:45:36.471757 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:45:36.471365    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f5zd5" event={"ID":"fc3cce66-9206-4ea3-b7e1-7f85aded460b","Type":"ContainerStarted","Data":"6a3e9d5c82194826e31a640464f7f78a6b4df754c62be57419a195cab5f28baa"}
Sep 17 16:45:40.491227 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:45:40.491195    2585 generic.go:356] "Generic (PLEG): container finished" podID="fc3cce66-9206-4ea3-b7e1-7f85aded460b" containerID="3962a930c7c8f86653a650a867657c00de24dfd8a1c54f0cdb7a55352bd7f986" exitCode=0
Sep 17 16:45:40.491615 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:45:40.491232    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f5zd5" event={"ID":"fc3cce66-9206-4ea3-b7e1-7f85aded460b","Type":"ContainerDied","Data":"3962a930c7c8f86653a650a867657c00de24dfd8a1c54f0cdb7a55352bd7f986"}
Sep 17 16:45:40.492518 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:45:40.492500    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 16:45:45.260442 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:45:45.260382    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p" podUID="f7b59977-2e71-4ea9-85b5-3b2ed101b446" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 17 16:45:52.557781 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:45:52.557733    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f5zd5" event={"ID":"fc3cce66-9206-4ea3-b7e1-7f85aded460b","Type":"ContainerStarted","Data":"e9192dfb499648479a3869e8e9582817d9551d97fa9c87a09c54827b4de463a7"}
Sep 17 16:45:52.558236 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:45:52.557807    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f5zd5" event={"ID":"fc3cce66-9206-4ea3-b7e1-7f85aded460b","Type":"ContainerStarted","Data":"530f8ed817ad4e7a45904e6940b3cda94caedf323118b7774c6bf94acb85ab0d"}
Sep 17 16:45:52.558236 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:45:52.558144    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f5zd5"
Sep 17 16:45:52.558236 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:45:52.558197    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f5zd5"
Sep 17 16:45:52.559608 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:45:52.559579    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f5zd5" podUID="fc3cce66-9206-4ea3-b7e1-7f85aded460b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 17 16:45:52.579657 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:45:52.579597    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f5zd5" podStartSLOduration=2.069857981 podStartE2EDuration="17.57957954s" totalImagesPullingTime="10.98881123s" totalInitContainerRuntime="4.520910329s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:45:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:45:52.577942852 +0000 UTC m=+3108.348274676" watchObservedRunningTime="2026-09-17 16:45:52.57957954 +0000 UTC m=+3108.349911338"
Sep 17 16:45:53.562931 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:45:53.562893    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f5zd5" podUID="fc3cce66-9206-4ea3-b7e1-7f85aded460b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 17 16:45:55.259585 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:45:55.259545    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p" podUID="f7b59977-2e71-4ea9-85b5-3b2ed101b446" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 17 16:45:58.567975 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:45:58.567945    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f5zd5"
Sep 17 16:45:58.568459 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:45:58.568406    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f5zd5" podUID="fc3cce66-9206-4ea3-b7e1-7f85aded460b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 17 16:46:05.260310 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:05.260273    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p" podUID="f7b59977-2e71-4ea9-85b5-3b2ed101b446" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 17 16:46:08.568960 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:08.568915    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f5zd5" podUID="fc3cce66-9206-4ea3-b7e1-7f85aded460b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 17 16:46:15.260308 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:15.260274    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p"
Sep 17 16:46:18.569083 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:18.569001    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f5zd5" podUID="fc3cce66-9206-4ea3-b7e1-7f85aded460b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 17 16:46:25.211694 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:25.211656    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p"]
Sep 17 16:46:25.213115 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:25.211910    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p" podUID="f7b59977-2e71-4ea9-85b5-3b2ed101b446" containerName="kserve-container" containerID="cri-o://bad62fc651aeb7fe2fbf3a595ab895e29568479203a03ea081b1ca38ee9700ed" gracePeriod=30
Sep 17 16:46:25.213115 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:25.211954    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p" podUID="f7b59977-2e71-4ea9-85b5-3b2ed101b446" containerName="kube-rbac-proxy" containerID="cri-o://188cd640d7f515ff7ef4a6dc2e88db622c51fef613513bcbd6fc705f448608ed" gracePeriod=30
Sep 17 16:46:25.255037 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:25.254997    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p" podUID="f7b59977-2e71-4ea9-85b5-3b2ed101b446" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.64:8643/healthz\": dial tcp 10.134.0.64:8643: connect: connection refused"
Sep 17 16:46:25.260378 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:25.260342    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p" podUID="f7b59977-2e71-4ea9-85b5-3b2ed101b446" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Sep 17 16:46:25.721946 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:25.721912    2585 generic.go:356] "Generic (PLEG): container finished" podID="f7b59977-2e71-4ea9-85b5-3b2ed101b446" containerID="188cd640d7f515ff7ef4a6dc2e88db622c51fef613513bcbd6fc705f448608ed" exitCode=2
Sep 17 16:46:25.722147 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:25.721986    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p" event={"ID":"f7b59977-2e71-4ea9-85b5-3b2ed101b446","Type":"ContainerDied","Data":"188cd640d7f515ff7ef4a6dc2e88db622c51fef613513bcbd6fc705f448608ed"}
Sep 17 16:46:28.568616 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:28.568563    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f5zd5" podUID="fc3cce66-9206-4ea3-b7e1-7f85aded460b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 17 16:46:30.159913 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:30.159888    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p"
Sep 17 16:46:30.233559 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:30.233524    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7b59977-2e71-4ea9-85b5-3b2ed101b446-kserve-provision-location\") pod \"f7b59977-2e71-4ea9-85b5-3b2ed101b446\" (UID: \"f7b59977-2e71-4ea9-85b5-3b2ed101b446\") "
Sep 17 16:46:30.233753 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:30.233582    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7b59977-2e71-4ea9-85b5-3b2ed101b446-proxy-tls\") pod \"f7b59977-2e71-4ea9-85b5-3b2ed101b446\" (UID: \"f7b59977-2e71-4ea9-85b5-3b2ed101b446\") "
Sep 17 16:46:30.233753 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:30.233603    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8fmjj\" (UniqueName: \"kubernetes.io/projected/f7b59977-2e71-4ea9-85b5-3b2ed101b446-kube-api-access-8fmjj\") pod \"f7b59977-2e71-4ea9-85b5-3b2ed101b446\" (UID: \"f7b59977-2e71-4ea9-85b5-3b2ed101b446\") "
Sep 17 16:46:30.233753 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:30.233672    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7b59977-2e71-4ea9-85b5-3b2ed101b446-cabundle-cert\") pod \"f7b59977-2e71-4ea9-85b5-3b2ed101b446\" (UID: \"f7b59977-2e71-4ea9-85b5-3b2ed101b446\") "
Sep 17 16:46:30.233753 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:30.233697    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7b59977-2e71-4ea9-85b5-3b2ed101b446-kube-rbac-proxy-sar-config\") pod \"f7b59977-2e71-4ea9-85b5-3b2ed101b446\" (UID: \"f7b59977-2e71-4ea9-85b5-3b2ed101b446\") "
Sep 17 16:46:30.234020 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:30.233919    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7b59977-2e71-4ea9-85b5-3b2ed101b446-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f7b59977-2e71-4ea9-85b5-3b2ed101b446" (UID: "f7b59977-2e71-4ea9-85b5-3b2ed101b446"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:46:30.234141 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:30.234116    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7b59977-2e71-4ea9-85b5-3b2ed101b446-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f7b59977-2e71-4ea9-85b5-3b2ed101b446" (UID: "f7b59977-2e71-4ea9-85b5-3b2ed101b446"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:46:30.234188 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:30.234151    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7b59977-2e71-4ea9-85b5-3b2ed101b446-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f7b59977-2e71-4ea9-85b5-3b2ed101b446" (UID: "f7b59977-2e71-4ea9-85b5-3b2ed101b446"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:46:30.235780 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:30.235760    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7b59977-2e71-4ea9-85b5-3b2ed101b446-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f7b59977-2e71-4ea9-85b5-3b2ed101b446" (UID: "f7b59977-2e71-4ea9-85b5-3b2ed101b446"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:46:30.235879 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:30.235862    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7b59977-2e71-4ea9-85b5-3b2ed101b446-kube-api-access-8fmjj" (OuterVolumeSpecName: "kube-api-access-8fmjj") pod "f7b59977-2e71-4ea9-85b5-3b2ed101b446" (UID: "f7b59977-2e71-4ea9-85b5-3b2ed101b446"). InnerVolumeSpecName "kube-api-access-8fmjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:46:30.335029 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:30.334985    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7b59977-2e71-4ea9-85b5-3b2ed101b446-cabundle-cert\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:46:30.335029 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:30.335023    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7b59977-2e71-4ea9-85b5-3b2ed101b446-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:46:30.335029 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:30.335033    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7b59977-2e71-4ea9-85b5-3b2ed101b446-kserve-provision-location\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:46:30.335271 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:30.335043    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7b59977-2e71-4ea9-85b5-3b2ed101b446-proxy-tls\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:46:30.335271 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:30.335053    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fmjj\" (UniqueName: \"kubernetes.io/projected/f7b59977-2e71-4ea9-85b5-3b2ed101b446-kube-api-access-8fmjj\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:46:30.746928 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:30.746893    2585 generic.go:356] "Generic (PLEG): container finished" podID="f7b59977-2e71-4ea9-85b5-3b2ed101b446" containerID="bad62fc651aeb7fe2fbf3a595ab895e29568479203a03ea081b1ca38ee9700ed" exitCode=0
Sep 17 16:46:30.747132 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:30.746949    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p" event={"ID":"f7b59977-2e71-4ea9-85b5-3b2ed101b446","Type":"ContainerDied","Data":"bad62fc651aeb7fe2fbf3a595ab895e29568479203a03ea081b1ca38ee9700ed"}
Sep 17 16:46:30.747132 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:30.746955    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p"
Sep 17 16:46:30.747132 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:30.746998    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p" event={"ID":"f7b59977-2e71-4ea9-85b5-3b2ed101b446","Type":"ContainerDied","Data":"48491d08502867f635c67fa171f51ee14df3c13996a72ce91092a0af2cd8386e"}
Sep 17 16:46:30.747132 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:30.747025    2585 scope.go:117] "RemoveContainer" containerID="188cd640d7f515ff7ef4a6dc2e88db622c51fef613513bcbd6fc705f448608ed"
Sep 17 16:46:30.756515 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:30.756495    2585 scope.go:117] "RemoveContainer" containerID="bad62fc651aeb7fe2fbf3a595ab895e29568479203a03ea081b1ca38ee9700ed"
Sep 17 16:46:30.764560 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:30.764538    2585 scope.go:117] "RemoveContainer" containerID="63224633e90c64e6456bc23888145affa5ad20c62809133fc4edb2322e68a36e"
Sep 17 16:46:30.768781 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:30.768757    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p"]
Sep 17 16:46:30.775253 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:30.773735    2585 scope.go:117] "RemoveContainer" containerID="188cd640d7f515ff7ef4a6dc2e88db622c51fef613513bcbd6fc705f448608ed"
Sep 17 16:46:30.775253 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:46:30.774158    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"188cd640d7f515ff7ef4a6dc2e88db622c51fef613513bcbd6fc705f448608ed\": container with ID starting with 188cd640d7f515ff7ef4a6dc2e88db622c51fef613513bcbd6fc705f448608ed not found: ID does not exist" containerID="188cd640d7f515ff7ef4a6dc2e88db622c51fef613513bcbd6fc705f448608ed"
Sep 17 16:46:30.775253 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:30.774189    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"188cd640d7f515ff7ef4a6dc2e88db622c51fef613513bcbd6fc705f448608ed"} err="failed to get container status \"188cd640d7f515ff7ef4a6dc2e88db622c51fef613513bcbd6fc705f448608ed\": rpc error: code = NotFound desc = could not find container \"188cd640d7f515ff7ef4a6dc2e88db622c51fef613513bcbd6fc705f448608ed\": container with ID starting with 188cd640d7f515ff7ef4a6dc2e88db622c51fef613513bcbd6fc705f448608ed not found: ID does not exist"
Sep 17 16:46:30.775253 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:30.774214    2585 scope.go:117] "RemoveContainer" containerID="bad62fc651aeb7fe2fbf3a595ab895e29568479203a03ea081b1ca38ee9700ed"
Sep 17 16:46:30.775253 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:46:30.774567    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bad62fc651aeb7fe2fbf3a595ab895e29568479203a03ea081b1ca38ee9700ed\": container with ID starting with bad62fc651aeb7fe2fbf3a595ab895e29568479203a03ea081b1ca38ee9700ed not found: ID does not exist" containerID="bad62fc651aeb7fe2fbf3a595ab895e29568479203a03ea081b1ca38ee9700ed"
Sep 17 16:46:30.775253 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:30.774593    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bad62fc651aeb7fe2fbf3a595ab895e29568479203a03ea081b1ca38ee9700ed"} err="failed to get container status \"bad62fc651aeb7fe2fbf3a595ab895e29568479203a03ea081b1ca38ee9700ed\": rpc error: code = NotFound desc = could not find container \"bad62fc651aeb7fe2fbf3a595ab895e29568479203a03ea081b1ca38ee9700ed\": container with ID starting with bad62fc651aeb7fe2fbf3a595ab895e29568479203a03ea081b1ca38ee9700ed not found: ID does not exist"
Sep 17 16:46:30.775253 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:30.774615    2585 scope.go:117] "RemoveContainer" containerID="63224633e90c64e6456bc23888145affa5ad20c62809133fc4edb2322e68a36e"
Sep 17 16:46:30.775253 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:46:30.775035    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63224633e90c64e6456bc23888145affa5ad20c62809133fc4edb2322e68a36e\": container with ID starting with 63224633e90c64e6456bc23888145affa5ad20c62809133fc4edb2322e68a36e not found: ID does not exist" containerID="63224633e90c64e6456bc23888145affa5ad20c62809133fc4edb2322e68a36e"
Sep 17 16:46:30.775253 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:30.775056    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63224633e90c64e6456bc23888145affa5ad20c62809133fc4edb2322e68a36e"} err="failed to get container status \"63224633e90c64e6456bc23888145affa5ad20c62809133fc4edb2322e68a36e\": rpc error: code = NotFound desc = could not find container \"63224633e90c64e6456bc23888145affa5ad20c62809133fc4edb2322e68a36e\": container with ID starting with 63224633e90c64e6456bc23888145affa5ad20c62809133fc4edb2322e68a36e not found: ID does not exist"
Sep 17 16:46:30.776332 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:30.776309    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-9698964dc-vmt7p"]
Sep 17 16:46:32.512197 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:32.512164    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7b59977-2e71-4ea9-85b5-3b2ed101b446" path="/var/lib/kubelet/pods/f7b59977-2e71-4ea9-85b5-3b2ed101b446/volumes"
Sep 17 16:46:37.667088 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:37.667049    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-dmd9t"]
Sep 17 16:46:37.667537 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:37.667409    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7b59977-2e71-4ea9-85b5-3b2ed101b446" containerName="kube-rbac-proxy"
Sep 17 16:46:37.667537 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:37.667438    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7b59977-2e71-4ea9-85b5-3b2ed101b446" containerName="kube-rbac-proxy"
Sep 17 16:46:37.667537 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:37.667452    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7b59977-2e71-4ea9-85b5-3b2ed101b446" containerName="storage-initializer"
Sep 17 16:46:37.667537 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:37.667458    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7b59977-2e71-4ea9-85b5-3b2ed101b446" containerName="storage-initializer"
Sep 17 16:46:37.667537 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:37.667467    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7b59977-2e71-4ea9-85b5-3b2ed101b446" containerName="kserve-container"
Sep 17 16:46:37.667537 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:37.667473    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7b59977-2e71-4ea9-85b5-3b2ed101b446" containerName="kserve-container"
Sep 17 16:46:37.667537 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:37.667529    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7b59977-2e71-4ea9-85b5-3b2ed101b446" containerName="kserve-container"
Sep 17 16:46:37.667537 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:37.667537    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7b59977-2e71-4ea9-85b5-3b2ed101b446" containerName="kube-rbac-proxy"
Sep 17 16:46:37.670774 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:37.670755    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-dmd9t"
Sep 17 16:46:37.673589 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:37.673565    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-predictor-serving-cert\""
Sep 17 16:46:37.674166 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:37.674147    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
Sep 17 16:46:37.683029 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:37.683004    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-dmd9t"]
Sep 17 16:46:37.800563 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:37.800521    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a53935f-ef87-4850-a1b4-e5d9a31f4355-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-dmd9t\" (UID: \"1a53935f-ef87-4850-a1b4-e5d9a31f4355\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-dmd9t"
Sep 17 16:46:37.800563 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:37.800573    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a53935f-ef87-4850-a1b4-e5d9a31f4355-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-dmd9t\" (UID: \"1a53935f-ef87-4850-a1b4-e5d9a31f4355\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-dmd9t"
Sep 17 16:46:37.800847 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:37.800670    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2vbr\" (UniqueName: \"kubernetes.io/projected/1a53935f-ef87-4850-a1b4-e5d9a31f4355-kube-api-access-n2vbr\") pod \"isvc-pmml-runtime-predictor-885fc8777-dmd9t\" (UID: \"1a53935f-ef87-4850-a1b4-e5d9a31f4355\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-dmd9t"
Sep 17 16:46:37.800847 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:37.800739    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a53935f-ef87-4850-a1b4-e5d9a31f4355-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-dmd9t\" (UID: \"1a53935f-ef87-4850-a1b4-e5d9a31f4355\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-dmd9t"
Sep 17 16:46:37.800847 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:37.800767    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a53935f-ef87-4850-a1b4-e5d9a31f4355-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-dmd9t\" (UID: \"1a53935f-ef87-4850-a1b4-e5d9a31f4355\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-dmd9t"
Sep 17 16:46:37.902085 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:37.902047    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a53935f-ef87-4850-a1b4-e5d9a31f4355-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-dmd9t\" (UID: \"1a53935f-ef87-4850-a1b4-e5d9a31f4355\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-dmd9t"
Sep 17 16:46:37.902085 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:37.902086    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a53935f-ef87-4850-a1b4-e5d9a31f4355-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-dmd9t\" (UID: \"1a53935f-ef87-4850-a1b4-e5d9a31f4355\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-dmd9t"
Sep 17 16:46:37.902337 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:37.902119    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a53935f-ef87-4850-a1b4-e5d9a31f4355-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-dmd9t\" (UID: \"1a53935f-ef87-4850-a1b4-e5d9a31f4355\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-dmd9t"
Sep 17 16:46:37.902337 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:37.902139    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a53935f-ef87-4850-a1b4-e5d9a31f4355-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-dmd9t\" (UID: \"1a53935f-ef87-4850-a1b4-e5d9a31f4355\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-dmd9t"
Sep 17 16:46:37.902337 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:37.902178    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2vbr\" (UniqueName: \"kubernetes.io/projected/1a53935f-ef87-4850-a1b4-e5d9a31f4355-kube-api-access-n2vbr\") pod \"isvc-pmml-runtime-predictor-885fc8777-dmd9t\" (UID: \"1a53935f-ef87-4850-a1b4-e5d9a31f4355\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-dmd9t"
Sep 17 16:46:37.902660 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:37.902635    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a53935f-ef87-4850-a1b4-e5d9a31f4355-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-dmd9t\" (UID: \"1a53935f-ef87-4850-a1b4-e5d9a31f4355\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-dmd9t"
Sep 17 16:46:37.902951 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:37.902929    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a53935f-ef87-4850-a1b4-e5d9a31f4355-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-dmd9t\" (UID: \"1a53935f-ef87-4850-a1b4-e5d9a31f4355\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-dmd9t"
Sep 17 16:46:37.902997 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:37.902929    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a53935f-ef87-4850-a1b4-e5d9a31f4355-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-dmd9t\" (UID: \"1a53935f-ef87-4850-a1b4-e5d9a31f4355\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-dmd9t"
Sep 17 16:46:37.904836 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:37.904813    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a53935f-ef87-4850-a1b4-e5d9a31f4355-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-dmd9t\" (UID: \"1a53935f-ef87-4850-a1b4-e5d9a31f4355\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-dmd9t"
Sep 17 16:46:37.920706 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:37.920640    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2vbr\" (UniqueName: \"kubernetes.io/projected/1a53935f-ef87-4850-a1b4-e5d9a31f4355-kube-api-access-n2vbr\") pod \"isvc-pmml-runtime-predictor-885fc8777-dmd9t\" (UID: \"1a53935f-ef87-4850-a1b4-e5d9a31f4355\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-dmd9t"
Sep 17 16:46:37.982845 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:37.982811    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-dmd9t"
Sep 17 16:46:38.281770 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:38.281734    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-dmd9t"]
Sep 17 16:46:38.284972 ip-10-0-135-43 kubenswrapper[2585]: W0917 16:46:38.284944    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a53935f_ef87_4850_a1b4_e5d9a31f4355.slice/crio-1d5f3191c549d9e04d0a067876dff166650488e6d993f5a884742ec072ef2a84 WatchSource:0}: Error finding container 1d5f3191c549d9e04d0a067876dff166650488e6d993f5a884742ec072ef2a84: Status 404 returned error can't find the container with id 1d5f3191c549d9e04d0a067876dff166650488e6d993f5a884742ec072ef2a84
Sep 17 16:46:38.568845 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:38.568755    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f5zd5" podUID="fc3cce66-9206-4ea3-b7e1-7f85aded460b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 17 16:46:38.787895 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:38.787854    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-dmd9t" event={"ID":"1a53935f-ef87-4850-a1b4-e5d9a31f4355","Type":"ContainerStarted","Data":"d73b362601f26a748661cecd6b4b5826b6147230cb4218375e6ddf75ac3c0da6"}
Sep 17 16:46:38.788323 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:38.787928    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-dmd9t" event={"ID":"1a53935f-ef87-4850-a1b4-e5d9a31f4355","Type":"ContainerStarted","Data":"1d5f3191c549d9e04d0a067876dff166650488e6d993f5a884742ec072ef2a84"}
Sep 17 16:46:42.807231 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:42.807202    2585 generic.go:356] "Generic (PLEG): container finished" podID="1a53935f-ef87-4850-a1b4-e5d9a31f4355" containerID="d73b362601f26a748661cecd6b4b5826b6147230cb4218375e6ddf75ac3c0da6" exitCode=0
Sep 17 16:46:42.807638 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:42.807269    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-dmd9t" event={"ID":"1a53935f-ef87-4850-a1b4-e5d9a31f4355","Type":"ContainerDied","Data":"d73b362601f26a748661cecd6b4b5826b6147230cb4218375e6ddf75ac3c0da6"}
Sep 17 16:46:48.569843 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:48.569801    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f5zd5"
Sep 17 16:46:50.856671 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:50.856634    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-dmd9t" event={"ID":"1a53935f-ef87-4850-a1b4-e5d9a31f4355","Type":"ContainerStarted","Data":"1a6de55ec8b25974baf1837db46e9c5129cbc606e1e855911f6fb713bbb946a2"}
Sep 17 16:46:50.857068 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:50.856698    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-dmd9t" event={"ID":"1a53935f-ef87-4850-a1b4-e5d9a31f4355","Type":"ContainerStarted","Data":"40bee69935cf3cf5c4b032243bc8bdf7616b2a9bbb4137392628694876761ebb"}
Sep 17 16:46:50.857068 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:50.856891    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-dmd9t"
Sep 17 16:46:50.886664 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:50.886603    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-dmd9t" podStartSLOduration=2.793669628 podStartE2EDuration="13.886583973s" totalImagesPullingTime="7.305759926s" totalInitContainerRuntime="3.787154419s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:46:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:46:50.883147664 +0000 UTC m=+3166.653479454" watchObservedRunningTime="2026-09-17 16:46:50.886583973 +0000 UTC m=+3166.656915771"
Sep 17 16:46:51.861178 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:51.861143    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-dmd9t"
Sep 17 16:46:51.862126 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:51.862100    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-dmd9t" podUID="1a53935f-ef87-4850-a1b4-e5d9a31f4355" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 17 16:46:52.865506 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:52.865464    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-dmd9t" podUID="1a53935f-ef87-4850-a1b4-e5d9a31f4355" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 17 16:46:56.779697 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:56.779661    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f5zd5"]
Sep 17 16:46:56.780112 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:56.779897    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f5zd5" podUID="fc3cce66-9206-4ea3-b7e1-7f85aded460b" containerName="kserve-container" containerID="cri-o://e9192dfb499648479a3869e8e9582817d9551d97fa9c87a09c54827b4de463a7" gracePeriod=30
Sep 17 16:46:56.780112 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:56.779983    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f5zd5" podUID="fc3cce66-9206-4ea3-b7e1-7f85aded460b" containerName="kube-rbac-proxy" containerID="cri-o://530f8ed817ad4e7a45904e6940b3cda94caedf323118b7774c6bf94acb85ab0d" gracePeriod=30
Sep 17 16:46:57.871861 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:57.871832    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-dmd9t"
Sep 17 16:46:57.872385 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:57.872360    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-dmd9t" podUID="1a53935f-ef87-4850-a1b4-e5d9a31f4355" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 17 16:46:57.891151 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:57.891121    2585 generic.go:356] "Generic (PLEG): container finished" podID="fc3cce66-9206-4ea3-b7e1-7f85aded460b" containerID="530f8ed817ad4e7a45904e6940b3cda94caedf323118b7774c6bf94acb85ab0d" exitCode=2
Sep 17 16:46:57.891327 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:57.891170    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f5zd5" event={"ID":"fc3cce66-9206-4ea3-b7e1-7f85aded460b","Type":"ContainerDied","Data":"530f8ed817ad4e7a45904e6940b3cda94caedf323118b7774c6bf94acb85ab0d"}
Sep 17 16:46:58.564007 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:58.563966    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f5zd5" podUID="fc3cce66-9206-4ea3-b7e1-7f85aded460b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.65:8643/healthz\": dial tcp 10.134.0.65:8643: connect: connection refused"
Sep 17 16:46:58.569401 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:46:58.569370    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f5zd5" podUID="fc3cce66-9206-4ea3-b7e1-7f85aded460b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 17 16:47:00.135908 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:00.135876    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f5zd5"
Sep 17 16:47:00.198642 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:00.198610    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc3cce66-9206-4ea3-b7e1-7f85aded460b-cabundle-cert\") pod \"fc3cce66-9206-4ea3-b7e1-7f85aded460b\" (UID: \"fc3cce66-9206-4ea3-b7e1-7f85aded460b\") "
Sep 17 16:47:00.198804 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:00.198657    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p267x\" (UniqueName: \"kubernetes.io/projected/fc3cce66-9206-4ea3-b7e1-7f85aded460b-kube-api-access-p267x\") pod \"fc3cce66-9206-4ea3-b7e1-7f85aded460b\" (UID: \"fc3cce66-9206-4ea3-b7e1-7f85aded460b\") "
Sep 17 16:47:00.198804 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:00.198699    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc3cce66-9206-4ea3-b7e1-7f85aded460b-kube-rbac-proxy-sar-config\") pod \"fc3cce66-9206-4ea3-b7e1-7f85aded460b\" (UID: \"fc3cce66-9206-4ea3-b7e1-7f85aded460b\") "
Sep 17 16:47:00.198804 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:00.198733    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc3cce66-9206-4ea3-b7e1-7f85aded460b-kserve-provision-location\") pod \"fc3cce66-9206-4ea3-b7e1-7f85aded460b\" (UID: \"fc3cce66-9206-4ea3-b7e1-7f85aded460b\") "
Sep 17 16:47:00.198804 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:00.198787    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc3cce66-9206-4ea3-b7e1-7f85aded460b-proxy-tls\") pod \"fc3cce66-9206-4ea3-b7e1-7f85aded460b\" (UID: \"fc3cce66-9206-4ea3-b7e1-7f85aded460b\") "
Sep 17 16:47:00.199046 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:00.199022    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc3cce66-9206-4ea3-b7e1-7f85aded460b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fc3cce66-9206-4ea3-b7e1-7f85aded460b" (UID: "fc3cce66-9206-4ea3-b7e1-7f85aded460b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:47:00.199161 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:00.199127    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc3cce66-9206-4ea3-b7e1-7f85aded460b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fc3cce66-9206-4ea3-b7e1-7f85aded460b" (UID: "fc3cce66-9206-4ea3-b7e1-7f85aded460b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:47:00.200929 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:00.200904    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc3cce66-9206-4ea3-b7e1-7f85aded460b-kube-api-access-p267x" (OuterVolumeSpecName: "kube-api-access-p267x") pod "fc3cce66-9206-4ea3-b7e1-7f85aded460b" (UID: "fc3cce66-9206-4ea3-b7e1-7f85aded460b"). InnerVolumeSpecName "kube-api-access-p267x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:47:00.201034 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:00.200980    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc3cce66-9206-4ea3-b7e1-7f85aded460b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fc3cce66-9206-4ea3-b7e1-7f85aded460b" (UID: "fc3cce66-9206-4ea3-b7e1-7f85aded460b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:47:00.208939 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:00.208908    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc3cce66-9206-4ea3-b7e1-7f85aded460b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fc3cce66-9206-4ea3-b7e1-7f85aded460b" (UID: "fc3cce66-9206-4ea3-b7e1-7f85aded460b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:47:00.300165 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:00.300119    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc3cce66-9206-4ea3-b7e1-7f85aded460b-cabundle-cert\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:47:00.300165 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:00.300155    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p267x\" (UniqueName: \"kubernetes.io/projected/fc3cce66-9206-4ea3-b7e1-7f85aded460b-kube-api-access-p267x\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:47:00.300165 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:00.300170    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc3cce66-9206-4ea3-b7e1-7f85aded460b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:47:00.300411 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:00.300183    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc3cce66-9206-4ea3-b7e1-7f85aded460b-kserve-provision-location\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:47:00.300411 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:00.300196    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc3cce66-9206-4ea3-b7e1-7f85aded460b-proxy-tls\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:47:00.907643 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:00.907558    2585 generic.go:356] "Generic (PLEG): container finished" podID="fc3cce66-9206-4ea3-b7e1-7f85aded460b" containerID="e9192dfb499648479a3869e8e9582817d9551d97fa9c87a09c54827b4de463a7" exitCode=0
Sep 17 16:47:00.907643 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:00.907626    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f5zd5"
Sep 17 16:47:00.907922 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:00.907641    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f5zd5" event={"ID":"fc3cce66-9206-4ea3-b7e1-7f85aded460b","Type":"ContainerDied","Data":"e9192dfb499648479a3869e8e9582817d9551d97fa9c87a09c54827b4de463a7"}
Sep 17 16:47:00.907922 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:00.907728    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f5zd5" event={"ID":"fc3cce66-9206-4ea3-b7e1-7f85aded460b","Type":"ContainerDied","Data":"6a3e9d5c82194826e31a640464f7f78a6b4df754c62be57419a195cab5f28baa"}
Sep 17 16:47:00.907922 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:00.907753    2585 scope.go:117] "RemoveContainer" containerID="530f8ed817ad4e7a45904e6940b3cda94caedf323118b7774c6bf94acb85ab0d"
Sep 17 16:47:00.917411 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:00.917391    2585 scope.go:117] "RemoveContainer" containerID="e9192dfb499648479a3869e8e9582817d9551d97fa9c87a09c54827b4de463a7"
Sep 17 16:47:00.925333 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:00.925313    2585 scope.go:117] "RemoveContainer" containerID="3962a930c7c8f86653a650a867657c00de24dfd8a1c54f0cdb7a55352bd7f986"
Sep 17 16:47:00.929091 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:00.929068    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f5zd5"]
Sep 17 16:47:00.930745 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:00.930721    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-f5zd5"]
Sep 17 16:47:00.934084 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:00.934065    2585 scope.go:117] "RemoveContainer" containerID="530f8ed817ad4e7a45904e6940b3cda94caedf323118b7774c6bf94acb85ab0d"
Sep 17 16:47:00.934431 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:47:00.934389    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"530f8ed817ad4e7a45904e6940b3cda94caedf323118b7774c6bf94acb85ab0d\": container with ID starting with 530f8ed817ad4e7a45904e6940b3cda94caedf323118b7774c6bf94acb85ab0d not found: ID does not exist" containerID="530f8ed817ad4e7a45904e6940b3cda94caedf323118b7774c6bf94acb85ab0d"
Sep 17 16:47:00.934493 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:00.934441    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"530f8ed817ad4e7a45904e6940b3cda94caedf323118b7774c6bf94acb85ab0d"} err="failed to get container status \"530f8ed817ad4e7a45904e6940b3cda94caedf323118b7774c6bf94acb85ab0d\": rpc error: code = NotFound desc = could not find container \"530f8ed817ad4e7a45904e6940b3cda94caedf323118b7774c6bf94acb85ab0d\": container with ID starting with 530f8ed817ad4e7a45904e6940b3cda94caedf323118b7774c6bf94acb85ab0d not found: ID does not exist"
Sep 17 16:47:00.934493 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:00.934460    2585 scope.go:117] "RemoveContainer" containerID="e9192dfb499648479a3869e8e9582817d9551d97fa9c87a09c54827b4de463a7"
Sep 17 16:47:00.934733 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:47:00.934714    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9192dfb499648479a3869e8e9582817d9551d97fa9c87a09c54827b4de463a7\": container with ID starting with e9192dfb499648479a3869e8e9582817d9551d97fa9c87a09c54827b4de463a7 not found: ID does not exist" containerID="e9192dfb499648479a3869e8e9582817d9551d97fa9c87a09c54827b4de463a7"
Sep 17 16:47:00.934776 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:00.934740    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9192dfb499648479a3869e8e9582817d9551d97fa9c87a09c54827b4de463a7"} err="failed to get container status \"e9192dfb499648479a3869e8e9582817d9551d97fa9c87a09c54827b4de463a7\": rpc error: code = NotFound desc = could not find container \"e9192dfb499648479a3869e8e9582817d9551d97fa9c87a09c54827b4de463a7\": container with ID starting with e9192dfb499648479a3869e8e9582817d9551d97fa9c87a09c54827b4de463a7 not found: ID does not exist"
Sep 17 16:47:00.934776 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:00.934755    2585 scope.go:117] "RemoveContainer" containerID="3962a930c7c8f86653a650a867657c00de24dfd8a1c54f0cdb7a55352bd7f986"
Sep 17 16:47:00.935033 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:47:00.935013    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3962a930c7c8f86653a650a867657c00de24dfd8a1c54f0cdb7a55352bd7f986\": container with ID starting with 3962a930c7c8f86653a650a867657c00de24dfd8a1c54f0cdb7a55352bd7f986 not found: ID does not exist" containerID="3962a930c7c8f86653a650a867657c00de24dfd8a1c54f0cdb7a55352bd7f986"
Sep 17 16:47:00.935092 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:00.935039    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3962a930c7c8f86653a650a867657c00de24dfd8a1c54f0cdb7a55352bd7f986"} err="failed to get container status \"3962a930c7c8f86653a650a867657c00de24dfd8a1c54f0cdb7a55352bd7f986\": rpc error: code = NotFound desc = could not find container \"3962a930c7c8f86653a650a867657c00de24dfd8a1c54f0cdb7a55352bd7f986\": container with ID starting with 3962a930c7c8f86653a650a867657c00de24dfd8a1c54f0cdb7a55352bd7f986 not found: ID does not exist"
Sep 17 16:47:02.099776 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:02.099741    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx"]
Sep 17 16:47:02.100182 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:02.100074    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc3cce66-9206-4ea3-b7e1-7f85aded460b" containerName="kube-rbac-proxy"
Sep 17 16:47:02.100182 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:02.100086    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc3cce66-9206-4ea3-b7e1-7f85aded460b" containerName="kube-rbac-proxy"
Sep 17 16:47:02.100182 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:02.100108    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc3cce66-9206-4ea3-b7e1-7f85aded460b" containerName="storage-initializer"
Sep 17 16:47:02.100182 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:02.100114    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc3cce66-9206-4ea3-b7e1-7f85aded460b" containerName="storage-initializer"
Sep 17 16:47:02.100182 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:02.100123    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc3cce66-9206-4ea3-b7e1-7f85aded460b" containerName="kserve-container"
Sep 17 16:47:02.100182 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:02.100130    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc3cce66-9206-4ea3-b7e1-7f85aded460b" containerName="kserve-container"
Sep 17 16:47:02.100381 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:02.100189    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc3cce66-9206-4ea3-b7e1-7f85aded460b" containerName="kube-rbac-proxy"
Sep 17 16:47:02.100381 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:02.100196    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc3cce66-9206-4ea3-b7e1-7f85aded460b" containerName="kserve-container"
Sep 17 16:47:02.108265 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:02.108241    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx"
Sep 17 16:47:02.112728 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:02.112695    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 17 16:47:02.112899 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:02.112785    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
Sep 17 16:47:02.114603 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:02.114576    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24ee774d-5b39-42c8-b901-4c52c375dec2-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx\" (UID: \"24ee774d-5b39-42c8-b901-4c52c375dec2\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx"
Sep 17 16:47:02.114746 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:02.114700    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24ee774d-5b39-42c8-b901-4c52c375dec2-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx\" (UID: \"24ee774d-5b39-42c8-b901-4c52c375dec2\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx"
Sep 17 16:47:02.115147 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:02.114742    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24ee774d-5b39-42c8-b901-4c52c375dec2-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx\" (UID: \"24ee774d-5b39-42c8-b901-4c52c375dec2\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx"
Sep 17 16:47:02.115147 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:02.114789    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24ee774d-5b39-42c8-b901-4c52c375dec2-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx\" (UID: \"24ee774d-5b39-42c8-b901-4c52c375dec2\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx"
Sep 17 16:47:02.115147 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:02.114825    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpcd8\" (UniqueName: \"kubernetes.io/projected/24ee774d-5b39-42c8-b901-4c52c375dec2-kube-api-access-hpcd8\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx\" (UID: \"24ee774d-5b39-42c8-b901-4c52c375dec2\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx"
Sep 17 16:47:02.121972 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:02.121944    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx"]
Sep 17 16:47:02.215656 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:02.215617    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24ee774d-5b39-42c8-b901-4c52c375dec2-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx\" (UID: \"24ee774d-5b39-42c8-b901-4c52c375dec2\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx"
Sep 17 16:47:02.215656 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:02.215659    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24ee774d-5b39-42c8-b901-4c52c375dec2-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx\" (UID: \"24ee774d-5b39-42c8-b901-4c52c375dec2\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx"
Sep 17 16:47:02.215939 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:02.215703    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24ee774d-5b39-42c8-b901-4c52c375dec2-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx\" (UID: \"24ee774d-5b39-42c8-b901-4c52c375dec2\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx"
Sep 17 16:47:02.215939 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:02.215740    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpcd8\" (UniqueName: \"kubernetes.io/projected/24ee774d-5b39-42c8-b901-4c52c375dec2-kube-api-access-hpcd8\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx\" (UID: \"24ee774d-5b39-42c8-b901-4c52c375dec2\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx"
Sep 17 16:47:02.215939 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:02.215777    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24ee774d-5b39-42c8-b901-4c52c375dec2-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx\" (UID: \"24ee774d-5b39-42c8-b901-4c52c375dec2\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx"
Sep 17 16:47:02.216165 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:02.216131    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24ee774d-5b39-42c8-b901-4c52c375dec2-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx\" (UID: \"24ee774d-5b39-42c8-b901-4c52c375dec2\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx"
Sep 17 16:47:02.216488 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:02.216467    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24ee774d-5b39-42c8-b901-4c52c375dec2-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx\" (UID: \"24ee774d-5b39-42c8-b901-4c52c375dec2\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx"
Sep 17 16:47:02.216581 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:02.216530    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24ee774d-5b39-42c8-b901-4c52c375dec2-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx\" (UID: \"24ee774d-5b39-42c8-b901-4c52c375dec2\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx"
Sep 17 16:47:02.218295 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:02.218270    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24ee774d-5b39-42c8-b901-4c52c375dec2-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx\" (UID: \"24ee774d-5b39-42c8-b901-4c52c375dec2\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx"
Sep 17 16:47:02.225356 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:02.225334    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpcd8\" (UniqueName: \"kubernetes.io/projected/24ee774d-5b39-42c8-b901-4c52c375dec2-kube-api-access-hpcd8\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx\" (UID: \"24ee774d-5b39-42c8-b901-4c52c375dec2\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx"
Sep 17 16:47:02.419262 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:02.419169    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx"
Sep 17 16:47:02.513650 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:02.513619    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc3cce66-9206-4ea3-b7e1-7f85aded460b" path="/var/lib/kubelet/pods/fc3cce66-9206-4ea3-b7e1-7f85aded460b/volumes"
Sep 17 16:47:02.695849 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:02.695783    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx"]
Sep 17 16:47:02.698749 ip-10-0-135-43 kubenswrapper[2585]: W0917 16:47:02.698714    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24ee774d_5b39_42c8_b901_4c52c375dec2.slice/crio-c1564cbc1eea2f403af444d7373b1087d0518d9a452a05b5cfb7081d28bc9ba8 WatchSource:0}: Error finding container c1564cbc1eea2f403af444d7373b1087d0518d9a452a05b5cfb7081d28bc9ba8: Status 404 returned error can't find the container with id c1564cbc1eea2f403af444d7373b1087d0518d9a452a05b5cfb7081d28bc9ba8
Sep 17 16:47:02.921307 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:02.921267    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx" event={"ID":"24ee774d-5b39-42c8-b901-4c52c375dec2","Type":"ContainerStarted","Data":"558fa4ce606197dd444f0fe829279080092a886a14b527b90e74b2b36863c652"}
Sep 17 16:47:02.921516 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:02.921351    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx" event={"ID":"24ee774d-5b39-42c8-b901-4c52c375dec2","Type":"ContainerStarted","Data":"c1564cbc1eea2f403af444d7373b1087d0518d9a452a05b5cfb7081d28bc9ba8"}
Sep 17 16:47:07.872344 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:07.872300    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-dmd9t" podUID="1a53935f-ef87-4850-a1b4-e5d9a31f4355" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 17 16:47:07.950010 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:07.949976    2585 generic.go:356] "Generic (PLEG): container finished" podID="24ee774d-5b39-42c8-b901-4c52c375dec2" containerID="558fa4ce606197dd444f0fe829279080092a886a14b527b90e74b2b36863c652" exitCode=0
Sep 17 16:47:07.950221 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:07.950067    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx" event={"ID":"24ee774d-5b39-42c8-b901-4c52c375dec2","Type":"ContainerDied","Data":"558fa4ce606197dd444f0fe829279080092a886a14b527b90e74b2b36863c652"}
Sep 17 16:47:08.956663 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:08.956627    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx" event={"ID":"24ee774d-5b39-42c8-b901-4c52c375dec2","Type":"ContainerStarted","Data":"2d8ff8b3ae8e8141ae737273f8a1be18c9d34d1a8f48f5f735bf44ba9e2ca7c5"}
Sep 17 16:47:08.957066 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:08.956694    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx" event={"ID":"24ee774d-5b39-42c8-b901-4c52c375dec2","Type":"ContainerStarted","Data":"82c5ed18b1f08003aee110e2ace688a49479e28ffaa168781ed75b83a65a89ed"}
Sep 17 16:47:08.957066 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:08.956890    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx"
Sep 17 16:47:08.983973 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:08.983918    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx" podStartSLOduration=2.315150793 podStartE2EDuration="6.98389997s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.668749177s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:47:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:47:08.983158968 +0000 UTC m=+3184.753490767" watchObservedRunningTime="2026-09-17 16:47:08.98389997 +0000 UTC m=+3184.754231767"
Sep 17 16:47:09.961902 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:09.961860    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx"
Sep 17 16:47:09.963121 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:09.963093    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx" podUID="24ee774d-5b39-42c8-b901-4c52c375dec2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Sep 17 16:47:10.966190 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:10.966149    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx" podUID="24ee774d-5b39-42c8-b901-4c52c375dec2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Sep 17 16:47:15.970639 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:15.970610    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx"
Sep 17 16:47:15.971221 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:15.971191    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx" podUID="24ee774d-5b39-42c8-b901-4c52c375dec2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Sep 17 16:47:17.873134 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:17.873092    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-dmd9t" podUID="1a53935f-ef87-4850-a1b4-e5d9a31f4355" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 17 16:47:25.971130 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:25.971087    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx" podUID="24ee774d-5b39-42c8-b901-4c52c375dec2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Sep 17 16:47:27.872594 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:27.872550    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-dmd9t" podUID="1a53935f-ef87-4850-a1b4-e5d9a31f4355" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 17 16:47:35.972107 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:35.972058    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx" podUID="24ee774d-5b39-42c8-b901-4c52c375dec2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Sep 17 16:47:37.873250 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:37.873201    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-dmd9t" podUID="1a53935f-ef87-4850-a1b4-e5d9a31f4355" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 17 16:47:45.972061 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:45.972015    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx" podUID="24ee774d-5b39-42c8-b901-4c52c375dec2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Sep 17 16:47:47.872876 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:47.872775    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-dmd9t" podUID="1a53935f-ef87-4850-a1b4-e5d9a31f4355" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 17 16:47:55.971705 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:55.971670    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx"
Sep 17 16:47:57.872606 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:47:57.872566    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-dmd9t" podUID="1a53935f-ef87-4850-a1b4-e5d9a31f4355" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 17 16:48:01.507509 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:01.507474    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-dmd9t" podUID="1a53935f-ef87-4850-a1b4-e5d9a31f4355" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 17 16:48:03.685909 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:03.685865    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx"]
Sep 17 16:48:03.686403 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:03.686161    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx" podUID="24ee774d-5b39-42c8-b901-4c52c375dec2" containerName="kserve-container" containerID="cri-o://82c5ed18b1f08003aee110e2ace688a49479e28ffaa168781ed75b83a65a89ed" gracePeriod=30
Sep 17 16:48:03.686403 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:03.686199    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx" podUID="24ee774d-5b39-42c8-b901-4c52c375dec2" containerName="kube-rbac-proxy" containerID="cri-o://2d8ff8b3ae8e8141ae737273f8a1be18c9d34d1a8f48f5f735bf44ba9e2ca7c5" gracePeriod=30
Sep 17 16:48:04.224364 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:04.224329    2585 generic.go:356] "Generic (PLEG): container finished" podID="24ee774d-5b39-42c8-b901-4c52c375dec2" containerID="2d8ff8b3ae8e8141ae737273f8a1be18c9d34d1a8f48f5f735bf44ba9e2ca7c5" exitCode=2
Sep 17 16:48:04.224558 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:04.224398    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx" event={"ID":"24ee774d-5b39-42c8-b901-4c52c375dec2","Type":"ContainerDied","Data":"2d8ff8b3ae8e8141ae737273f8a1be18c9d34d1a8f48f5f735bf44ba9e2ca7c5"}
Sep 17 16:48:05.967053 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:05.967004    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx" podUID="24ee774d-5b39-42c8-b901-4c52c375dec2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.67:8643/healthz\": dial tcp 10.134.0.67:8643: connect: connection refused"
Sep 17 16:48:05.971408 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:05.971380    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx" podUID="24ee774d-5b39-42c8-b901-4c52c375dec2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.67:8080: connect: connection refused"
Sep 17 16:48:06.931656 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:06.931632    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx"
Sep 17 16:48:06.988994 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:06.988895    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24ee774d-5b39-42c8-b901-4c52c375dec2-kube-rbac-proxy-sar-config\") pod \"24ee774d-5b39-42c8-b901-4c52c375dec2\" (UID: \"24ee774d-5b39-42c8-b901-4c52c375dec2\") "
Sep 17 16:48:06.988994 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:06.988966    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hpcd8\" (UniqueName: \"kubernetes.io/projected/24ee774d-5b39-42c8-b901-4c52c375dec2-kube-api-access-hpcd8\") pod \"24ee774d-5b39-42c8-b901-4c52c375dec2\" (UID: \"24ee774d-5b39-42c8-b901-4c52c375dec2\") "
Sep 17 16:48:06.989586 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:06.989058    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24ee774d-5b39-42c8-b901-4c52c375dec2-cabundle-cert\") pod \"24ee774d-5b39-42c8-b901-4c52c375dec2\" (UID: \"24ee774d-5b39-42c8-b901-4c52c375dec2\") "
Sep 17 16:48:06.989586 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:06.989109    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24ee774d-5b39-42c8-b901-4c52c375dec2-proxy-tls\") pod \"24ee774d-5b39-42c8-b901-4c52c375dec2\" (UID: \"24ee774d-5b39-42c8-b901-4c52c375dec2\") "
Sep 17 16:48:06.989586 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:06.989146    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24ee774d-5b39-42c8-b901-4c52c375dec2-kserve-provision-location\") pod \"24ee774d-5b39-42c8-b901-4c52c375dec2\" (UID: \"24ee774d-5b39-42c8-b901-4c52c375dec2\") "
Sep 17 16:48:06.989586 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:06.989347    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24ee774d-5b39-42c8-b901-4c52c375dec2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "24ee774d-5b39-42c8-b901-4c52c375dec2" (UID: "24ee774d-5b39-42c8-b901-4c52c375dec2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:48:06.989818 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:06.989488    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24ee774d-5b39-42c8-b901-4c52c375dec2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "24ee774d-5b39-42c8-b901-4c52c375dec2" (UID: "24ee774d-5b39-42c8-b901-4c52c375dec2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:48:06.991319 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:06.991293    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24ee774d-5b39-42c8-b901-4c52c375dec2-kube-api-access-hpcd8" (OuterVolumeSpecName: "kube-api-access-hpcd8") pod "24ee774d-5b39-42c8-b901-4c52c375dec2" (UID: "24ee774d-5b39-42c8-b901-4c52c375dec2"). InnerVolumeSpecName "kube-api-access-hpcd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:48:06.991319 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:06.991301    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24ee774d-5b39-42c8-b901-4c52c375dec2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "24ee774d-5b39-42c8-b901-4c52c375dec2" (UID: "24ee774d-5b39-42c8-b901-4c52c375dec2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:48:06.999639 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:06.999610    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24ee774d-5b39-42c8-b901-4c52c375dec2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "24ee774d-5b39-42c8-b901-4c52c375dec2" (UID: "24ee774d-5b39-42c8-b901-4c52c375dec2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:48:07.090758 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:07.090721    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24ee774d-5b39-42c8-b901-4c52c375dec2-kserve-provision-location\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:48:07.090758 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:07.090752    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24ee774d-5b39-42c8-b901-4c52c375dec2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:48:07.090758 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:07.090762    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpcd8\" (UniqueName: \"kubernetes.io/projected/24ee774d-5b39-42c8-b901-4c52c375dec2-kube-api-access-hpcd8\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:48:07.090758 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:07.090772    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24ee774d-5b39-42c8-b901-4c52c375dec2-cabundle-cert\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:48:07.091040 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:07.090781    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24ee774d-5b39-42c8-b901-4c52c375dec2-proxy-tls\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:48:07.239991 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:07.239899    2585 generic.go:356] "Generic (PLEG): container finished" podID="24ee774d-5b39-42c8-b901-4c52c375dec2" containerID="82c5ed18b1f08003aee110e2ace688a49479e28ffaa168781ed75b83a65a89ed" exitCode=0
Sep 17 16:48:07.239991 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:07.239957    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx"
Sep 17 16:48:07.239991 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:07.239982    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx" event={"ID":"24ee774d-5b39-42c8-b901-4c52c375dec2","Type":"ContainerDied","Data":"82c5ed18b1f08003aee110e2ace688a49479e28ffaa168781ed75b83a65a89ed"}
Sep 17 16:48:07.240223 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:07.240043    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx" event={"ID":"24ee774d-5b39-42c8-b901-4c52c375dec2","Type":"ContainerDied","Data":"c1564cbc1eea2f403af444d7373b1087d0518d9a452a05b5cfb7081d28bc9ba8"}
Sep 17 16:48:07.240223 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:07.240080    2585 scope.go:117] "RemoveContainer" containerID="2d8ff8b3ae8e8141ae737273f8a1be18c9d34d1a8f48f5f735bf44ba9e2ca7c5"
Sep 17 16:48:07.249328 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:07.249309    2585 scope.go:117] "RemoveContainer" containerID="82c5ed18b1f08003aee110e2ace688a49479e28ffaa168781ed75b83a65a89ed"
Sep 17 16:48:07.257758 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:07.257737    2585 scope.go:117] "RemoveContainer" containerID="558fa4ce606197dd444f0fe829279080092a886a14b527b90e74b2b36863c652"
Sep 17 16:48:07.264982 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:07.264958    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx"]
Sep 17 16:48:07.266401 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:07.266388    2585 scope.go:117] "RemoveContainer" containerID="2d8ff8b3ae8e8141ae737273f8a1be18c9d34d1a8f48f5f735bf44ba9e2ca7c5"
Sep 17 16:48:07.266714 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:48:07.266693    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d8ff8b3ae8e8141ae737273f8a1be18c9d34d1a8f48f5f735bf44ba9e2ca7c5\": container with ID starting with 2d8ff8b3ae8e8141ae737273f8a1be18c9d34d1a8f48f5f735bf44ba9e2ca7c5 not found: ID does not exist" containerID="2d8ff8b3ae8e8141ae737273f8a1be18c9d34d1a8f48f5f735bf44ba9e2ca7c5"
Sep 17 16:48:07.266763 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:07.266720    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d8ff8b3ae8e8141ae737273f8a1be18c9d34d1a8f48f5f735bf44ba9e2ca7c5"} err="failed to get container status \"2d8ff8b3ae8e8141ae737273f8a1be18c9d34d1a8f48f5f735bf44ba9e2ca7c5\": rpc error: code = NotFound desc = could not find container \"2d8ff8b3ae8e8141ae737273f8a1be18c9d34d1a8f48f5f735bf44ba9e2ca7c5\": container with ID starting with 2d8ff8b3ae8e8141ae737273f8a1be18c9d34d1a8f48f5f735bf44ba9e2ca7c5 not found: ID does not exist"
Sep 17 16:48:07.266763 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:07.266737    2585 scope.go:117] "RemoveContainer" containerID="82c5ed18b1f08003aee110e2ace688a49479e28ffaa168781ed75b83a65a89ed"
Sep 17 16:48:07.266983 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:48:07.266966    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82c5ed18b1f08003aee110e2ace688a49479e28ffaa168781ed75b83a65a89ed\": container with ID starting with 82c5ed18b1f08003aee110e2ace688a49479e28ffaa168781ed75b83a65a89ed not found: ID does not exist" containerID="82c5ed18b1f08003aee110e2ace688a49479e28ffaa168781ed75b83a65a89ed"
Sep 17 16:48:07.267026 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:07.266984    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82c5ed18b1f08003aee110e2ace688a49479e28ffaa168781ed75b83a65a89ed"} err="failed to get container status \"82c5ed18b1f08003aee110e2ace688a49479e28ffaa168781ed75b83a65a89ed\": rpc error: code = NotFound desc = could not find container \"82c5ed18b1f08003aee110e2ace688a49479e28ffaa168781ed75b83a65a89ed\": container with ID starting with 82c5ed18b1f08003aee110e2ace688a49479e28ffaa168781ed75b83a65a89ed not found: ID does not exist"
Sep 17 16:48:07.267026 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:07.266997    2585 scope.go:117] "RemoveContainer" containerID="558fa4ce606197dd444f0fe829279080092a886a14b527b90e74b2b36863c652"
Sep 17 16:48:07.267236 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:48:07.267218    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"558fa4ce606197dd444f0fe829279080092a886a14b527b90e74b2b36863c652\": container with ID starting with 558fa4ce606197dd444f0fe829279080092a886a14b527b90e74b2b36863c652 not found: ID does not exist" containerID="558fa4ce606197dd444f0fe829279080092a886a14b527b90e74b2b36863c652"
Sep 17 16:48:07.267280 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:07.267242    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"558fa4ce606197dd444f0fe829279080092a886a14b527b90e74b2b36863c652"} err="failed to get container status \"558fa4ce606197dd444f0fe829279080092a886a14b527b90e74b2b36863c652\": rpc error: code = NotFound desc = could not find container \"558fa4ce606197dd444f0fe829279080092a886a14b527b90e74b2b36863c652\": container with ID starting with 558fa4ce606197dd444f0fe829279080092a886a14b527b90e74b2b36863c652 not found: ID does not exist"
Sep 17 16:48:07.270571 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:07.270549    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-khcvx"]
Sep 17 16:48:08.512259 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:08.512224    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24ee774d-5b39-42c8-b901-4c52c375dec2" path="/var/lib/kubelet/pods/24ee774d-5b39-42c8-b901-4c52c375dec2/volumes"
Sep 17 16:48:11.508413 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:11.508370    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-dmd9t" podUID="1a53935f-ef87-4850-a1b4-e5d9a31f4355" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 17 16:48:21.508605 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:21.508569    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-dmd9t"
Sep 17 16:48:29.249598 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:29.249562    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-dmd9t"]
Sep 17 16:48:29.250073 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:29.249824    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-dmd9t" podUID="1a53935f-ef87-4850-a1b4-e5d9a31f4355" containerName="kserve-container" containerID="cri-o://40bee69935cf3cf5c4b032243bc8bdf7616b2a9bbb4137392628694876761ebb" gracePeriod=30
Sep 17 16:48:29.250073 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:29.249870    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-dmd9t" podUID="1a53935f-ef87-4850-a1b4-e5d9a31f4355" containerName="kube-rbac-proxy" containerID="cri-o://1a6de55ec8b25974baf1837db46e9c5129cbc606e1e855911f6fb713bbb946a2" gracePeriod=30
Sep 17 16:48:30.345105 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:30.345072    2585 generic.go:356] "Generic (PLEG): container finished" podID="1a53935f-ef87-4850-a1b4-e5d9a31f4355" containerID="1a6de55ec8b25974baf1837db46e9c5129cbc606e1e855911f6fb713bbb946a2" exitCode=2
Sep 17 16:48:30.345512 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:30.345145    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-dmd9t" event={"ID":"1a53935f-ef87-4850-a1b4-e5d9a31f4355","Type":"ContainerDied","Data":"1a6de55ec8b25974baf1837db46e9c5129cbc606e1e855911f6fb713bbb946a2"}
Sep 17 16:48:31.507976 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:31.507934    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-dmd9t" podUID="1a53935f-ef87-4850-a1b4-e5d9a31f4355" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.66:8080: connect: connection refused"
Sep 17 16:48:32.865753 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:32.865708    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-dmd9t" podUID="1a53935f-ef87-4850-a1b4-e5d9a31f4355" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.66:8643/healthz\": dial tcp 10.134.0.66:8643: connect: connection refused"
Sep 17 16:48:33.799114 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:33.799090    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-dmd9t"
Sep 17 16:48:33.925830 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:33.925787    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2vbr\" (UniqueName: \"kubernetes.io/projected/1a53935f-ef87-4850-a1b4-e5d9a31f4355-kube-api-access-n2vbr\") pod \"1a53935f-ef87-4850-a1b4-e5d9a31f4355\" (UID: \"1a53935f-ef87-4850-a1b4-e5d9a31f4355\") "
Sep 17 16:48:33.925830 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:33.925833    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a53935f-ef87-4850-a1b4-e5d9a31f4355-cabundle-cert\") pod \"1a53935f-ef87-4850-a1b4-e5d9a31f4355\" (UID: \"1a53935f-ef87-4850-a1b4-e5d9a31f4355\") "
Sep 17 16:48:33.926386 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:33.925852    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a53935f-ef87-4850-a1b4-e5d9a31f4355-proxy-tls\") pod \"1a53935f-ef87-4850-a1b4-e5d9a31f4355\" (UID: \"1a53935f-ef87-4850-a1b4-e5d9a31f4355\") "
Sep 17 16:48:33.926386 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:33.925875    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a53935f-ef87-4850-a1b4-e5d9a31f4355-kserve-provision-location\") pod \"1a53935f-ef87-4850-a1b4-e5d9a31f4355\" (UID: \"1a53935f-ef87-4850-a1b4-e5d9a31f4355\") "
Sep 17 16:48:33.926386 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:33.925935    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a53935f-ef87-4850-a1b4-e5d9a31f4355-kube-rbac-proxy-sar-config\") pod \"1a53935f-ef87-4850-a1b4-e5d9a31f4355\" (UID: \"1a53935f-ef87-4850-a1b4-e5d9a31f4355\") "
Sep 17 16:48:33.926386 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:33.926268    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a53935f-ef87-4850-a1b4-e5d9a31f4355-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1a53935f-ef87-4850-a1b4-e5d9a31f4355" (UID: "1a53935f-ef87-4850-a1b4-e5d9a31f4355"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:48:33.926386 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:33.926279    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a53935f-ef87-4850-a1b4-e5d9a31f4355-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1a53935f-ef87-4850-a1b4-e5d9a31f4355" (UID: "1a53935f-ef87-4850-a1b4-e5d9a31f4355"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:48:33.926386 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:33.926354    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a53935f-ef87-4850-a1b4-e5d9a31f4355-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1a53935f-ef87-4850-a1b4-e5d9a31f4355" (UID: "1a53935f-ef87-4850-a1b4-e5d9a31f4355"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:48:33.928025 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:33.928003    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a53935f-ef87-4850-a1b4-e5d9a31f4355-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1a53935f-ef87-4850-a1b4-e5d9a31f4355" (UID: "1a53935f-ef87-4850-a1b4-e5d9a31f4355"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:48:33.928158 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:33.928133    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a53935f-ef87-4850-a1b4-e5d9a31f4355-kube-api-access-n2vbr" (OuterVolumeSpecName: "kube-api-access-n2vbr") pod "1a53935f-ef87-4850-a1b4-e5d9a31f4355" (UID: "1a53935f-ef87-4850-a1b4-e5d9a31f4355"). InnerVolumeSpecName "kube-api-access-n2vbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:48:34.027381 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.027278    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2vbr\" (UniqueName: \"kubernetes.io/projected/1a53935f-ef87-4850-a1b4-e5d9a31f4355-kube-api-access-n2vbr\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:48:34.027381 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.027316    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a53935f-ef87-4850-a1b4-e5d9a31f4355-cabundle-cert\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:48:34.027381 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.027330    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a53935f-ef87-4850-a1b4-e5d9a31f4355-proxy-tls\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:48:34.027381 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.027341    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a53935f-ef87-4850-a1b4-e5d9a31f4355-kserve-provision-location\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:48:34.027381 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.027352    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a53935f-ef87-4850-a1b4-e5d9a31f4355-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:48:34.364040 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.363947    2585 generic.go:356] "Generic (PLEG): container finished" podID="1a53935f-ef87-4850-a1b4-e5d9a31f4355" containerID="40bee69935cf3cf5c4b032243bc8bdf7616b2a9bbb4137392628694876761ebb" exitCode=0
Sep 17 16:48:34.364040 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.364011    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-dmd9t" event={"ID":"1a53935f-ef87-4850-a1b4-e5d9a31f4355","Type":"ContainerDied","Data":"40bee69935cf3cf5c4b032243bc8bdf7616b2a9bbb4137392628694876761ebb"}
Sep 17 16:48:34.364040 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.364017    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-dmd9t"
Sep 17 16:48:34.364315 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.364059    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-dmd9t" event={"ID":"1a53935f-ef87-4850-a1b4-e5d9a31f4355","Type":"ContainerDied","Data":"1d5f3191c549d9e04d0a067876dff166650488e6d993f5a884742ec072ef2a84"}
Sep 17 16:48:34.364315 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.364085    2585 scope.go:117] "RemoveContainer" containerID="1a6de55ec8b25974baf1837db46e9c5129cbc606e1e855911f6fb713bbb946a2"
Sep 17 16:48:34.373349 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.373330    2585 scope.go:117] "RemoveContainer" containerID="40bee69935cf3cf5c4b032243bc8bdf7616b2a9bbb4137392628694876761ebb"
Sep 17 16:48:34.381685 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.381665    2585 scope.go:117] "RemoveContainer" containerID="d73b362601f26a748661cecd6b4b5826b6147230cb4218375e6ddf75ac3c0da6"
Sep 17 16:48:34.388976 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.388958    2585 scope.go:117] "RemoveContainer" containerID="1a6de55ec8b25974baf1837db46e9c5129cbc606e1e855911f6fb713bbb946a2"
Sep 17 16:48:34.389242 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:48:34.389219    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a6de55ec8b25974baf1837db46e9c5129cbc606e1e855911f6fb713bbb946a2\": container with ID starting with 1a6de55ec8b25974baf1837db46e9c5129cbc606e1e855911f6fb713bbb946a2 not found: ID does not exist" containerID="1a6de55ec8b25974baf1837db46e9c5129cbc606e1e855911f6fb713bbb946a2"
Sep 17 16:48:34.389304 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.389250    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a6de55ec8b25974baf1837db46e9c5129cbc606e1e855911f6fb713bbb946a2"} err="failed to get container status \"1a6de55ec8b25974baf1837db46e9c5129cbc606e1e855911f6fb713bbb946a2\": rpc error: code = NotFound desc = could not find container \"1a6de55ec8b25974baf1837db46e9c5129cbc606e1e855911f6fb713bbb946a2\": container with ID starting with 1a6de55ec8b25974baf1837db46e9c5129cbc606e1e855911f6fb713bbb946a2 not found: ID does not exist"
Sep 17 16:48:34.389304 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.389268    2585 scope.go:117] "RemoveContainer" containerID="40bee69935cf3cf5c4b032243bc8bdf7616b2a9bbb4137392628694876761ebb"
Sep 17 16:48:34.389613 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:48:34.389593    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40bee69935cf3cf5c4b032243bc8bdf7616b2a9bbb4137392628694876761ebb\": container with ID starting with 40bee69935cf3cf5c4b032243bc8bdf7616b2a9bbb4137392628694876761ebb not found: ID does not exist" containerID="40bee69935cf3cf5c4b032243bc8bdf7616b2a9bbb4137392628694876761ebb"
Sep 17 16:48:34.389684 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.389618    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40bee69935cf3cf5c4b032243bc8bdf7616b2a9bbb4137392628694876761ebb"} err="failed to get container status \"40bee69935cf3cf5c4b032243bc8bdf7616b2a9bbb4137392628694876761ebb\": rpc error: code = NotFound desc = could not find container \"40bee69935cf3cf5c4b032243bc8bdf7616b2a9bbb4137392628694876761ebb\": container with ID starting with 40bee69935cf3cf5c4b032243bc8bdf7616b2a9bbb4137392628694876761ebb not found: ID does not exist"
Sep 17 16:48:34.389684 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.389635    2585 scope.go:117] "RemoveContainer" containerID="d73b362601f26a748661cecd6b4b5826b6147230cb4218375e6ddf75ac3c0da6"
Sep 17 16:48:34.389896 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:48:34.389879    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d73b362601f26a748661cecd6b4b5826b6147230cb4218375e6ddf75ac3c0da6\": container with ID starting with d73b362601f26a748661cecd6b4b5826b6147230cb4218375e6ddf75ac3c0da6 not found: ID does not exist" containerID="d73b362601f26a748661cecd6b4b5826b6147230cb4218375e6ddf75ac3c0da6"
Sep 17 16:48:34.389955 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.389902    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d73b362601f26a748661cecd6b4b5826b6147230cb4218375e6ddf75ac3c0da6"} err="failed to get container status \"d73b362601f26a748661cecd6b4b5826b6147230cb4218375e6ddf75ac3c0da6\": rpc error: code = NotFound desc = could not find container \"d73b362601f26a748661cecd6b4b5826b6147230cb4218375e6ddf75ac3c0da6\": container with ID starting with d73b362601f26a748661cecd6b4b5826b6147230cb4218375e6ddf75ac3c0da6 not found: ID does not exist"
Sep 17 16:48:34.398705 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.398683    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-dmd9t"]
Sep 17 16:48:34.411740 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.411711    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-dmd9t"]
Sep 17 16:48:34.501261 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.501227    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs"]
Sep 17 16:48:34.501587 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.501573    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24ee774d-5b39-42c8-b901-4c52c375dec2" containerName="kserve-container"
Sep 17 16:48:34.501641 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.501588    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="24ee774d-5b39-42c8-b901-4c52c375dec2" containerName="kserve-container"
Sep 17 16:48:34.501641 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.501596    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a53935f-ef87-4850-a1b4-e5d9a31f4355" containerName="kube-rbac-proxy"
Sep 17 16:48:34.501641 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.501601    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a53935f-ef87-4850-a1b4-e5d9a31f4355" containerName="kube-rbac-proxy"
Sep 17 16:48:34.501641 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.501610    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24ee774d-5b39-42c8-b901-4c52c375dec2" containerName="kube-rbac-proxy"
Sep 17 16:48:34.501641 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.501615    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="24ee774d-5b39-42c8-b901-4c52c375dec2" containerName="kube-rbac-proxy"
Sep 17 16:48:34.501641 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.501624    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24ee774d-5b39-42c8-b901-4c52c375dec2" containerName="storage-initializer"
Sep 17 16:48:34.501641 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.501631    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="24ee774d-5b39-42c8-b901-4c52c375dec2" containerName="storage-initializer"
Sep 17 16:48:34.501641 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.501639    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a53935f-ef87-4850-a1b4-e5d9a31f4355" containerName="storage-initializer"
Sep 17 16:48:34.501894 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.501645    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a53935f-ef87-4850-a1b4-e5d9a31f4355" containerName="storage-initializer"
Sep 17 16:48:34.501894 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.501655    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a53935f-ef87-4850-a1b4-e5d9a31f4355" containerName="kserve-container"
Sep 17 16:48:34.501894 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.501660    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a53935f-ef87-4850-a1b4-e5d9a31f4355" containerName="kserve-container"
Sep 17 16:48:34.501894 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.501737    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="24ee774d-5b39-42c8-b901-4c52c375dec2" containerName="kube-rbac-proxy"
Sep 17 16:48:34.501894 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.501744    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a53935f-ef87-4850-a1b4-e5d9a31f4355" containerName="kserve-container"
Sep 17 16:48:34.501894 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.501752    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="24ee774d-5b39-42c8-b901-4c52c375dec2" containerName="kserve-container"
Sep 17 16:48:34.501894 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.501761    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a53935f-ef87-4850-a1b4-e5d9a31f4355" containerName="kube-rbac-proxy"
Sep 17 16:48:34.506571 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.506548    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs"
Sep 17 16:48:34.510996 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.510971    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 17 16:48:34.512047 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.512021    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:48:34.512047 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.512042    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 16:48:34.512234 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.512086    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\""
Sep 17 16:48:34.512234 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.512128    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 16:48:34.512234 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.512227    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 16:48:34.512456 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.512433    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xkc4g\""
Sep 17 16:48:34.512721 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.512703    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a53935f-ef87-4850-a1b4-e5d9a31f4355" path="/var/lib/kubelet/pods/1a53935f-ef87-4850-a1b4-e5d9a31f4355/volumes"
Sep 17 16:48:34.523026 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.523000    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs"]
Sep 17 16:48:34.632592 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.632503    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a23a841-26d3-4078-bb33-7057c7b42baa-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs\" (UID: \"7a23a841-26d3-4078-bb33-7057c7b42baa\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs"
Sep 17 16:48:34.632592 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.632540    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7vqk\" (UniqueName: \"kubernetes.io/projected/7a23a841-26d3-4078-bb33-7057c7b42baa-kube-api-access-l7vqk\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs\" (UID: \"7a23a841-26d3-4078-bb33-7057c7b42baa\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs"
Sep 17 16:48:34.632798 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.632659    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a23a841-26d3-4078-bb33-7057c7b42baa-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs\" (UID: \"7a23a841-26d3-4078-bb33-7057c7b42baa\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs"
Sep 17 16:48:34.632798 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.632688    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a23a841-26d3-4078-bb33-7057c7b42baa-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs\" (UID: \"7a23a841-26d3-4078-bb33-7057c7b42baa\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs"
Sep 17 16:48:34.632798 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.632718    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a23a841-26d3-4078-bb33-7057c7b42baa-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs\" (UID: \"7a23a841-26d3-4078-bb33-7057c7b42baa\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs"
Sep 17 16:48:34.733608 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.733568    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a23a841-26d3-4078-bb33-7057c7b42baa-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs\" (UID: \"7a23a841-26d3-4078-bb33-7057c7b42baa\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs"
Sep 17 16:48:34.733608 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.733604    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a23a841-26d3-4078-bb33-7057c7b42baa-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs\" (UID: \"7a23a841-26d3-4078-bb33-7057c7b42baa\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs"
Sep 17 16:48:34.733837 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.733639    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a23a841-26d3-4078-bb33-7057c7b42baa-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs\" (UID: \"7a23a841-26d3-4078-bb33-7057c7b42baa\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs"
Sep 17 16:48:34.733837 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.733687    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a23a841-26d3-4078-bb33-7057c7b42baa-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs\" (UID: \"7a23a841-26d3-4078-bb33-7057c7b42baa\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs"
Sep 17 16:48:34.733837 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.733715    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7vqk\" (UniqueName: \"kubernetes.io/projected/7a23a841-26d3-4078-bb33-7057c7b42baa-kube-api-access-l7vqk\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs\" (UID: \"7a23a841-26d3-4078-bb33-7057c7b42baa\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs"
Sep 17 16:48:34.734205 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.734179    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a23a841-26d3-4078-bb33-7057c7b42baa-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs\" (UID: \"7a23a841-26d3-4078-bb33-7057c7b42baa\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs"
Sep 17 16:48:34.734399 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.734378    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a23a841-26d3-4078-bb33-7057c7b42baa-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs\" (UID: \"7a23a841-26d3-4078-bb33-7057c7b42baa\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs"
Sep 17 16:48:34.734472 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.734381    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a23a841-26d3-4078-bb33-7057c7b42baa-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs\" (UID: \"7a23a841-26d3-4078-bb33-7057c7b42baa\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs"
Sep 17 16:48:34.736154 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.736133    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a23a841-26d3-4078-bb33-7057c7b42baa-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs\" (UID: \"7a23a841-26d3-4078-bb33-7057c7b42baa\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs"
Sep 17 16:48:34.743262 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.743240    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7vqk\" (UniqueName: \"kubernetes.io/projected/7a23a841-26d3-4078-bb33-7057c7b42baa-kube-api-access-l7vqk\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs\" (UID: \"7a23a841-26d3-4078-bb33-7057c7b42baa\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs"
Sep 17 16:48:34.817385 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:34.817353    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs"
Sep 17 16:48:35.103928 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:35.103901    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs"]
Sep 17 16:48:35.106137 ip-10-0-135-43 kubenswrapper[2585]: W0917 16:48:35.106108    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a23a841_26d3_4078_bb33_7057c7b42baa.slice/crio-0d3e2d33893791c77c82ce52c85548d75ae52e5f9a1d4d598de7f3943604769d WatchSource:0}: Error finding container 0d3e2d33893791c77c82ce52c85548d75ae52e5f9a1d4d598de7f3943604769d: Status 404 returned error can't find the container with id 0d3e2d33893791c77c82ce52c85548d75ae52e5f9a1d4d598de7f3943604769d
Sep 17 16:48:35.370614 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:35.370499    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs" event={"ID":"7a23a841-26d3-4078-bb33-7057c7b42baa","Type":"ContainerStarted","Data":"273c6104a9e210f753c986dbb555ebb9f52caff68575ae5e433e34f7e3ad9174"}
Sep 17 16:48:35.370614 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:35.370579    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs" event={"ID":"7a23a841-26d3-4078-bb33-7057c7b42baa","Type":"ContainerStarted","Data":"0d3e2d33893791c77c82ce52c85548d75ae52e5f9a1d4d598de7f3943604769d"}
Sep 17 16:48:39.390668 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:39.390635    2585 generic.go:356] "Generic (PLEG): container finished" podID="7a23a841-26d3-4078-bb33-7057c7b42baa" containerID="273c6104a9e210f753c986dbb555ebb9f52caff68575ae5e433e34f7e3ad9174" exitCode=0
Sep 17 16:48:39.391117 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:39.390690    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs" event={"ID":"7a23a841-26d3-4078-bb33-7057c7b42baa","Type":"ContainerDied","Data":"273c6104a9e210f753c986dbb555ebb9f52caff68575ae5e433e34f7e3ad9174"}
Sep 17 16:48:40.397408 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:40.397374    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs" event={"ID":"7a23a841-26d3-4078-bb33-7057c7b42baa","Type":"ContainerStarted","Data":"99b18dab783a8e1dffdd01a13b9ec5856ec007fbf8a2ca915a3cf6aef9ebeb3c"}
Sep 17 16:48:40.397960 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:40.397459    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs" event={"ID":"7a23a841-26d3-4078-bb33-7057c7b42baa","Type":"ContainerStarted","Data":"5e7e0d8929d835e512a62f93fcf91b5f60ee1a6e6cc12e1bbd76a04f2d988fc3"}
Sep 17 16:48:40.397960 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:40.397773    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs"
Sep 17 16:48:40.397960 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:40.397952    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs"
Sep 17 16:48:40.398976 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:40.398953    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs" podUID="7a23a841-26d3-4078-bb33-7057c7b42baa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Sep 17 16:48:40.443033 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:40.442978    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs" podStartSLOduration=2.355357488 podStartE2EDuration="6.44295954s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.087602052s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:48:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:48:40.442483814 +0000 UTC m=+3276.212815612" watchObservedRunningTime="2026-09-17 16:48:40.44295954 +0000 UTC m=+3276.213291337"
Sep 17 16:48:41.401527 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:41.401480    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs" podUID="7a23a841-26d3-4078-bb33-7057c7b42baa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Sep 17 16:48:46.407027 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:46.406996    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs"
Sep 17 16:48:46.407526 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:46.407501    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs" podUID="7a23a841-26d3-4078-bb33-7057c7b42baa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Sep 17 16:48:56.407672 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:48:56.407626    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs" podUID="7a23a841-26d3-4078-bb33-7057c7b42baa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Sep 17 16:49:06.407602 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:49:06.407560    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs" podUID="7a23a841-26d3-4078-bb33-7057c7b42baa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Sep 17 16:49:16.408233 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:49:16.408192    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs" podUID="7a23a841-26d3-4078-bb33-7057c7b42baa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Sep 17 16:49:26.408115 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:49:26.408071    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs" podUID="7a23a841-26d3-4078-bb33-7057c7b42baa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Sep 17 16:49:36.407634 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:49:36.407589    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs" podUID="7a23a841-26d3-4078-bb33-7057c7b42baa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Sep 17 16:49:46.407762 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:49:46.407722    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs" podUID="7a23a841-26d3-4078-bb33-7057c7b42baa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Sep 17 16:49:47.857657 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:49:47.857626    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2b47p_a1850a4d-4d01-4a71-8636-b5918f1426c2/ovn-acl-logging/0.log"
Sep 17 16:49:47.866284 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:49:47.866262    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2b47p_a1850a4d-4d01-4a71-8636-b5918f1426c2/ovn-acl-logging/0.log"
Sep 17 16:49:56.407906 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:49:56.407865    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs" podUID="7a23a841-26d3-4078-bb33-7057c7b42baa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Sep 17 16:49:59.507579 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:49:59.507538    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs" podUID="7a23a841-26d3-4078-bb33-7057c7b42baa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Sep 17 16:50:09.508408 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:09.508375    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs"
Sep 17 16:50:16.181095 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:16.181059    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs"]
Sep 17 16:50:16.181607 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:16.181443    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs" podUID="7a23a841-26d3-4078-bb33-7057c7b42baa" containerName="kserve-container" containerID="cri-o://5e7e0d8929d835e512a62f93fcf91b5f60ee1a6e6cc12e1bbd76a04f2d988fc3" gracePeriod=30
Sep 17 16:50:16.181607 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:16.181480    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs" podUID="7a23a841-26d3-4078-bb33-7057c7b42baa" containerName="kube-rbac-proxy" containerID="cri-o://99b18dab783a8e1dffdd01a13b9ec5856ec007fbf8a2ca915a3cf6aef9ebeb3c" gracePeriod=30
Sep 17 16:50:16.401811 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:16.401757    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs" podUID="7a23a841-26d3-4078-bb33-7057c7b42baa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.68:8643/healthz\": dial tcp 10.134.0.68:8643: connect: connection refused"
Sep 17 16:50:16.846202 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:16.846170    2585 generic.go:356] "Generic (PLEG): container finished" podID="7a23a841-26d3-4078-bb33-7057c7b42baa" containerID="99b18dab783a8e1dffdd01a13b9ec5856ec007fbf8a2ca915a3cf6aef9ebeb3c" exitCode=2
Sep 17 16:50:16.846394 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:16.846251    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs" event={"ID":"7a23a841-26d3-4078-bb33-7057c7b42baa","Type":"ContainerDied","Data":"99b18dab783a8e1dffdd01a13b9ec5856ec007fbf8a2ca915a3cf6aef9ebeb3c"}
Sep 17 16:50:19.508449 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:19.508395    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs" podUID="7a23a841-26d3-4078-bb33-7057c7b42baa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Sep 17 16:50:20.527511 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:20.527488    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs"
Sep 17 16:50:20.645881 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:20.645789    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a23a841-26d3-4078-bb33-7057c7b42baa-kserve-provision-location\") pod \"7a23a841-26d3-4078-bb33-7057c7b42baa\" (UID: \"7a23a841-26d3-4078-bb33-7057c7b42baa\") "
Sep 17 16:50:20.645881 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:20.645837    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a23a841-26d3-4078-bb33-7057c7b42baa-cabundle-cert\") pod \"7a23a841-26d3-4078-bb33-7057c7b42baa\" (UID: \"7a23a841-26d3-4078-bb33-7057c7b42baa\") "
Sep 17 16:50:20.645881 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:20.645865    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a23a841-26d3-4078-bb33-7057c7b42baa-proxy-tls\") pod \"7a23a841-26d3-4078-bb33-7057c7b42baa\" (UID: \"7a23a841-26d3-4078-bb33-7057c7b42baa\") "
Sep 17 16:50:20.646180 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:20.645889    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7vqk\" (UniqueName: \"kubernetes.io/projected/7a23a841-26d3-4078-bb33-7057c7b42baa-kube-api-access-l7vqk\") pod \"7a23a841-26d3-4078-bb33-7057c7b42baa\" (UID: \"7a23a841-26d3-4078-bb33-7057c7b42baa\") "
Sep 17 16:50:20.646180 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:20.645940    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a23a841-26d3-4078-bb33-7057c7b42baa-kube-rbac-proxy-sar-config\") pod \"7a23a841-26d3-4078-bb33-7057c7b42baa\" (UID: \"7a23a841-26d3-4078-bb33-7057c7b42baa\") "
Sep 17 16:50:20.646180 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:20.646146    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a23a841-26d3-4078-bb33-7057c7b42baa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7a23a841-26d3-4078-bb33-7057c7b42baa" (UID: "7a23a841-26d3-4078-bb33-7057c7b42baa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:50:20.646349 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:20.646233    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a23a841-26d3-4078-bb33-7057c7b42baa-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7a23a841-26d3-4078-bb33-7057c7b42baa" (UID: "7a23a841-26d3-4078-bb33-7057c7b42baa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:50:20.646349 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:20.646304    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a23a841-26d3-4078-bb33-7057c7b42baa-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7a23a841-26d3-4078-bb33-7057c7b42baa" (UID: "7a23a841-26d3-4078-bb33-7057c7b42baa"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:50:20.648117 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:20.648085    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a23a841-26d3-4078-bb33-7057c7b42baa-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7a23a841-26d3-4078-bb33-7057c7b42baa" (UID: "7a23a841-26d3-4078-bb33-7057c7b42baa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:50:20.648226 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:20.648123    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a23a841-26d3-4078-bb33-7057c7b42baa-kube-api-access-l7vqk" (OuterVolumeSpecName: "kube-api-access-l7vqk") pod "7a23a841-26d3-4078-bb33-7057c7b42baa" (UID: "7a23a841-26d3-4078-bb33-7057c7b42baa"). InnerVolumeSpecName "kube-api-access-l7vqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:50:20.746601 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:20.746561    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a23a841-26d3-4078-bb33-7057c7b42baa-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:50:20.746601 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:20.746591    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a23a841-26d3-4078-bb33-7057c7b42baa-kserve-provision-location\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:50:20.746601 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:20.746601    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a23a841-26d3-4078-bb33-7057c7b42baa-cabundle-cert\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:50:20.746601 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:20.746611    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a23a841-26d3-4078-bb33-7057c7b42baa-proxy-tls\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:50:20.746871 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:20.746620    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7vqk\" (UniqueName: \"kubernetes.io/projected/7a23a841-26d3-4078-bb33-7057c7b42baa-kube-api-access-l7vqk\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:50:20.868460 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:20.868400    2585 generic.go:356] "Generic (PLEG): container finished" podID="7a23a841-26d3-4078-bb33-7057c7b42baa" containerID="5e7e0d8929d835e512a62f93fcf91b5f60ee1a6e6cc12e1bbd76a04f2d988fc3" exitCode=0
Sep 17 16:50:20.868631 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:20.868472    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs"
Sep 17 16:50:20.868631 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:20.868484    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs" event={"ID":"7a23a841-26d3-4078-bb33-7057c7b42baa","Type":"ContainerDied","Data":"5e7e0d8929d835e512a62f93fcf91b5f60ee1a6e6cc12e1bbd76a04f2d988fc3"}
Sep 17 16:50:20.868631 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:20.868559    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs" event={"ID":"7a23a841-26d3-4078-bb33-7057c7b42baa","Type":"ContainerDied","Data":"0d3e2d33893791c77c82ce52c85548d75ae52e5f9a1d4d598de7f3943604769d"}
Sep 17 16:50:20.868631 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:20.868588    2585 scope.go:117] "RemoveContainer" containerID="99b18dab783a8e1dffdd01a13b9ec5856ec007fbf8a2ca915a3cf6aef9ebeb3c"
Sep 17 16:50:20.877997 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:20.877974    2585 scope.go:117] "RemoveContainer" containerID="5e7e0d8929d835e512a62f93fcf91b5f60ee1a6e6cc12e1bbd76a04f2d988fc3"
Sep 17 16:50:20.885838 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:20.885821    2585 scope.go:117] "RemoveContainer" containerID="273c6104a9e210f753c986dbb555ebb9f52caff68575ae5e433e34f7e3ad9174"
Sep 17 16:50:20.893765 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:20.893738    2585 scope.go:117] "RemoveContainer" containerID="99b18dab783a8e1dffdd01a13b9ec5856ec007fbf8a2ca915a3cf6aef9ebeb3c"
Sep 17 16:50:20.894116 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:50:20.894090    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99b18dab783a8e1dffdd01a13b9ec5856ec007fbf8a2ca915a3cf6aef9ebeb3c\": container with ID starting with 99b18dab783a8e1dffdd01a13b9ec5856ec007fbf8a2ca915a3cf6aef9ebeb3c not found: ID does not exist" containerID="99b18dab783a8e1dffdd01a13b9ec5856ec007fbf8a2ca915a3cf6aef9ebeb3c"
Sep 17 16:50:20.894211 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:20.894127    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99b18dab783a8e1dffdd01a13b9ec5856ec007fbf8a2ca915a3cf6aef9ebeb3c"} err="failed to get container status \"99b18dab783a8e1dffdd01a13b9ec5856ec007fbf8a2ca915a3cf6aef9ebeb3c\": rpc error: code = NotFound desc = could not find container \"99b18dab783a8e1dffdd01a13b9ec5856ec007fbf8a2ca915a3cf6aef9ebeb3c\": container with ID starting with 99b18dab783a8e1dffdd01a13b9ec5856ec007fbf8a2ca915a3cf6aef9ebeb3c not found: ID does not exist"
Sep 17 16:50:20.894211 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:20.894146    2585 scope.go:117] "RemoveContainer" containerID="5e7e0d8929d835e512a62f93fcf91b5f60ee1a6e6cc12e1bbd76a04f2d988fc3"
Sep 17 16:50:20.894563 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:50:20.894494    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e7e0d8929d835e512a62f93fcf91b5f60ee1a6e6cc12e1bbd76a04f2d988fc3\": container with ID starting with 5e7e0d8929d835e512a62f93fcf91b5f60ee1a6e6cc12e1bbd76a04f2d988fc3 not found: ID does not exist" containerID="5e7e0d8929d835e512a62f93fcf91b5f60ee1a6e6cc12e1bbd76a04f2d988fc3"
Sep 17 16:50:20.894789 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:20.894573    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e7e0d8929d835e512a62f93fcf91b5f60ee1a6e6cc12e1bbd76a04f2d988fc3"} err="failed to get container status \"5e7e0d8929d835e512a62f93fcf91b5f60ee1a6e6cc12e1bbd76a04f2d988fc3\": rpc error: code = NotFound desc = could not find container \"5e7e0d8929d835e512a62f93fcf91b5f60ee1a6e6cc12e1bbd76a04f2d988fc3\": container with ID starting with 5e7e0d8929d835e512a62f93fcf91b5f60ee1a6e6cc12e1bbd76a04f2d988fc3 not found: ID does not exist"
Sep 17 16:50:20.894789 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:20.894596    2585 scope.go:117] "RemoveContainer" containerID="273c6104a9e210f753c986dbb555ebb9f52caff68575ae5e433e34f7e3ad9174"
Sep 17 16:50:20.895542 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:50:20.895513    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"273c6104a9e210f753c986dbb555ebb9f52caff68575ae5e433e34f7e3ad9174\": container with ID starting with 273c6104a9e210f753c986dbb555ebb9f52caff68575ae5e433e34f7e3ad9174 not found: ID does not exist" containerID="273c6104a9e210f753c986dbb555ebb9f52caff68575ae5e433e34f7e3ad9174"
Sep 17 16:50:20.895651 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:20.895550    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"273c6104a9e210f753c986dbb555ebb9f52caff68575ae5e433e34f7e3ad9174"} err="failed to get container status \"273c6104a9e210f753c986dbb555ebb9f52caff68575ae5e433e34f7e3ad9174\": rpc error: code = NotFound desc = could not find container \"273c6104a9e210f753c986dbb555ebb9f52caff68575ae5e433e34f7e3ad9174\": container with ID starting with 273c6104a9e210f753c986dbb555ebb9f52caff68575ae5e433e34f7e3ad9174 not found: ID does not exist"
Sep 17 16:50:20.897206 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:20.897144    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs"]
Sep 17 16:50:20.901353 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:20.901327    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-bzzhs"]
Sep 17 16:50:21.401864 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:21.401823    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z"]
Sep 17 16:50:21.402164 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:21.402151    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a23a841-26d3-4078-bb33-7057c7b42baa" containerName="kserve-container"
Sep 17 16:50:21.402215 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:21.402165    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a23a841-26d3-4078-bb33-7057c7b42baa" containerName="kserve-container"
Sep 17 16:50:21.402215 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:21.402185    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a23a841-26d3-4078-bb33-7057c7b42baa" containerName="storage-initializer"
Sep 17 16:50:21.402215 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:21.402190    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a23a841-26d3-4078-bb33-7057c7b42baa" containerName="storage-initializer"
Sep 17 16:50:21.402215 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:21.402200    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a23a841-26d3-4078-bb33-7057c7b42baa" containerName="kube-rbac-proxy"
Sep 17 16:50:21.402215 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:21.402206    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a23a841-26d3-4078-bb33-7057c7b42baa" containerName="kube-rbac-proxy"
Sep 17 16:50:21.402369 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:21.402270    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a23a841-26d3-4078-bb33-7057c7b42baa" containerName="kube-rbac-proxy"
Sep 17 16:50:21.402369 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:21.402281    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a23a841-26d3-4078-bb33-7057c7b42baa" containerName="kserve-container"
Sep 17 16:50:21.406808 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:21.406783    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z"
Sep 17 16:50:21.409322 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:21.409305    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-c11ff7-predictor-serving-cert\""
Sep 17 16:50:21.409487 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:21.409467    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xkc4g\""
Sep 17 16:50:21.409565 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:21.409485    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:50:21.409565 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:21.409493    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-c11ff7-kube-rbac-proxy-sar-config\""
Sep 17 16:50:21.409654 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:21.409588    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 16:50:21.410463 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:21.410448    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 16:50:21.410515 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:21.410484    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 16:50:21.418495 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:21.418470    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z"]
Sep 17 16:50:21.553127 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:21.553086    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6bc871a6-2c25-474a-a672-b3b93bc8a308-kube-rbac-proxy-sar-config\") pod \"isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z\" (UID: \"6bc871a6-2c25-474a-a672-b3b93bc8a308\") " pod="kserve-ci-e2e-test/isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z"
Sep 17 16:50:21.553127 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:21.553127    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thf8p\" (UniqueName: \"kubernetes.io/projected/6bc871a6-2c25-474a-a672-b3b93bc8a308-kube-api-access-thf8p\") pod \"isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z\" (UID: \"6bc871a6-2c25-474a-a672-b3b93bc8a308\") " pod="kserve-ci-e2e-test/isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z"
Sep 17 16:50:21.553705 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:21.553146    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6bc871a6-2c25-474a-a672-b3b93bc8a308-proxy-tls\") pod \"isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z\" (UID: \"6bc871a6-2c25-474a-a672-b3b93bc8a308\") " pod="kserve-ci-e2e-test/isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z"
Sep 17 16:50:21.553705 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:21.553267    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6bc871a6-2c25-474a-a672-b3b93bc8a308-cabundle-cert\") pod \"isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z\" (UID: \"6bc871a6-2c25-474a-a672-b3b93bc8a308\") " pod="kserve-ci-e2e-test/isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z"
Sep 17 16:50:21.553705 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:21.553374    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6bc871a6-2c25-474a-a672-b3b93bc8a308-kserve-provision-location\") pod \"isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z\" (UID: \"6bc871a6-2c25-474a-a672-b3b93bc8a308\") " pod="kserve-ci-e2e-test/isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z"
Sep 17 16:50:21.654777 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:21.654674    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6bc871a6-2c25-474a-a672-b3b93bc8a308-kserve-provision-location\") pod \"isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z\" (UID: \"6bc871a6-2c25-474a-a672-b3b93bc8a308\") " pod="kserve-ci-e2e-test/isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z"
Sep 17 16:50:21.654777 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:21.654731    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6bc871a6-2c25-474a-a672-b3b93bc8a308-kube-rbac-proxy-sar-config\") pod \"isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z\" (UID: \"6bc871a6-2c25-474a-a672-b3b93bc8a308\") " pod="kserve-ci-e2e-test/isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z"
Sep 17 16:50:21.654777 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:21.654750    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thf8p\" (UniqueName: \"kubernetes.io/projected/6bc871a6-2c25-474a-a672-b3b93bc8a308-kube-api-access-thf8p\") pod \"isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z\" (UID: \"6bc871a6-2c25-474a-a672-b3b93bc8a308\") " pod="kserve-ci-e2e-test/isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z"
Sep 17 16:50:21.654777 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:21.654770    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6bc871a6-2c25-474a-a672-b3b93bc8a308-proxy-tls\") pod \"isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z\" (UID: \"6bc871a6-2c25-474a-a672-b3b93bc8a308\") " pod="kserve-ci-e2e-test/isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z"
Sep 17 16:50:21.655137 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:21.654822    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6bc871a6-2c25-474a-a672-b3b93bc8a308-cabundle-cert\") pod \"isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z\" (UID: \"6bc871a6-2c25-474a-a672-b3b93bc8a308\") " pod="kserve-ci-e2e-test/isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z"
Sep 17 16:50:21.655137 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:50:21.654949    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-primary-c11ff7-predictor-serving-cert: secret "isvc-primary-c11ff7-predictor-serving-cert" not found
Sep 17 16:50:21.655137 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:50:21.655025    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bc871a6-2c25-474a-a672-b3b93bc8a308-proxy-tls podName:6bc871a6-2c25-474a-a672-b3b93bc8a308 nodeName:}" failed. No retries permitted until 2026-09-17 16:50:22.155009025 +0000 UTC m=+3377.925340801 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6bc871a6-2c25-474a-a672-b3b93bc8a308-proxy-tls") pod "isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z" (UID: "6bc871a6-2c25-474a-a672-b3b93bc8a308") : secret "isvc-primary-c11ff7-predictor-serving-cert" not found
Sep 17 16:50:21.655282 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:21.655190    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6bc871a6-2c25-474a-a672-b3b93bc8a308-kserve-provision-location\") pod \"isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z\" (UID: \"6bc871a6-2c25-474a-a672-b3b93bc8a308\") " pod="kserve-ci-e2e-test/isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z"
Sep 17 16:50:21.655513 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:21.655494    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6bc871a6-2c25-474a-a672-b3b93bc8a308-kube-rbac-proxy-sar-config\") pod \"isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z\" (UID: \"6bc871a6-2c25-474a-a672-b3b93bc8a308\") " pod="kserve-ci-e2e-test/isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z"
Sep 17 16:50:21.655556 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:21.655509    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6bc871a6-2c25-474a-a672-b3b93bc8a308-cabundle-cert\") pod \"isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z\" (UID: \"6bc871a6-2c25-474a-a672-b3b93bc8a308\") " pod="kserve-ci-e2e-test/isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z"
Sep 17 16:50:21.664478 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:21.664455    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thf8p\" (UniqueName: \"kubernetes.io/projected/6bc871a6-2c25-474a-a672-b3b93bc8a308-kube-api-access-thf8p\") pod \"isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z\" (UID: \"6bc871a6-2c25-474a-a672-b3b93bc8a308\") " pod="kserve-ci-e2e-test/isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z"
Sep 17 16:50:22.159510 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:22.159475    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6bc871a6-2c25-474a-a672-b3b93bc8a308-proxy-tls\") pod \"isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z\" (UID: \"6bc871a6-2c25-474a-a672-b3b93bc8a308\") " pod="kserve-ci-e2e-test/isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z"
Sep 17 16:50:22.162003 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:22.161978    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6bc871a6-2c25-474a-a672-b3b93bc8a308-proxy-tls\") pod \"isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z\" (UID: \"6bc871a6-2c25-474a-a672-b3b93bc8a308\") " pod="kserve-ci-e2e-test/isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z"
Sep 17 16:50:22.318526 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:22.318488    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z"
Sep 17 16:50:22.512814 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:22.512787    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a23a841-26d3-4078-bb33-7057c7b42baa" path="/var/lib/kubelet/pods/7a23a841-26d3-4078-bb33-7057c7b42baa/volumes"
Sep 17 16:50:22.600663 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:22.600587    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z"]
Sep 17 16:50:22.603164 ip-10-0-135-43 kubenswrapper[2585]: W0917 16:50:22.603137    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bc871a6_2c25_474a_a672_b3b93bc8a308.slice/crio-e6826cd559f1bbf4e3057751863a49963cd0431835874078cba9289166efdcc9 WatchSource:0}: Error finding container e6826cd559f1bbf4e3057751863a49963cd0431835874078cba9289166efdcc9: Status 404 returned error can't find the container with id e6826cd559f1bbf4e3057751863a49963cd0431835874078cba9289166efdcc9
Sep 17 16:50:22.880646 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:22.880551    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z" event={"ID":"6bc871a6-2c25-474a-a672-b3b93bc8a308","Type":"ContainerStarted","Data":"baeee853a6a184a3748cdd135ec69dbc4d32e657ee2dcc9eacffd977191b3b3d"}
Sep 17 16:50:22.880646 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:22.880634    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z" event={"ID":"6bc871a6-2c25-474a-a672-b3b93bc8a308","Type":"ContainerStarted","Data":"e6826cd559f1bbf4e3057751863a49963cd0431835874078cba9289166efdcc9"}
Sep 17 16:50:26.901037 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:26.900945    2585 generic.go:356] "Generic (PLEG): container finished" podID="6bc871a6-2c25-474a-a672-b3b93bc8a308" containerID="baeee853a6a184a3748cdd135ec69dbc4d32e657ee2dcc9eacffd977191b3b3d" exitCode=0
Sep 17 16:50:26.901037 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:26.901022    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z" event={"ID":"6bc871a6-2c25-474a-a672-b3b93bc8a308","Type":"ContainerDied","Data":"baeee853a6a184a3748cdd135ec69dbc4d32e657ee2dcc9eacffd977191b3b3d"}
Sep 17 16:50:27.907922 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:27.907887    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z" event={"ID":"6bc871a6-2c25-474a-a672-b3b93bc8a308","Type":"ContainerStarted","Data":"e9a5e43ff11bff58090031fb42f28cfde57716a99f2926ac6b93f8b84f00d6e5"}
Sep 17 16:50:27.908293 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:27.907951    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z" event={"ID":"6bc871a6-2c25-474a-a672-b3b93bc8a308","Type":"ContainerStarted","Data":"f1d4d65e6017f2e3e3e0c022590d0a3f999387f11fcd30960de91eb718261934"}
Sep 17 16:50:27.908293 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:27.908213    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z"
Sep 17 16:50:27.908385 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:27.908348    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z"
Sep 17 16:50:27.909534 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:27.909509    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z" podUID="6bc871a6-2c25-474a-a672-b3b93bc8a308" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Sep 17 16:50:27.938473 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:27.938401    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z" podStartSLOduration=3.084406655 podStartE2EDuration="6.938383729s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.853977074s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:50:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:50:27.936870581 +0000 UTC m=+3383.707202381" watchObservedRunningTime="2026-09-17 16:50:27.938383729 +0000 UTC m=+3383.708715527"
Sep 17 16:50:28.912136 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:28.912094    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z" podUID="6bc871a6-2c25-474a-a672-b3b93bc8a308" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Sep 17 16:50:33.918119 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:33.918091    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z"
Sep 17 16:50:33.918647 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:33.918619    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z" podUID="6bc871a6-2c25-474a-a672-b3b93bc8a308" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Sep 17 16:50:43.918672 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:43.918631    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z" podUID="6bc871a6-2c25-474a-a672-b3b93bc8a308" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Sep 17 16:50:53.919276 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:50:53.919188    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z" podUID="6bc871a6-2c25-474a-a672-b3b93bc8a308" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Sep 17 16:51:03.919061 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:51:03.919016    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z" podUID="6bc871a6-2c25-474a-a672-b3b93bc8a308" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Sep 17 16:51:13.919244 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:51:13.919204    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z" podUID="6bc871a6-2c25-474a-a672-b3b93bc8a308" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Sep 17 16:51:23.919463 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:51:23.919390    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z" podUID="6bc871a6-2c25-474a-a672-b3b93bc8a308" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Sep 17 16:51:33.918761 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:51:33.918716    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z" podUID="6bc871a6-2c25-474a-a672-b3b93bc8a308" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.69:8080: connect: connection refused"
Sep 17 16:51:43.919234 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:51:43.919199    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z"
Sep 17 16:52:06.733638 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:06.733601    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-c11ff7-predictor-5b75b4bb59-9kzx4"]
Sep 17 16:52:06.737255 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:06.737237    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-c11ff7-predictor-5b75b4bb59-9kzx4"
Sep 17 16:52:06.740797 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:06.740770    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-c11ff7-dockercfg-g2r2q\""
Sep 17 16:52:06.740953 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:06.740932    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-c11ff7\""
Sep 17 16:52:06.741740 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:06.741725    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-c11ff7-predictor-serving-cert\""
Sep 17 16:52:06.741819 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:06.741786    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-c11ff7-kube-rbac-proxy-sar-config\""
Sep 17 16:52:06.755893 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:06.755865    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-c11ff7-predictor-5b75b4bb59-9kzx4"]
Sep 17 16:52:06.825576 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:06.825544    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28625585-1263-4a19-b85f-8171a75647db-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-c11ff7-predictor-5b75b4bb59-9kzx4\" (UID: \"28625585-1263-4a19-b85f-8171a75647db\") " pod="kserve-ci-e2e-test/isvc-secondary-c11ff7-predictor-5b75b4bb59-9kzx4"
Sep 17 16:52:06.825758 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:06.825586    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/28625585-1263-4a19-b85f-8171a75647db-cabundle-cert\") pod \"isvc-secondary-c11ff7-predictor-5b75b4bb59-9kzx4\" (UID: \"28625585-1263-4a19-b85f-8171a75647db\") " pod="kserve-ci-e2e-test/isvc-secondary-c11ff7-predictor-5b75b4bb59-9kzx4"
Sep 17 16:52:06.825758 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:06.825614    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28625585-1263-4a19-b85f-8171a75647db-kserve-provision-location\") pod \"isvc-secondary-c11ff7-predictor-5b75b4bb59-9kzx4\" (UID: \"28625585-1263-4a19-b85f-8171a75647db\") " pod="kserve-ci-e2e-test/isvc-secondary-c11ff7-predictor-5b75b4bb59-9kzx4"
Sep 17 16:52:06.825758 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:06.825694    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28625585-1263-4a19-b85f-8171a75647db-proxy-tls\") pod \"isvc-secondary-c11ff7-predictor-5b75b4bb59-9kzx4\" (UID: \"28625585-1263-4a19-b85f-8171a75647db\") " pod="kserve-ci-e2e-test/isvc-secondary-c11ff7-predictor-5b75b4bb59-9kzx4"
Sep 17 16:52:06.825925 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:06.825780    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twmfz\" (UniqueName: \"kubernetes.io/projected/28625585-1263-4a19-b85f-8171a75647db-kube-api-access-twmfz\") pod \"isvc-secondary-c11ff7-predictor-5b75b4bb59-9kzx4\" (UID: \"28625585-1263-4a19-b85f-8171a75647db\") " pod="kserve-ci-e2e-test/isvc-secondary-c11ff7-predictor-5b75b4bb59-9kzx4"
Sep 17 16:52:06.926594 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:06.926554    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28625585-1263-4a19-b85f-8171a75647db-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-c11ff7-predictor-5b75b4bb59-9kzx4\" (UID: \"28625585-1263-4a19-b85f-8171a75647db\") " pod="kserve-ci-e2e-test/isvc-secondary-c11ff7-predictor-5b75b4bb59-9kzx4"
Sep 17 16:52:06.926594 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:06.926601    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/28625585-1263-4a19-b85f-8171a75647db-cabundle-cert\") pod \"isvc-secondary-c11ff7-predictor-5b75b4bb59-9kzx4\" (UID: \"28625585-1263-4a19-b85f-8171a75647db\") " pod="kserve-ci-e2e-test/isvc-secondary-c11ff7-predictor-5b75b4bb59-9kzx4"
Sep 17 16:52:06.926869 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:06.926626    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28625585-1263-4a19-b85f-8171a75647db-kserve-provision-location\") pod \"isvc-secondary-c11ff7-predictor-5b75b4bb59-9kzx4\" (UID: \"28625585-1263-4a19-b85f-8171a75647db\") " pod="kserve-ci-e2e-test/isvc-secondary-c11ff7-predictor-5b75b4bb59-9kzx4"
Sep 17 16:52:06.926869 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:06.926790    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28625585-1263-4a19-b85f-8171a75647db-proxy-tls\") pod \"isvc-secondary-c11ff7-predictor-5b75b4bb59-9kzx4\" (UID: \"28625585-1263-4a19-b85f-8171a75647db\") " pod="kserve-ci-e2e-test/isvc-secondary-c11ff7-predictor-5b75b4bb59-9kzx4"
Sep 17 16:52:06.926869 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:06.926849    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twmfz\" (UniqueName: \"kubernetes.io/projected/28625585-1263-4a19-b85f-8171a75647db-kube-api-access-twmfz\") pod \"isvc-secondary-c11ff7-predictor-5b75b4bb59-9kzx4\" (UID: \"28625585-1263-4a19-b85f-8171a75647db\") " pod="kserve-ci-e2e-test/isvc-secondary-c11ff7-predictor-5b75b4bb59-9kzx4"
Sep 17 16:52:06.927040 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:06.927002    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28625585-1263-4a19-b85f-8171a75647db-kserve-provision-location\") pod \"isvc-secondary-c11ff7-predictor-5b75b4bb59-9kzx4\" (UID: \"28625585-1263-4a19-b85f-8171a75647db\") " pod="kserve-ci-e2e-test/isvc-secondary-c11ff7-predictor-5b75b4bb59-9kzx4"
Sep 17 16:52:06.927339 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:06.927316    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/28625585-1263-4a19-b85f-8171a75647db-cabundle-cert\") pod \"isvc-secondary-c11ff7-predictor-5b75b4bb59-9kzx4\" (UID: \"28625585-1263-4a19-b85f-8171a75647db\") " pod="kserve-ci-e2e-test/isvc-secondary-c11ff7-predictor-5b75b4bb59-9kzx4"
Sep 17 16:52:06.927460 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:06.927356    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28625585-1263-4a19-b85f-8171a75647db-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-c11ff7-predictor-5b75b4bb59-9kzx4\" (UID: \"28625585-1263-4a19-b85f-8171a75647db\") " pod="kserve-ci-e2e-test/isvc-secondary-c11ff7-predictor-5b75b4bb59-9kzx4"
Sep 17 16:52:06.929203 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:06.929187    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28625585-1263-4a19-b85f-8171a75647db-proxy-tls\") pod \"isvc-secondary-c11ff7-predictor-5b75b4bb59-9kzx4\" (UID: \"28625585-1263-4a19-b85f-8171a75647db\") " pod="kserve-ci-e2e-test/isvc-secondary-c11ff7-predictor-5b75b4bb59-9kzx4"
Sep 17 16:52:06.943347 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:06.943313    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twmfz\" (UniqueName: \"kubernetes.io/projected/28625585-1263-4a19-b85f-8171a75647db-kube-api-access-twmfz\") pod \"isvc-secondary-c11ff7-predictor-5b75b4bb59-9kzx4\" (UID: \"28625585-1263-4a19-b85f-8171a75647db\") " pod="kserve-ci-e2e-test/isvc-secondary-c11ff7-predictor-5b75b4bb59-9kzx4"
Sep 17 16:52:07.047821 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:07.047729    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-c11ff7-predictor-5b75b4bb59-9kzx4"
Sep 17 16:52:07.329331 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:07.329303    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-c11ff7-predictor-5b75b4bb59-9kzx4"]
Sep 17 16:52:07.329897 ip-10-0-135-43 kubenswrapper[2585]: W0917 16:52:07.329873    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28625585_1263_4a19_b85f_8171a75647db.slice/crio-325ad189a9955c3d45481269f58895c17dd68d7ac14359bb37ac8502c779009d WatchSource:0}: Error finding container 325ad189a9955c3d45481269f58895c17dd68d7ac14359bb37ac8502c779009d: Status 404 returned error can't find the container with id 325ad189a9955c3d45481269f58895c17dd68d7ac14359bb37ac8502c779009d
Sep 17 16:52:07.357152 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:07.357121    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-c11ff7-predictor-5b75b4bb59-9kzx4" event={"ID":"28625585-1263-4a19-b85f-8171a75647db","Type":"ContainerStarted","Data":"325ad189a9955c3d45481269f58895c17dd68d7ac14359bb37ac8502c779009d"}
Sep 17 16:52:08.363910 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:08.363872    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-c11ff7-predictor-5b75b4bb59-9kzx4" event={"ID":"28625585-1263-4a19-b85f-8171a75647db","Type":"ContainerStarted","Data":"9ad8cbd9191d5969681b28833cd5a26eb8664b20ac2c8a7f47b5cdd98127d4cc"}
Sep 17 16:52:12.384460 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:12.384413    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-c11ff7-predictor-5b75b4bb59-9kzx4_28625585-1263-4a19-b85f-8171a75647db/storage-initializer/0.log"
Sep 17 16:52:12.384876 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:12.384476    2585 generic.go:356] "Generic (PLEG): container finished" podID="28625585-1263-4a19-b85f-8171a75647db" containerID="9ad8cbd9191d5969681b28833cd5a26eb8664b20ac2c8a7f47b5cdd98127d4cc" exitCode=1
Sep 17 16:52:12.384876 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:12.384514    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-c11ff7-predictor-5b75b4bb59-9kzx4" event={"ID":"28625585-1263-4a19-b85f-8171a75647db","Type":"ContainerDied","Data":"9ad8cbd9191d5969681b28833cd5a26eb8664b20ac2c8a7f47b5cdd98127d4cc"}
Sep 17 16:52:13.390875 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:13.390842    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-c11ff7-predictor-5b75b4bb59-9kzx4_28625585-1263-4a19-b85f-8171a75647db/storage-initializer/0.log"
Sep 17 16:52:13.391268 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:13.390919    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-c11ff7-predictor-5b75b4bb59-9kzx4" event={"ID":"28625585-1263-4a19-b85f-8171a75647db","Type":"ContainerStarted","Data":"f9965f5e90ae1beb927aebe0d16df37b8fbfe80bdab8ef8c53b44009d01652ba"}
Sep 17 16:52:17.410301 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:17.410216    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-c11ff7-predictor-5b75b4bb59-9kzx4_28625585-1263-4a19-b85f-8171a75647db/storage-initializer/1.log"
Sep 17 16:52:17.410771 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:17.410663    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-c11ff7-predictor-5b75b4bb59-9kzx4_28625585-1263-4a19-b85f-8171a75647db/storage-initializer/0.log"
Sep 17 16:52:17.410771 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:17.410700    2585 generic.go:356] "Generic (PLEG): container finished" podID="28625585-1263-4a19-b85f-8171a75647db" containerID="f9965f5e90ae1beb927aebe0d16df37b8fbfe80bdab8ef8c53b44009d01652ba" exitCode=1
Sep 17 16:52:17.410901 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:17.410772    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-c11ff7-predictor-5b75b4bb59-9kzx4" event={"ID":"28625585-1263-4a19-b85f-8171a75647db","Type":"ContainerDied","Data":"f9965f5e90ae1beb927aebe0d16df37b8fbfe80bdab8ef8c53b44009d01652ba"}
Sep 17 16:52:17.410901 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:17.410836    2585 scope.go:117] "RemoveContainer" containerID="9ad8cbd9191d5969681b28833cd5a26eb8664b20ac2c8a7f47b5cdd98127d4cc"
Sep 17 16:52:17.411233 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:17.411216    2585 scope.go:117] "RemoveContainer" containerID="9ad8cbd9191d5969681b28833cd5a26eb8664b20ac2c8a7f47b5cdd98127d4cc"
Sep 17 16:52:17.422020 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:52:17.421991    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-c11ff7-predictor-5b75b4bb59-9kzx4_kserve-ci-e2e-test(28625585-1263-4a19-b85f-8171a75647db)\"" pod="kserve-ci-e2e-test/isvc-secondary-c11ff7-predictor-5b75b4bb59-9kzx4" podUID="28625585-1263-4a19-b85f-8171a75647db"
Sep 17 16:52:17.422214 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:52:17.422186    2585 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-c11ff7-predictor-5b75b4bb59-9kzx4_kserve-ci-e2e-test_28625585-1263-4a19-b85f-8171a75647db_0 in pod sandbox 325ad189a9955c3d45481269f58895c17dd68d7ac14359bb37ac8502c779009d from index: no such id: '9ad8cbd9191d5969681b28833cd5a26eb8664b20ac2c8a7f47b5cdd98127d4cc'" containerID="9ad8cbd9191d5969681b28833cd5a26eb8664b20ac2c8a7f47b5cdd98127d4cc"
Sep 17 16:52:17.422267 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:17.422227    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ad8cbd9191d5969681b28833cd5a26eb8664b20ac2c8a7f47b5cdd98127d4cc"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-c11ff7-predictor-5b75b4bb59-9kzx4_kserve-ci-e2e-test_28625585-1263-4a19-b85f-8171a75647db_0 in pod sandbox 325ad189a9955c3d45481269f58895c17dd68d7ac14359bb37ac8502c779009d from index: no such id: '9ad8cbd9191d5969681b28833cd5a26eb8664b20ac2c8a7f47b5cdd98127d4cc'"
Sep 17 16:52:18.420025 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:18.419992    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-c11ff7-predictor-5b75b4bb59-9kzx4_28625585-1263-4a19-b85f-8171a75647db/storage-initializer/1.log"
Sep 17 16:52:24.715756 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:24.715714    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-c11ff7-predictor-5b75b4bb59-9kzx4"]
Sep 17 16:52:24.761839 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:24.761802    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z"]
Sep 17 16:52:24.762123 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:24.762094    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z" podUID="6bc871a6-2c25-474a-a672-b3b93bc8a308" containerName="kserve-container" containerID="cri-o://f1d4d65e6017f2e3e3e0c022590d0a3f999387f11fcd30960de91eb718261934" gracePeriod=30
Sep 17 16:52:24.762282 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:24.762158    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z" podUID="6bc871a6-2c25-474a-a672-b3b93bc8a308" containerName="kube-rbac-proxy" containerID="cri-o://e9a5e43ff11bff58090031fb42f28cfde57716a99f2926ac6b93f8b84f00d6e5" gracePeriod=30
Sep 17 16:52:24.883544 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:24.883520    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-c11ff7-predictor-5b75b4bb59-9kzx4_28625585-1263-4a19-b85f-8171a75647db/storage-initializer/1.log"
Sep 17 16:52:24.885671 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:24.885656    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-c11ff7-predictor-5b75b4bb59-9kzx4"
Sep 17 16:52:24.995381 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:24.995288    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/28625585-1263-4a19-b85f-8171a75647db-cabundle-cert\") pod \"28625585-1263-4a19-b85f-8171a75647db\" (UID: \"28625585-1263-4a19-b85f-8171a75647db\") "
Sep 17 16:52:24.995381 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:24.995362    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28625585-1263-4a19-b85f-8171a75647db-kserve-provision-location\") pod \"28625585-1263-4a19-b85f-8171a75647db\" (UID: \"28625585-1263-4a19-b85f-8171a75647db\") "
Sep 17 16:52:24.995646 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:24.995393    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-twmfz\" (UniqueName: \"kubernetes.io/projected/28625585-1263-4a19-b85f-8171a75647db-kube-api-access-twmfz\") pod \"28625585-1263-4a19-b85f-8171a75647db\" (UID: \"28625585-1263-4a19-b85f-8171a75647db\") "
Sep 17 16:52:24.995646 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:24.995498    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28625585-1263-4a19-b85f-8171a75647db-kube-rbac-proxy-sar-config\") pod \"28625585-1263-4a19-b85f-8171a75647db\" (UID: \"28625585-1263-4a19-b85f-8171a75647db\") "
Sep 17 16:52:24.995646 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:24.995525    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28625585-1263-4a19-b85f-8171a75647db-proxy-tls\") pod \"28625585-1263-4a19-b85f-8171a75647db\" (UID: \"28625585-1263-4a19-b85f-8171a75647db\") "
Sep 17 16:52:24.995803 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:24.995682    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28625585-1263-4a19-b85f-8171a75647db-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "28625585-1263-4a19-b85f-8171a75647db" (UID: "28625585-1263-4a19-b85f-8171a75647db"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:52:24.995803 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:24.995769    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28625585-1263-4a19-b85f-8171a75647db-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "28625585-1263-4a19-b85f-8171a75647db" (UID: "28625585-1263-4a19-b85f-8171a75647db"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:52:24.995803 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:24.995795    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28625585-1263-4a19-b85f-8171a75647db-kserve-provision-location\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:52:24.995911 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:24.995864    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28625585-1263-4a19-b85f-8171a75647db-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "28625585-1263-4a19-b85f-8171a75647db" (UID: "28625585-1263-4a19-b85f-8171a75647db"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:52:24.997661 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:24.997637    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28625585-1263-4a19-b85f-8171a75647db-kube-api-access-twmfz" (OuterVolumeSpecName: "kube-api-access-twmfz") pod "28625585-1263-4a19-b85f-8171a75647db" (UID: "28625585-1263-4a19-b85f-8171a75647db"). InnerVolumeSpecName "kube-api-access-twmfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:52:24.997661 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:24.997645    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28625585-1263-4a19-b85f-8171a75647db-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "28625585-1263-4a19-b85f-8171a75647db" (UID: "28625585-1263-4a19-b85f-8171a75647db"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:52:25.096213 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:25.096171    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twmfz\" (UniqueName: \"kubernetes.io/projected/28625585-1263-4a19-b85f-8171a75647db-kube-api-access-twmfz\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:52:25.096213 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:25.096204    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28625585-1263-4a19-b85f-8171a75647db-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:52:25.096213 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:25.096214    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28625585-1263-4a19-b85f-8171a75647db-proxy-tls\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:52:25.096495 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:25.096227    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/28625585-1263-4a19-b85f-8171a75647db-cabundle-cert\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:52:25.454147 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:25.454119    2585 generic.go:356] "Generic (PLEG): container finished" podID="6bc871a6-2c25-474a-a672-b3b93bc8a308" containerID="e9a5e43ff11bff58090031fb42f28cfde57716a99f2926ac6b93f8b84f00d6e5" exitCode=2
Sep 17 16:52:25.454309 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:25.454191    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z" event={"ID":"6bc871a6-2c25-474a-a672-b3b93bc8a308","Type":"ContainerDied","Data":"e9a5e43ff11bff58090031fb42f28cfde57716a99f2926ac6b93f8b84f00d6e5"}
Sep 17 16:52:25.455467 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:25.455450    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-c11ff7-predictor-5b75b4bb59-9kzx4_28625585-1263-4a19-b85f-8171a75647db/storage-initializer/1.log"
Sep 17 16:52:25.455565 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:25.455499    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-c11ff7-predictor-5b75b4bb59-9kzx4" event={"ID":"28625585-1263-4a19-b85f-8171a75647db","Type":"ContainerDied","Data":"325ad189a9955c3d45481269f58895c17dd68d7ac14359bb37ac8502c779009d"}
Sep 17 16:52:25.455565 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:25.455539    2585 scope.go:117] "RemoveContainer" containerID="f9965f5e90ae1beb927aebe0d16df37b8fbfe80bdab8ef8c53b44009d01652ba"
Sep 17 16:52:25.455565 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:25.455543    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-c11ff7-predictor-5b75b4bb59-9kzx4"
Sep 17 16:52:25.497433 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:25.497373    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-c11ff7-predictor-5b75b4bb59-9kzx4"]
Sep 17 16:52:25.506285 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:25.506252    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-c11ff7-predictor-5b75b4bb59-9kzx4"]
Sep 17 16:52:26.513179 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:26.513146    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28625585-1263-4a19-b85f-8171a75647db" path="/var/lib/kubelet/pods/28625585-1263-4a19-b85f-8171a75647db/volumes"
Sep 17 16:52:28.912823 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:28.912779    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z" podUID="6bc871a6-2c25-474a-a672-b3b93bc8a308" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.69:8643/healthz\": dial tcp 10.134.0.69:8643: connect: connection refused"
Sep 17 16:52:29.712358 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:29.712333    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z"
Sep 17 16:52:29.731347 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:29.731312    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-thf8p\" (UniqueName: \"kubernetes.io/projected/6bc871a6-2c25-474a-a672-b3b93bc8a308-kube-api-access-thf8p\") pod \"6bc871a6-2c25-474a-a672-b3b93bc8a308\" (UID: \"6bc871a6-2c25-474a-a672-b3b93bc8a308\") "
Sep 17 16:52:29.731347 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:29.731346    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6bc871a6-2c25-474a-a672-b3b93bc8a308-cabundle-cert\") pod \"6bc871a6-2c25-474a-a672-b3b93bc8a308\" (UID: \"6bc871a6-2c25-474a-a672-b3b93bc8a308\") "
Sep 17 16:52:29.731653 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:29.731375    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6bc871a6-2c25-474a-a672-b3b93bc8a308-kube-rbac-proxy-sar-config\") pod \"6bc871a6-2c25-474a-a672-b3b93bc8a308\" (UID: \"6bc871a6-2c25-474a-a672-b3b93bc8a308\") "
Sep 17 16:52:29.731653 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:29.731411    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6bc871a6-2c25-474a-a672-b3b93bc8a308-kserve-provision-location\") pod \"6bc871a6-2c25-474a-a672-b3b93bc8a308\" (UID: \"6bc871a6-2c25-474a-a672-b3b93bc8a308\") "
Sep 17 16:52:29.731653 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:29.731457    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6bc871a6-2c25-474a-a672-b3b93bc8a308-proxy-tls\") pod \"6bc871a6-2c25-474a-a672-b3b93bc8a308\" (UID: \"6bc871a6-2c25-474a-a672-b3b93bc8a308\") "
Sep 17 16:52:29.731805 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:29.731737    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bc871a6-2c25-474a-a672-b3b93bc8a308-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6bc871a6-2c25-474a-a672-b3b93bc8a308" (UID: "6bc871a6-2c25-474a-a672-b3b93bc8a308"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:52:29.731805 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:29.731787    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bc871a6-2c25-474a-a672-b3b93bc8a308-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6bc871a6-2c25-474a-a672-b3b93bc8a308" (UID: "6bc871a6-2c25-474a-a672-b3b93bc8a308"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:52:29.731879 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:29.731811    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bc871a6-2c25-474a-a672-b3b93bc8a308-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6bc871a6-2c25-474a-a672-b3b93bc8a308" (UID: "6bc871a6-2c25-474a-a672-b3b93bc8a308"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:52:29.733583 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:29.733557    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bc871a6-2c25-474a-a672-b3b93bc8a308-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6bc871a6-2c25-474a-a672-b3b93bc8a308" (UID: "6bc871a6-2c25-474a-a672-b3b93bc8a308"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:52:29.733683 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:29.733607    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bc871a6-2c25-474a-a672-b3b93bc8a308-kube-api-access-thf8p" (OuterVolumeSpecName: "kube-api-access-thf8p") pod "6bc871a6-2c25-474a-a672-b3b93bc8a308" (UID: "6bc871a6-2c25-474a-a672-b3b93bc8a308"). InnerVolumeSpecName "kube-api-access-thf8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:52:29.832825 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:29.832736    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6bc871a6-2c25-474a-a672-b3b93bc8a308-proxy-tls\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:52:29.832825 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:29.832767    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thf8p\" (UniqueName: \"kubernetes.io/projected/6bc871a6-2c25-474a-a672-b3b93bc8a308-kube-api-access-thf8p\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:52:29.832825 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:29.832780    2585 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6bc871a6-2c25-474a-a672-b3b93bc8a308-cabundle-cert\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:52:29.832825 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:29.832789    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6bc871a6-2c25-474a-a672-b3b93bc8a308-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:52:29.832825 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:29.832797    2585 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6bc871a6-2c25-474a-a672-b3b93bc8a308-kserve-provision-location\") on node \"ip-10-0-135-43.ec2.internal\" DevicePath \"\""
Sep 17 16:52:30.482819 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:30.482784    2585 generic.go:356] "Generic (PLEG): container finished" podID="6bc871a6-2c25-474a-a672-b3b93bc8a308" containerID="f1d4d65e6017f2e3e3e0c022590d0a3f999387f11fcd30960de91eb718261934" exitCode=0
Sep 17 16:52:30.483315 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:30.482830    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z" event={"ID":"6bc871a6-2c25-474a-a672-b3b93bc8a308","Type":"ContainerDied","Data":"f1d4d65e6017f2e3e3e0c022590d0a3f999387f11fcd30960de91eb718261934"}
Sep 17 16:52:30.483315 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:30.482850    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z"
Sep 17 16:52:30.483315 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:30.482879    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z" event={"ID":"6bc871a6-2c25-474a-a672-b3b93bc8a308","Type":"ContainerDied","Data":"e6826cd559f1bbf4e3057751863a49963cd0431835874078cba9289166efdcc9"}
Sep 17 16:52:30.483315 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:30.482906    2585 scope.go:117] "RemoveContainer" containerID="e9a5e43ff11bff58090031fb42f28cfde57716a99f2926ac6b93f8b84f00d6e5"
Sep 17 16:52:30.492453 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:30.492412    2585 scope.go:117] "RemoveContainer" containerID="f1d4d65e6017f2e3e3e0c022590d0a3f999387f11fcd30960de91eb718261934"
Sep 17 16:52:30.499926 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:30.499908    2585 scope.go:117] "RemoveContainer" containerID="baeee853a6a184a3748cdd135ec69dbc4d32e657ee2dcc9eacffd977191b3b3d"
Sep 17 16:52:30.507957 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:30.507935    2585 scope.go:117] "RemoveContainer" containerID="e9a5e43ff11bff58090031fb42f28cfde57716a99f2926ac6b93f8b84f00d6e5"
Sep 17 16:52:30.508842 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:52:30.508344    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9a5e43ff11bff58090031fb42f28cfde57716a99f2926ac6b93f8b84f00d6e5\": container with ID starting with e9a5e43ff11bff58090031fb42f28cfde57716a99f2926ac6b93f8b84f00d6e5 not found: ID does not exist" containerID="e9a5e43ff11bff58090031fb42f28cfde57716a99f2926ac6b93f8b84f00d6e5"
Sep 17 16:52:30.509253 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:30.509037    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9a5e43ff11bff58090031fb42f28cfde57716a99f2926ac6b93f8b84f00d6e5"} err="failed to get container status \"e9a5e43ff11bff58090031fb42f28cfde57716a99f2926ac6b93f8b84f00d6e5\": rpc error: code = NotFound desc = could not find container \"e9a5e43ff11bff58090031fb42f28cfde57716a99f2926ac6b93f8b84f00d6e5\": container with ID starting with e9a5e43ff11bff58090031fb42f28cfde57716a99f2926ac6b93f8b84f00d6e5 not found: ID does not exist"
Sep 17 16:52:30.509253 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:30.509091    2585 scope.go:117] "RemoveContainer" containerID="f1d4d65e6017f2e3e3e0c022590d0a3f999387f11fcd30960de91eb718261934"
Sep 17 16:52:30.513445 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:52:30.513396    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1d4d65e6017f2e3e3e0c022590d0a3f999387f11fcd30960de91eb718261934\": container with ID starting with f1d4d65e6017f2e3e3e0c022590d0a3f999387f11fcd30960de91eb718261934 not found: ID does not exist" containerID="f1d4d65e6017f2e3e3e0c022590d0a3f999387f11fcd30960de91eb718261934"
Sep 17 16:52:30.513591 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:30.513455    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1d4d65e6017f2e3e3e0c022590d0a3f999387f11fcd30960de91eb718261934"} err="failed to get container status \"f1d4d65e6017f2e3e3e0c022590d0a3f999387f11fcd30960de91eb718261934\": rpc error: code = NotFound desc = could not find container \"f1d4d65e6017f2e3e3e0c022590d0a3f999387f11fcd30960de91eb718261934\": container with ID starting with f1d4d65e6017f2e3e3e0c022590d0a3f999387f11fcd30960de91eb718261934 not found: ID does not exist"
Sep 17 16:52:30.513591 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:30.513481    2585 scope.go:117] "RemoveContainer" containerID="baeee853a6a184a3748cdd135ec69dbc4d32e657ee2dcc9eacffd977191b3b3d"
Sep 17 16:52:30.514932 ip-10-0-135-43 kubenswrapper[2585]: E0917 16:52:30.514895    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baeee853a6a184a3748cdd135ec69dbc4d32e657ee2dcc9eacffd977191b3b3d\": container with ID starting with baeee853a6a184a3748cdd135ec69dbc4d32e657ee2dcc9eacffd977191b3b3d not found: ID does not exist" containerID="baeee853a6a184a3748cdd135ec69dbc4d32e657ee2dcc9eacffd977191b3b3d"
Sep 17 16:52:30.515057 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:30.514940    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baeee853a6a184a3748cdd135ec69dbc4d32e657ee2dcc9eacffd977191b3b3d"} err="failed to get container status \"baeee853a6a184a3748cdd135ec69dbc4d32e657ee2dcc9eacffd977191b3b3d\": rpc error: code = NotFound desc = could not find container \"baeee853a6a184a3748cdd135ec69dbc4d32e657ee2dcc9eacffd977191b3b3d\": container with ID starting with baeee853a6a184a3748cdd135ec69dbc4d32e657ee2dcc9eacffd977191b3b3d not found: ID does not exist"
Sep 17 16:52:30.518624 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:30.518597    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z"]
Sep 17 16:52:30.518722 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:30.518631    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-c11ff7-predictor-77b4b498bd-q5m2z"]
Sep 17 16:52:32.512735 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:52:32.512704    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bc871a6-2c25-474a-a672-b3b93bc8a308" path="/var/lib/kubelet/pods/6bc871a6-2c25-474a-a672-b3b93bc8a308/volumes"
Sep 17 16:53:12.376693 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:12.376662    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2f4bn_94d1451b-3eae-4481-a504-c29b4315e62e/global-pull-secret-syncer/0.log"
Sep 17 16:53:12.568357 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:12.568322    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ntl9t_8249ef17-a5c4-414e-a2ca-a22cc46ef3eb/konnectivity-agent/0.log"
Sep 17 16:53:12.672002 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:12.671911    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-43.ec2.internal_5692cd509d685463b01f1d043c0bcb98/haproxy/0.log"
Sep 17 16:53:17.538340 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:17.538306    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-gsbw2_0b74e69e-14e3-41ab-a94c-44487eafb173/cluster-monitoring-operator/0.log"
Sep 17 16:53:17.600888 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:17.600854    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-5bcr2_c1fb615a-b839-4e47-b63b-47eec13d57cc/kube-state-metrics/0.log"
Sep 17 16:53:17.648591 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:17.648563    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-5bcr2_c1fb615a-b839-4e47-b63b-47eec13d57cc/kube-rbac-proxy-main/0.log"
Sep 17 16:53:17.718258 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:17.718226    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-5bcr2_c1fb615a-b839-4e47-b63b-47eec13d57cc/kube-rbac-proxy-self/0.log"
Sep 17 16:53:18.012832 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:18.012803    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-99ndh_8368d252-ce4a-4aea-94d4-7956729c3219/node-exporter/0.log"
Sep 17 16:53:18.094951 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:18.094925    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-99ndh_8368d252-ce4a-4aea-94d4-7956729c3219/kube-rbac-proxy/0.log"
Sep 17 16:53:18.165593 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:18.165561    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-99ndh_8368d252-ce4a-4aea-94d4-7956729c3219/init-textfile/0.log"
Sep 17 16:53:19.290509 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:19.290479    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-fch49_cf657178-2022-4f3b-a53e-c45eb342c5e6/prometheus-operator/0.log"
Sep 17 16:53:19.316665 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:19.316633    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-fch49_cf657178-2022-4f3b-a53e-c45eb342c5e6/kube-rbac-proxy/0.log"
Sep 17 16:53:19.528696 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:19.528671    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5976878f85-g6lhx_fc63c498-e311-4eac-9aba-166507716705/thanos-query/0.log"
Sep 17 16:53:19.555011 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:19.554933    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5976878f85-g6lhx_fc63c498-e311-4eac-9aba-166507716705/kube-rbac-proxy-web/0.log"
Sep 17 16:53:19.579585 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:19.579562    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5976878f85-g6lhx_fc63c498-e311-4eac-9aba-166507716705/kube-rbac-proxy/0.log"
Sep 17 16:53:19.608759 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:19.608727    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5976878f85-g6lhx_fc63c498-e311-4eac-9aba-166507716705/prom-label-proxy/0.log"
Sep 17 16:53:19.646533 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:19.646503    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5976878f85-g6lhx_fc63c498-e311-4eac-9aba-166507716705/kube-rbac-proxy-rules/0.log"
Sep 17 16:53:19.685869 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:19.685841    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5976878f85-g6lhx_fc63c498-e311-4eac-9aba-166507716705/kube-rbac-proxy-metrics/0.log"
Sep 17 16:53:20.586226 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:20.586192    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-94gf4/perf-node-gather-daemonset-kbzx6"]
Sep 17 16:53:20.586661 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:20.586540    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bc871a6-2c25-474a-a672-b3b93bc8a308" containerName="storage-initializer"
Sep 17 16:53:20.586661 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:20.586552    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bc871a6-2c25-474a-a672-b3b93bc8a308" containerName="storage-initializer"
Sep 17 16:53:20.586661 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:20.586561    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bc871a6-2c25-474a-a672-b3b93bc8a308" containerName="kube-rbac-proxy"
Sep 17 16:53:20.586661 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:20.586567    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bc871a6-2c25-474a-a672-b3b93bc8a308" containerName="kube-rbac-proxy"
Sep 17 16:53:20.586661 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:20.586576    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28625585-1263-4a19-b85f-8171a75647db" containerName="storage-initializer"
Sep 17 16:53:20.586661 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:20.586582    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="28625585-1263-4a19-b85f-8171a75647db" containerName="storage-initializer"
Sep 17 16:53:20.586661 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:20.586593    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bc871a6-2c25-474a-a672-b3b93bc8a308" containerName="kserve-container"
Sep 17 16:53:20.586661 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:20.586598    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bc871a6-2c25-474a-a672-b3b93bc8a308" containerName="kserve-container"
Sep 17 16:53:20.586661 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:20.586610    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28625585-1263-4a19-b85f-8171a75647db" containerName="storage-initializer"
Sep 17 16:53:20.586661 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:20.586615    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="28625585-1263-4a19-b85f-8171a75647db" containerName="storage-initializer"
Sep 17 16:53:20.586975 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:20.586676    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="28625585-1263-4a19-b85f-8171a75647db" containerName="storage-initializer"
Sep 17 16:53:20.586975 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:20.586683    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="28625585-1263-4a19-b85f-8171a75647db" containerName="storage-initializer"
Sep 17 16:53:20.586975 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:20.586689    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bc871a6-2c25-474a-a672-b3b93bc8a308" containerName="kube-rbac-proxy"
Sep 17 16:53:20.586975 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:20.586695    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bc871a6-2c25-474a-a672-b3b93bc8a308" containerName="kserve-container"
Sep 17 16:53:20.590986 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:20.590965    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-94gf4/perf-node-gather-daemonset-kbzx6"
Sep 17 16:53:20.593551 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:20.593527    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-94gf4\"/\"openshift-service-ca.crt\""
Sep 17 16:53:20.593687 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:20.593527    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-94gf4\"/\"kube-root-ca.crt\""
Sep 17 16:53:20.593687 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:20.593592    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-94gf4\"/\"default-dockercfg-fjpj5\""
Sep 17 16:53:20.603182 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:20.603153    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-94gf4/perf-node-gather-daemonset-kbzx6"]
Sep 17 16:53:20.664878 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:20.664836    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/464a7be3-8348-45c3-9ec8-c2314337b599-sys\") pod \"perf-node-gather-daemonset-kbzx6\" (UID: \"464a7be3-8348-45c3-9ec8-c2314337b599\") " pod="openshift-must-gather-94gf4/perf-node-gather-daemonset-kbzx6"
Sep 17 16:53:20.664878 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:20.664885    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/464a7be3-8348-45c3-9ec8-c2314337b599-podres\") pod \"perf-node-gather-daemonset-kbzx6\" (UID: \"464a7be3-8348-45c3-9ec8-c2314337b599\") " pod="openshift-must-gather-94gf4/perf-node-gather-daemonset-kbzx6"
Sep 17 16:53:20.665105 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:20.664905    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/464a7be3-8348-45c3-9ec8-c2314337b599-proc\") pod \"perf-node-gather-daemonset-kbzx6\" (UID: \"464a7be3-8348-45c3-9ec8-c2314337b599\") " pod="openshift-must-gather-94gf4/perf-node-gather-daemonset-kbzx6"
Sep 17 16:53:20.665105 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:20.664920    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4drxm\" (UniqueName: \"kubernetes.io/projected/464a7be3-8348-45c3-9ec8-c2314337b599-kube-api-access-4drxm\") pod \"perf-node-gather-daemonset-kbzx6\" (UID: \"464a7be3-8348-45c3-9ec8-c2314337b599\") " pod="openshift-must-gather-94gf4/perf-node-gather-daemonset-kbzx6"
Sep 17 16:53:20.665105 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:20.664996    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/464a7be3-8348-45c3-9ec8-c2314337b599-lib-modules\") pod \"perf-node-gather-daemonset-kbzx6\" (UID: \"464a7be3-8348-45c3-9ec8-c2314337b599\") " pod="openshift-must-gather-94gf4/perf-node-gather-daemonset-kbzx6"
Sep 17 16:53:20.766017 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:20.765983    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/464a7be3-8348-45c3-9ec8-c2314337b599-sys\") pod \"perf-node-gather-daemonset-kbzx6\" (UID: \"464a7be3-8348-45c3-9ec8-c2314337b599\") " pod="openshift-must-gather-94gf4/perf-node-gather-daemonset-kbzx6"
Sep 17 16:53:20.766017 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:20.766019    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/464a7be3-8348-45c3-9ec8-c2314337b599-podres\") pod \"perf-node-gather-daemonset-kbzx6\" (UID: \"464a7be3-8348-45c3-9ec8-c2314337b599\") " pod="openshift-must-gather-94gf4/perf-node-gather-daemonset-kbzx6"
Sep 17 16:53:20.766237 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:20.766050    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/464a7be3-8348-45c3-9ec8-c2314337b599-proc\") pod \"perf-node-gather-daemonset-kbzx6\" (UID: \"464a7be3-8348-45c3-9ec8-c2314337b599\") " pod="openshift-must-gather-94gf4/perf-node-gather-daemonset-kbzx6"
Sep 17 16:53:20.766237 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:20.766074    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4drxm\" (UniqueName: \"kubernetes.io/projected/464a7be3-8348-45c3-9ec8-c2314337b599-kube-api-access-4drxm\") pod \"perf-node-gather-daemonset-kbzx6\" (UID: \"464a7be3-8348-45c3-9ec8-c2314337b599\") " pod="openshift-must-gather-94gf4/perf-node-gather-daemonset-kbzx6"
Sep 17 16:53:20.766237 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:20.766117    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/464a7be3-8348-45c3-9ec8-c2314337b599-sys\") pod \"perf-node-gather-daemonset-kbzx6\" (UID: \"464a7be3-8348-45c3-9ec8-c2314337b599\") " pod="openshift-must-gather-94gf4/perf-node-gather-daemonset-kbzx6"
Sep 17 16:53:20.766237 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:20.766152    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/464a7be3-8348-45c3-9ec8-c2314337b599-lib-modules\") pod \"perf-node-gather-daemonset-kbzx6\" (UID: \"464a7be3-8348-45c3-9ec8-c2314337b599\") " pod="openshift-must-gather-94gf4/perf-node-gather-daemonset-kbzx6"
Sep 17 16:53:20.766237 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:20.766175    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/464a7be3-8348-45c3-9ec8-c2314337b599-podres\") pod \"perf-node-gather-daemonset-kbzx6\" (UID: \"464a7be3-8348-45c3-9ec8-c2314337b599\") " pod="openshift-must-gather-94gf4/perf-node-gather-daemonset-kbzx6"
Sep 17 16:53:20.766237 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:20.766194    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/464a7be3-8348-45c3-9ec8-c2314337b599-proc\") pod \"perf-node-gather-daemonset-kbzx6\" (UID: \"464a7be3-8348-45c3-9ec8-c2314337b599\") " pod="openshift-must-gather-94gf4/perf-node-gather-daemonset-kbzx6"
Sep 17 16:53:20.766459 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:20.766263    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/464a7be3-8348-45c3-9ec8-c2314337b599-lib-modules\") pod \"perf-node-gather-daemonset-kbzx6\" (UID: \"464a7be3-8348-45c3-9ec8-c2314337b599\") " pod="openshift-must-gather-94gf4/perf-node-gather-daemonset-kbzx6"
Sep 17 16:53:20.776658 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:20.776610    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4drxm\" (UniqueName: \"kubernetes.io/projected/464a7be3-8348-45c3-9ec8-c2314337b599-kube-api-access-4drxm\") pod \"perf-node-gather-daemonset-kbzx6\" (UID: \"464a7be3-8348-45c3-9ec8-c2314337b599\") " pod="openshift-must-gather-94gf4/perf-node-gather-daemonset-kbzx6"
Sep 17 16:53:20.901396 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:20.901306    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-94gf4/perf-node-gather-daemonset-kbzx6"
Sep 17 16:53:21.184118 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:21.184091    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-94gf4/perf-node-gather-daemonset-kbzx6"]
Sep 17 16:53:21.186536 ip-10-0-135-43 kubenswrapper[2585]: W0917 16:53:21.186503    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod464a7be3_8348_45c3_9ec8_c2314337b599.slice/crio-3605e1fd81ce9c637825aede6c9d0acb9f5cec95889ee7babe951994455e9155 WatchSource:0}: Error finding container 3605e1fd81ce9c637825aede6c9d0acb9f5cec95889ee7babe951994455e9155: Status 404 returned error can't find the container with id 3605e1fd81ce9c637825aede6c9d0acb9f5cec95889ee7babe951994455e9155
Sep 17 16:53:21.721861 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:21.721823    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-94gf4/perf-node-gather-daemonset-kbzx6" event={"ID":"464a7be3-8348-45c3-9ec8-c2314337b599","Type":"ContainerStarted","Data":"9d17695c18c21c7b3d8923ce0ca7f902498d96426e20bac7a46f507600cad7c8"}
Sep 17 16:53:21.722284 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:21.721878    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-94gf4/perf-node-gather-daemonset-kbzx6" event={"ID":"464a7be3-8348-45c3-9ec8-c2314337b599","Type":"ContainerStarted","Data":"3605e1fd81ce9c637825aede6c9d0acb9f5cec95889ee7babe951994455e9155"}
Sep 17 16:53:21.722284 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:21.721951    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-94gf4/perf-node-gather-daemonset-kbzx6"
Sep 17 16:53:21.741562 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:21.741509    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-94gf4/perf-node-gather-daemonset-kbzx6" podStartSLOduration=1.741491849 podStartE2EDuration="1.741491849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:53:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:53:21.739983975 +0000 UTC m=+3557.510315773" watchObservedRunningTime="2026-09-17 16:53:21.741491849 +0000 UTC m=+3557.511823647"
Sep 17 16:53:21.754354 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:21.754327    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-c5p45_eb2b5491-45b0-43b3-8882-e4cc21d2d52f/download-server/0.log"
Sep 17 16:53:22.419470 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:22.419435    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-hzd4v_81d65819-6f7d-4d1e-a851-ef3ae13b5535/volume-data-source-validator/0.log"
Sep 17 16:53:23.343885 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:23.343858    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cvmck_5cad5942-b49d-4bad-b6c4-d881654b6ef5/dns/0.log"
Sep 17 16:53:23.368463 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:23.368436    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cvmck_5cad5942-b49d-4bad-b6c4-d881654b6ef5/kube-rbac-proxy/0.log"
Sep 17 16:53:23.559258 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:23.559228    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xhtm4_c4438a49-8ea1-40aa-bdd9-2707192617af/dns-node-resolver/0.log"
Sep 17 16:53:24.005176 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:24.005147    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6ffcc66d5f-h4njt_8d471801-4e18-4b24-beb7-e1e8f56c0a0e/registry/0.log"
Sep 17 16:53:24.060553 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:24.060525    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dtbtf_28231271-38ce-4985-9020-e3494fd366c7/node-ca/0.log"
Sep 17 16:53:25.303497 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:25.303468    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6qpcq_1f3d6b60-625e-4015-942c-6c8150fa364f/serve-healthcheck-canary/0.log"
Sep 17 16:53:26.144362 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:26.144334    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-n84xc_53c4c887-7ae3-491b-bcec-246b29c06a84/kube-rbac-proxy/0.log"
Sep 17 16:53:26.183138 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:26.183111    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-n84xc_53c4c887-7ae3-491b-bcec-246b29c06a84/exporter/0.log"
Sep 17 16:53:26.226037 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:26.226009    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-n84xc_53c4c887-7ae3-491b-bcec-246b29c06a84/extractor/0.log"
Sep 17 16:53:27.737604 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:27.737579    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-94gf4/perf-node-gather-daemonset-kbzx6"
Sep 17 16:53:28.862338 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:28.862303    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-6fdd8844c6-x5r6k_94a399c0-cbd4-49d6-995f-b8e121a88af2/manager/0.log"
Sep 17 16:53:28.924456 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:28.924409    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-w46wj_a6accedc-04f7-466d-86ea-c8215e9b416e/server/0.log"
Sep 17 16:53:29.198539 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:29.198513    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-nvjlp_033170f4-5975-4cf6-91d6-6abcd2bf9251/manager/0.log"
Sep 17 16:53:29.676262 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:29.676230    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-7x29p_e27478c9-a7f6-491a-bd3f-5e37c0b3808a/seaweedfs/0.log"
Sep 17 16:53:29.736402 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:29.736373    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-custom-8b44d767d-nnnmx_9a0d05c9-f3bf-41dd-afd2-ce3f47a7ab4d/seaweedfs-tls-custom/0.log"
Sep 17 16:53:36.599823 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:36.599793    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cm82k_7e89f60d-872c-4594-8555-427b0be13b7b/kube-multus-additional-cni-plugins/0.log"
Sep 17 16:53:36.634108 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:36.634078    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cm82k_7e89f60d-872c-4594-8555-427b0be13b7b/egress-router-binary-copy/0.log"
Sep 17 16:53:36.663556 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:36.663530    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cm82k_7e89f60d-872c-4594-8555-427b0be13b7b/cni-plugins/0.log"
Sep 17 16:53:36.693047 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:36.693023    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cm82k_7e89f60d-872c-4594-8555-427b0be13b7b/bond-cni-plugin/0.log"
Sep 17 16:53:36.720699 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:36.720676    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cm82k_7e89f60d-872c-4594-8555-427b0be13b7b/routeoverride-cni/0.log"
Sep 17 16:53:36.748020 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:36.747992    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cm82k_7e89f60d-872c-4594-8555-427b0be13b7b/whereabouts-cni-bincopy/0.log"
Sep 17 16:53:36.784210 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:36.784174    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cm82k_7e89f60d-872c-4594-8555-427b0be13b7b/whereabouts-cni/0.log"
Sep 17 16:53:37.311880 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:37.311836    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-xhrb8_4afdccbc-d33e-41ac-98fd-63585ff4d59b/kube-multus/0.log"
Sep 17 16:53:37.391221 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:37.391185    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-m6w9z_7adc5269-e778-473a-a6c7-04754f5e585c/network-metrics-daemon/0.log"
Sep 17 16:53:37.415848 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:37.415814    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-m6w9z_7adc5269-e778-473a-a6c7-04754f5e585c/kube-rbac-proxy/0.log"
Sep 17 16:53:38.292401 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:38.292372    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2b47p_a1850a4d-4d01-4a71-8636-b5918f1426c2/ovn-controller/0.log"
Sep 17 16:53:38.313026 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:38.312996    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2b47p_a1850a4d-4d01-4a71-8636-b5918f1426c2/ovn-acl-logging/0.log"
Sep 17 16:53:38.313026 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:38.313021    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2b47p_a1850a4d-4d01-4a71-8636-b5918f1426c2/ovn-acl-logging/1.log"
Sep 17 16:53:38.333322 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:38.333293    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2b47p_a1850a4d-4d01-4a71-8636-b5918f1426c2/kube-rbac-proxy-node/0.log"
Sep 17 16:53:38.358346 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:38.358317    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2b47p_a1850a4d-4d01-4a71-8636-b5918f1426c2/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 16:53:38.393005 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:38.392978    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2b47p_a1850a4d-4d01-4a71-8636-b5918f1426c2/northd/0.log"
Sep 17 16:53:38.424653 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:38.424622    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2b47p_a1850a4d-4d01-4a71-8636-b5918f1426c2/nbdb/0.log"
Sep 17 16:53:38.450267 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:38.450231    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2b47p_a1850a4d-4d01-4a71-8636-b5918f1426c2/sbdb/0.log"
Sep 17 16:53:38.500898 ip-10-0-135-43 kubenswrapper[2585]: I0917 16:53:38.500867    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2b47p_a1850a4d-4d01-4a71-8636-b5918f1426c2/ovnkube-controller/0.log"